Re: [gentoo-dev] Idea: User centric kernel configuration
Am Freitag, 12. März 2021, 19:28:51 CET schrieb Mike Pagano: > > On 3/12/21 1:04 PM, Gerion Entrup wrote: > > Am Freitag, 12. März 2021, 17:40:02 CET schrieb Mike Pagano: > >> > >> On 3/12/21 11:34 AM, Gerion Entrup wrote: > >>> Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge: > Hi, > > Gerion Entrup wrote: > > the Linux kernel has _a lot of_ configuration options, way too many to > > configure them by hand. > > I actually disagree strongly with that; I think it's important to > actively decide what kernels include, and I routinely do, but of > course not everyone will. I've made sure to include a kernel build > when teaching systems administration courses and would again. > > As the kernel becomes more complex the threshold for the first > configuration also rises, but it's still completely possible to learn > what you need in order to successfully configure your own kernel. > I guess it's on the order of a weekend project given some basic > understanding of computer architecture and programming. > >>> > >>> I think, we mean two different things here. I for myself also configured > >>> my own kernel(s) for years. I also actively teach students to do so. > >>> However, I have never looked into all 18000 configuration options. I > >>> don't understand them all (by far). > >>> > >>> Actually, most of the times, I do a `make localmodconfig`, click through > >>> various subsystems and activate what I think that it is useful or sounds > >>> nice. Then, to update a kernel, I use `make oldconfig` and answer the > >>> questions as good as I can (taking the default otherwise). > >>> > >>> If my kernel is not capable to do something (for example run docker > >>> containers), I take a look in the Gentoo Wiki and "copy" the options > >>> into my own config. > >>> > >>> But for me, this is not an informed decision. Of course, I learn > >>> something in this process, maybe also more than a precompiled kernel > >>> user, but I have by far not created a minimal config or even begin > >>> to understand all subsystems and different configuration options. > >>> > >>> I'm sure that this is similar to most Gentoo users. > >>> > >>> BTW, here is a project to create a minimal config (kind of "ideal" > >>> config) based on a "golden run" [1]. However, I have not compared such > >>> a config with my own homegrown config. > >>> > >>> > > This requires a mapping between user oriented "features" and the kernel > > internal configuration options. > > So the challenge here is that the kernel is disjoint from user space, > and while the kernel API remains stable over time consumer requirements > such as "docker" or "cryptsetup" will mean different things for > different versions of particular user space software. > > > > Do you think that it is useful and feasible to combine these two > > mechanisms? > > AFAIK there's no generic method for formal kernel requirements in user > space packages and there's also no sanctioned method for quering > kernel capabilities. The only thing available is /proc/config if that > was enabled in the kernel build, and there are of course reasons to > leave it out, and it only applies to the particular running kernel, > e.g. useless for cross-compilation. There, it would be possible to > read the kernel configuration file if the kernel source code is > available when the userspace package is being built, but that's not > guaranteed. > > In Gentoo, linux-info.eclass provides linux_config_exists() to do all > of this, but order to become a widespread success there would have to > be one method for upstreams to maintain these requirements as part of > their packages, rather than forcing the burden on package maintainers > to repeat the same detective task in every single distribution. > > I think it would be very useful to create something generic for that, > but that's certainly no small task. > > And realistically I only see it succeeding if Linux Foundation decides > to push it onto the world. > >>> > >>> Sorry, I may have expressed this not clearly. I guess, we actually have > >>> the same opinion here. I don't want to do this mapping automatically. I > >>> don't want a "fully automatic" config or touch any (user space) ebuilds. > >>> > >>> My idea is to patch Kconfig (as part of gentoo-sources) to provide the > >>> same "features" (i.e. mappings) that are already present in the Gentoo > >>> wiki. > >>> > >>> For example this could result in this installation description for Docker > >>> (within the Wiki): > >>> "To enable kernel support for Docker, enable > >>> 'Gentoo Linux -> Support for user space programs -> Docker'" > >>> > >>> This Kconfig flag than depends on all options that are needed for proper > >>> Docker support (as already described in the
Re: [gentoo-dev] Idea: User centric kernel configuration
On Fri, 2021-03-12 at 11:40 -0500, Mike Pagano wrote: > > have for systemd and openrc where you select something and we preselect > a bunch of required/suggested kernel configs options. Another good candidate for this mechanism would be the recommendations on https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project/Recommended_Settings perhaps linked to a hardened USE flag and/or profile. Hans signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Idea: User centric kernel configuration
On 3/12/21 1:04 PM, Gerion Entrup wrote: Am Freitag, 12. März 2021, 17:40:02 CET schrieb Mike Pagano: On 3/12/21 11:34 AM, Gerion Entrup wrote: Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge: Hi, Gerion Entrup wrote: the Linux kernel has _a lot of_ configuration options, way too many to configure them by hand. I actually disagree strongly with that; I think it's important to actively decide what kernels include, and I routinely do, but of course not everyone will. I've made sure to include a kernel build when teaching systems administration courses and would again. As the kernel becomes more complex the threshold for the first configuration also rises, but it's still completely possible to learn what you need in order to successfully configure your own kernel. I guess it's on the order of a weekend project given some basic understanding of computer architecture and programming. I think, we mean two different things here. I for myself also configured my own kernel(s) for years. I also actively teach students to do so. However, I have never looked into all 18000 configuration options. I don't understand them all (by far). Actually, most of the times, I do a `make localmodconfig`, click through various subsystems and activate what I think that it is useful or sounds nice. Then, to update a kernel, I use `make oldconfig` and answer the questions as good as I can (taking the default otherwise). If my kernel is not capable to do something (for example run docker containers), I take a look in the Gentoo Wiki and "copy" the options into my own config. But for me, this is not an informed decision. Of course, I learn something in this process, maybe also more than a precompiled kernel user, but I have by far not created a minimal config or even begin to understand all subsystems and different configuration options. I'm sure that this is similar to most Gentoo users. BTW, here is a project to create a minimal config (kind of "ideal" config) based on a "golden run" [1]. However, I have not compared such a config with my own homegrown config. This requires a mapping between user oriented "features" and the kernel internal configuration options. So the challenge here is that the kernel is disjoint from user space, and while the kernel API remains stable over time consumer requirements such as "docker" or "cryptsetup" will mean different things for different versions of particular user space software. Do you think that it is useful and feasible to combine these two mechanisms? AFAIK there's no generic method for formal kernel requirements in user space packages and there's also no sanctioned method for quering kernel capabilities. The only thing available is /proc/config if that was enabled in the kernel build, and there are of course reasons to leave it out, and it only applies to the particular running kernel, e.g. useless for cross-compilation. There, it would be possible to read the kernel configuration file if the kernel source code is available when the userspace package is being built, but that's not guaranteed. In Gentoo, linux-info.eclass provides linux_config_exists() to do all of this, but order to become a widespread success there would have to be one method for upstreams to maintain these requirements as part of their packages, rather than forcing the burden on package maintainers to repeat the same detective task in every single distribution. I think it would be very useful to create something generic for that, but that's certainly no small task. And realistically I only see it succeeding if Linux Foundation decides to push it onto the world. Sorry, I may have expressed this not clearly. I guess, we actually have the same opinion here. I don't want to do this mapping automatically. I don't want a "fully automatic" config or touch any (user space) ebuilds. My idea is to patch Kconfig (as part of gentoo-sources) to provide the same "features" (i.e. mappings) that are already present in the Gentoo wiki. For example this could result in this installation description for Docker (within the Wiki): "To enable kernel support for Docker, enable 'Gentoo Linux -> Support for user space programs -> Docker'" This Kconfig flag than depends on all options that are needed for proper Docker support (as already described in the Wiki) like cgroups etc.. This would allow users to configure there kernels the same way as before but with some additional convenience shortcuts. Of course, this is a high maintenance burden for the kernel package maintainers. Therefore, this mechanism maybe can be automated: Grep the wiki pages for kernel config snippets and automatically condense that in a Kconfig readable configuration option. A possible way could be to automatically extract the kernel config flags from the wiki pages and map them to Kconfig options. At very best that will only be valid for some particular point in time, like current CONFIG_CHECK in ebuilds using linux_
Re: [gentoo-dev] Idea: User centric kernel configuration
Am Freitag, 12. März 2021, 17:40:02 CET schrieb Mike Pagano: > > On 3/12/21 11:34 AM, Gerion Entrup wrote: > > Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge: > >> Hi, > >> > >> Gerion Entrup wrote: > >>> the Linux kernel has _a lot of_ configuration options, way too many to > >>> configure them by hand. > >> > >> I actually disagree strongly with that; I think it's important to > >> actively decide what kernels include, and I routinely do, but of > >> course not everyone will. I've made sure to include a kernel build > >> when teaching systems administration courses and would again. > >> > >> As the kernel becomes more complex the threshold for the first > >> configuration also rises, but it's still completely possible to learn > >> what you need in order to successfully configure your own kernel. > >> I guess it's on the order of a weekend project given some basic > >> understanding of computer architecture and programming. > > > > I think, we mean two different things here. I for myself also configured > > my own kernel(s) for years. I also actively teach students to do so. > > However, I have never looked into all 18000 configuration options. I > > don't understand them all (by far). > > > > Actually, most of the times, I do a `make localmodconfig`, click through > > various subsystems and activate what I think that it is useful or sounds > > nice. Then, to update a kernel, I use `make oldconfig` and answer the > > questions as good as I can (taking the default otherwise). > > > > If my kernel is not capable to do something (for example run docker > > containers), I take a look in the Gentoo Wiki and "copy" the options > > into my own config. > > > > But for me, this is not an informed decision. Of course, I learn > > something in this process, maybe also more than a precompiled kernel > > user, but I have by far not created a minimal config or even begin > > to understand all subsystems and different configuration options. > > > > I'm sure that this is similar to most Gentoo users. > > > > BTW, here is a project to create a minimal config (kind of "ideal" > > config) based on a "golden run" [1]. However, I have not compared such > > a config with my own homegrown config. > > > > > >>> This requires a mapping between user oriented "features" and the kernel > >>> internal configuration options. > >> > >> So the challenge here is that the kernel is disjoint from user space, > >> and while the kernel API remains stable over time consumer requirements > >> such as "docker" or "cryptsetup" will mean different things for > >> different versions of particular user space software. > >> > >> > >>> Do you think that it is useful and feasible to combine these two > >>> mechanisms? > >> > >> AFAIK there's no generic method for formal kernel requirements in user > >> space packages and there's also no sanctioned method for quering > >> kernel capabilities. The only thing available is /proc/config if that > >> was enabled in the kernel build, and there are of course reasons to > >> leave it out, and it only applies to the particular running kernel, > >> e.g. useless for cross-compilation. There, it would be possible to > >> read the kernel configuration file if the kernel source code is > >> available when the userspace package is being built, but that's not > >> guaranteed. > >> > >> In Gentoo, linux-info.eclass provides linux_config_exists() to do all > >> of this, but order to become a widespread success there would have to > >> be one method for upstreams to maintain these requirements as part of > >> their packages, rather than forcing the burden on package maintainers > >> to repeat the same detective task in every single distribution. > >> > >> I think it would be very useful to create something generic for that, > >> but that's certainly no small task. > >> > >> And realistically I only see it succeeding if Linux Foundation decides > >> to push it onto the world. > > > > Sorry, I may have expressed this not clearly. I guess, we actually have > > the same opinion here. I don't want to do this mapping automatically. I > > don't want a "fully automatic" config or touch any (user space) ebuilds. > > > > My idea is to patch Kconfig (as part of gentoo-sources) to provide the > > same "features" (i.e. mappings) that are already present in the Gentoo > > wiki. > > > > For example this could result in this installation description for Docker > > (within the Wiki): > > "To enable kernel support for Docker, enable > > 'Gentoo Linux -> Support for user space programs -> Docker'" > > > > This Kconfig flag than depends on all options that are needed for proper > > Docker support (as already described in the Wiki) like cgroups etc.. > > > > This would allow users to configure there kernels the same way as before > > but with some additional convenience shortcuts. > > > > Of course, this is a high maintenance burden for the kernel package > > maintainers. Therefore, this mechanism maybe can
Re: [gentoo-dev] Idea: User centric kernel configuration
On 3/12/21 11:34 AM, Gerion Entrup wrote: Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge: Hi, Gerion Entrup wrote: the Linux kernel has _a lot of_ configuration options, way too many to configure them by hand. I actually disagree strongly with that; I think it's important to actively decide what kernels include, and I routinely do, but of course not everyone will. I've made sure to include a kernel build when teaching systems administration courses and would again. As the kernel becomes more complex the threshold for the first configuration also rises, but it's still completely possible to learn what you need in order to successfully configure your own kernel. I guess it's on the order of a weekend project given some basic understanding of computer architecture and programming. I think, we mean two different things here. I for myself also configured my own kernel(s) for years. I also actively teach students to do so. However, I have never looked into all 18000 configuration options. I don't understand them all (by far). Actually, most of the times, I do a `make localmodconfig`, click through various subsystems and activate what I think that it is useful or sounds nice. Then, to update a kernel, I use `make oldconfig` and answer the questions as good as I can (taking the default otherwise). If my kernel is not capable to do something (for example run docker containers), I take a look in the Gentoo Wiki and "copy" the options into my own config. But for me, this is not an informed decision. Of course, I learn something in this process, maybe also more than a precompiled kernel user, but I have by far not created a minimal config or even begin to understand all subsystems and different configuration options. I'm sure that this is similar to most Gentoo users. BTW, here is a project to create a minimal config (kind of "ideal" config) based on a "golden run" [1]. However, I have not compared such a config with my own homegrown config. This requires a mapping between user oriented "features" and the kernel internal configuration options. So the challenge here is that the kernel is disjoint from user space, and while the kernel API remains stable over time consumer requirements such as "docker" or "cryptsetup" will mean different things for different versions of particular user space software. Do you think that it is useful and feasible to combine these two mechanisms? AFAIK there's no generic method for formal kernel requirements in user space packages and there's also no sanctioned method for quering kernel capabilities. The only thing available is /proc/config if that was enabled in the kernel build, and there are of course reasons to leave it out, and it only applies to the particular running kernel, e.g. useless for cross-compilation. There, it would be possible to read the kernel configuration file if the kernel source code is available when the userspace package is being built, but that's not guaranteed. In Gentoo, linux-info.eclass provides linux_config_exists() to do all of this, but order to become a widespread success there would have to be one method for upstreams to maintain these requirements as part of their packages, rather than forcing the burden on package maintainers to repeat the same detective task in every single distribution. I think it would be very useful to create something generic for that, but that's certainly no small task. And realistically I only see it succeeding if Linux Foundation decides to push it onto the world. Sorry, I may have expressed this not clearly. I guess, we actually have the same opinion here. I don't want to do this mapping automatically. I don't want a "fully automatic" config or touch any (user space) ebuilds. My idea is to patch Kconfig (as part of gentoo-sources) to provide the same "features" (i.e. mappings) that are already present in the Gentoo wiki. For example this could result in this installation description for Docker (within the Wiki): "To enable kernel support for Docker, enable 'Gentoo Linux -> Support for user space programs -> Docker'" This Kconfig flag than depends on all options that are needed for proper Docker support (as already described in the Wiki) like cgroups etc.. This would allow users to configure there kernels the same way as before but with some additional convenience shortcuts. Of course, this is a high maintenance burden for the kernel package maintainers. Therefore, this mechanism maybe can be automated: Grep the wiki pages for kernel config snippets and automatically condense that in a Kconfig readable configuration option. A possible way could be to automatically extract the kernel config flags from the wiki pages and map them to Kconfig options. At very best that will only be valid for some particular point in time, like current CONFIG_CHECK in ebuilds using linux_config_exists() are only valid for particular package versions. At worst it's plain wrong because the req
Re: [gentoo-dev] Idea: User centric kernel configuration
Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge: > Hi, > > Gerion Entrup wrote: > > the Linux kernel has _a lot of_ configuration options, way too many to > > configure them by hand. > > I actually disagree strongly with that; I think it's important to > actively decide what kernels include, and I routinely do, but of > course not everyone will. I've made sure to include a kernel build > when teaching systems administration courses and would again. > > As the kernel becomes more complex the threshold for the first > configuration also rises, but it's still completely possible to learn > what you need in order to successfully configure your own kernel. > I guess it's on the order of a weekend project given some basic > understanding of computer architecture and programming. I think, we mean two different things here. I for myself also configured my own kernel(s) for years. I also actively teach students to do so. However, I have never looked into all 18000 configuration options. I don't understand them all (by far). Actually, most of the times, I do a `make localmodconfig`, click through various subsystems and activate what I think that it is useful or sounds nice. Then, to update a kernel, I use `make oldconfig` and answer the questions as good as I can (taking the default otherwise). If my kernel is not capable to do something (for example run docker containers), I take a look in the Gentoo Wiki and "copy" the options into my own config. But for me, this is not an informed decision. Of course, I learn something in this process, maybe also more than a precompiled kernel user, but I have by far not created a minimal config or even begin to understand all subsystems and different configuration options. I'm sure that this is similar to most Gentoo users. BTW, here is a project to create a minimal config (kind of "ideal" config) based on a "golden run" [1]. However, I have not compared such a config with my own homegrown config. > > This requires a mapping between user oriented "features" and the kernel > > internal configuration options. > > So the challenge here is that the kernel is disjoint from user space, > and while the kernel API remains stable over time consumer requirements > such as "docker" or "cryptsetup" will mean different things for > different versions of particular user space software. > > > > Do you think that it is useful and feasible to combine these two mechanisms? > > AFAIK there's no generic method for formal kernel requirements in user > space packages and there's also no sanctioned method for quering > kernel capabilities. The only thing available is /proc/config if that > was enabled in the kernel build, and there are of course reasons to > leave it out, and it only applies to the particular running kernel, > e.g. useless for cross-compilation. There, it would be possible to > read the kernel configuration file if the kernel source code is > available when the userspace package is being built, but that's not > guaranteed. > > In Gentoo, linux-info.eclass provides linux_config_exists() to do all > of this, but order to become a widespread success there would have to > be one method for upstreams to maintain these requirements as part of > their packages, rather than forcing the burden on package maintainers > to repeat the same detective task in every single distribution. > > I think it would be very useful to create something generic for that, > but that's certainly no small task. > > And realistically I only see it succeeding if Linux Foundation decides > to push it onto the world. Sorry, I may have expressed this not clearly. I guess, we actually have the same opinion here. I don't want to do this mapping automatically. I don't want a "fully automatic" config or touch any (user space) ebuilds. My idea is to patch Kconfig (as part of gentoo-sources) to provide the same "features" (i.e. mappings) that are already present in the Gentoo wiki. For example this could result in this installation description for Docker (within the Wiki): "To enable kernel support for Docker, enable 'Gentoo Linux -> Support for user space programs -> Docker'" This Kconfig flag than depends on all options that are needed for proper Docker support (as already described in the Wiki) like cgroups etc.. This would allow users to configure there kernels the same way as before but with some additional convenience shortcuts. Of course, this is a high maintenance burden for the kernel package maintainers. Therefore, this mechanism maybe can be automated: Grep the wiki pages for kernel config snippets and automatically condense that in a Kconfig readable configuration option. > > A possible way could be to automatically extract the kernel config > > flags from the wiki pages and map them to Kconfig options. > > At very best that will only be valid for some particular point in time, > like current CONFIG_CHECK in ebuilds using linux_config_exists() are > only valid for partic
Re: [gentoo-dev] Idea: User centric kernel configuration
Hi, Gerion Entrup wrote: > the Linux kernel has _a lot of_ configuration options, way too many to > configure them by hand. I actually disagree strongly with that; I think it's important to actively decide what kernels include, and I routinely do, but of course not everyone will. I've made sure to include a kernel build when teaching systems administration courses and would again. As the kernel becomes more complex the threshold for the first configuration also rises, but it's still completely possible to learn what you need in order to successfully configure your own kernel. I guess it's on the order of a weekend project given some basic understanding of computer architecture and programming. > This requires a mapping between user oriented "features" and the kernel > internal configuration options. So the challenge here is that the kernel is disjoint from user space, and while the kernel API remains stable over time consumer requirements such as "docker" or "cryptsetup" will mean different things for different versions of particular user space software. > Do you think that it is useful and feasible to combine these two mechanisms? AFAIK there's no generic method for formal kernel requirements in user space packages and there's also no sanctioned method for quering kernel capabilities. The only thing available is /proc/config if that was enabled in the kernel build, and there are of course reasons to leave it out, and it only applies to the particular running kernel, e.g. useless for cross-compilation. There, it would be possible to read the kernel configuration file if the kernel source code is available when the userspace package is being built, but that's not guaranteed. In Gentoo, linux-info.eclass provides linux_config_exists() to do all of this, but order to become a widespread success there would have to be one method for upstreams to maintain these requirements as part of their packages, rather than forcing the burden on package maintainers to repeat the same detective task in every single distribution. I think it would be very useful to create something generic for that, but that's certainly no small task. And realistically I only see it succeeding if Linux Foundation decides to push it onto the world. > A possible way could be to automatically extract the kernel config > flags from the wiki pages and map them to Kconfig options. At very best that will only be valid for some particular point in time, like current CONFIG_CHECK in ebuilds using linux_config_exists() are only valid for particular package versions. At worst it's plain wrong because the requirements have to be reverse-engineered downstream. //Peter
[gentoo-dev] Idea: User centric kernel configuration
Hi, the Linux kernel has _a lot of_ configuration options, way too many to configure them by hand. Many of these configuration options seem to be kernel centric (they deactivate a specific compilation unit, they deactivate specific source code parts). However, my main configuration requirements as a user are mostly user space program or feature centric. I want to activate kernel support for a specific feature, e.g. docker support or cryptsetup support. This requires a mapping between user oriented "features" and the kernel internal configuration options. This mapping exists already at the Gentoo wiki [e.g. 1, 2, 3]. Gentoo also provides (via gentoo-sources) a set of patches that adds some meta configuration options to enable some of exactly that user specific features (Systemd, OpenRC and Portage support). Do you think that it is useful and feasible to combine these two mechanisms? A possible way could be to automatically extract the kernel config flags from the wiki pages and map them to Kconfig options. Best regards, Gerion [1] https://wiki.gentoo.org/wiki/Docker#Kernel [2] https://wiki.gentoo.org/wiki/Dm-crypt#Kernel_Configuration [3] https://wiki.gentoo.org/wiki/AMDGPU#Kernel signature.asc Description: This is a digitally signed message part.
