Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
Robin H. Johnson wrote: It does NOT check /proc/config.gz presently. The original logic against not checking /proc was that we were targeting the kernel being built, but that's moot given the use of `uname -r` in OUTPUT_DIR. That seems like a bug. OUTPUT_DIR should default to unset which would cause the internal KV_OUT_DIR to be set to KV_DIR. However I can see from the code that's not the case, and it's not clear why. Relying on uname -r or /proc/config.gz is still something we should avoid (unless user configured, perhaps, or in very exceptional circumstances) however it looks like this isn't a change that you are proposing. Thanks for working on this. It's a sensitive area so please take care and help people pick up the pieces. If you are really motivated, it might be worth rethinking the whole separate output directory implementation. Daniel
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
On Mon, Sep 07, 2009 at 10:28:11AM +0100, Daniel Drake wrote: Robin H. Johnson wrote: It does NOT check /proc/config.gz presently. The original logic against not checking /proc was that we were targeting the kernel being built, but that's moot given the use of `uname -r` in OUTPUT_DIR. That seems like a bug. OUTPUT_DIR should default to unset which would cause the internal KV_OUT_DIR to be set to KV_DIR. However I can see from the code that's not the case, and it's not clear why. It's messy. I'll maybe tackle it after I'm done the work on configs and USE=modules. Relying on uname -r or /proc/config.gz is still something we should avoid (unless user configured, perhaps, or in very exceptional circumstances) however it looks like this isn't a change that you are proposing. The only time I'm saying that uname -r and /proc/config.gz are looked as, is where there is no kernel sources or they are installed but not configured. But that is as part of a larger body of work to fix up all packages that are merely checking the kernel options during build, not relying on the config to build. Kernel modules still require configured sources to be installed. Thanks for working on this. It's a sensitive area so please take care and help people pick up the pieces. If you are really motivated, it might be worth rethinking the whole separate output directory implementation. For anybody following this, we're doing the work on two tracker bugs: 283320 - Changes to linux-info 283900 - USE=modules change for linux-mod and packages. We've got the core eclass stuff done, but not enabled yet. If you want to test it it: 1. mv your .config or entire kernel sources away 2. I_KNOW_WHAT_I_AM_DOING=1 emerge udev You should get warnings, but no errors. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgptG3mmraEGR.pgp Description: PGP signature
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Ok,
Here's the list of ebuilds that feature a CONFIG_CHECK, and which ones
I've been through to check for either NONEED (already using ~option),
MODULE (builds a module so may really want to bomb out), and FIXED
(packages which I've fixed).
About half the list is done, I'm posting it here as a progress/grab 'em
if you want to do 'em type list (as requested by robbat2).
Mike 5:)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.11 (GNU/Linux)
iEYEARECAAYFAkqcUhoACgkQu7rWomwgFXp3GwCfddJEgoKsDgaGTLzkouavCdVp
srQAoLUMEEwmobCBwWAHasrwJjjSt9k5
=iMAd
-END PGP SIGNATURE-
FIXED app-admin/longrun/longrun-0.9-r3.ebuild:CONFIG_CHECK=X86_MSR X86_CPUID
FIXED app-cdr/nero/nero-3.5.2.3.ebuild:CONFIG_CHECK=CHR_DEV_SG
FIXED app-cdr/nero/nero-3.5.3.1.ebuild:CONFIG_CHECK=CHR_DEV_SG
FIXED app-crypt/truecrypt/truecrypt-6.2.ebuild: local
CONFIG_CHECK=BLK_DEV_DM DM_CRYPT FUSE_FS CRYPTO
FIXED app-crypt/truecrypt/truecrypt-6.2a.ebuild: local
CONFIG_CHECK=BLK_DEV_DM DM_CRYPT FUSE_FS CRYPTO
MODULE app-emulation/vmware-modules/vmware-modules-1.0.0.15-r2.ebuild:
CONFIG_CHECK=
MODULE app-emulation/vmware-modules/vmware-modules-1.0.0.15-r2.ebuild:
CONFIG_CHECK=UNUSED_SYMBOLS
MODULE app-laptop/acer_acpi/acer_acpi-0.10.ebuild:CONFIG_CHECK=LEDS_CLASS
BACKLIGHT_LCD_SUPPORT
MODULE app-laptop/acer_acpi/acer_acpi-0.11.2.ebuild:CONFIG_CHECK=LEDS_CLASS
BACKLIGHT_LCD_SUPPORT
MODULE app-laptop/acer_acpi/acer_acpi-0.8.2.ebuild:CONFIG_CHECK=LEDS_CLASS
MODULE app-laptop/acpi4asus/acpi4asus-0.41.ebuild:
CONFIG_CHECK=LEDS_CLASS
MODULE app-laptop/acpi4asus/acpi4asus-0.41.ebuild:
CONFIG_CHECK=~ASUS_LAPTOP
DONEapp-laptop/hdapsd/hdapsd-20060409-r1.ebuild:CONFIG_CHECK=SENSORS_HDAPS
DONEapp-laptop/hdapsd/hdapsd-20060409-r3.ebuild:
CONFIG_CHECK=SENSORS_HDAPS
DONEapp-laptop/hdapsd/hdapsd-20060409.ebuild:CONFIG_CHECK=SENSORS_HDAPS
MODULE
app-laptop/lenovo-sl-laptop/lenovo-sl-laptop-.ebuild:CONFIG_CHECK=HWMON
BACKLIGHT_CLASS_DEVICE
MODULE app-laptop/tp_smapi/tp_smapi-0.37.ebuild:
CONFIG_CHECK=!SENSORS_HDAPS
MODULE app-laptop/tp_smapi/tp_smapi-0.39.ebuild:
CONFIG_CHECK=!SENSORS_HDAPS
MODULE app-laptop/tp_smapi/tp_smapi-0.40.ebuild:
CONFIG_CHECK=~INPUT_UINPUT
MODULE app-laptop/tp_smapi/tp_smapi-0.40.ebuild:
CONFIG_CHECK=!SENSORS_HDAPS
NONEED app-laptop/tpb/tpb-0.6.4.ebuild:CONFIG_CHECK=~NVRAM
NONEED app-misc/actkbd/actkbd-0.2.8.ebuild:CONFIG_CHECK=~INPUT_EVDEV
NONEED app-misc/dnetc/dnetc-2.9013.498.ebuild: local CONFIG_CHECK=~SYSVIPC
NONEED app-misc/dnetc/dnetc-2.9015.504.ebuild: local CONFIG_CHECK=~SYSVIPC
MODULE
app-misc/sdricoh_cs/sdricoh_cs-0.1.3_p20080521.ebuild:CONFIG_CHECK=MMC_BLOCK
MODULE
app-misc/sdricoh_cs/sdricoh_cs-0.1.3_p20080525.ebuild:CONFIG_CHECK=MMC_BLOCK
FIXED
app-mobilephone/gnokii/gnokii-0.6.27-r2.ebuild:CONFIG_CHECK=UNIX98_PTYS
(STABLE)
FIXED
app-mobilephone/gnokii/gnokii-0.6.27-r4.ebuild:CONFIG_CHECK=UNIX98_PTYS
FIXED app-mobilephone/gnokii/gnokii-.ebuild:CONFIG_CHECK=UNIX98_PTYS
MODULE
dev-embedded/parapin-driver/parapin-driver-1.0.0.ebuild:CONFIG_CHECK=PARPORT
FIXED dev-java/jusb/jusb-0.4.4-r1.ebuild:CONFIG_CHECK=USB_DEVICEFS
MODULE dev-util/sysprof/sysprof-1.0.12-r1.ebuild: CONFIG_CHECK=PROFILING
MODULE dev-util/sysprof/sysprof-1.0.12.ebuild: CONFIG_CHECK=PROFILING
NONEED dev-util/systemtap/systemtap-0.7.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED dev-util/systemtap/systemtap-0.8.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED dev-util/systemtap/systemtap-0.9.5.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED dev-util/systemtap/systemtap-0.9.7.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED dev-util/systemtap/systemtap-0.9.8.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED dev-util/systemtap/systemtap-0.9.9.ebuild:CONFIG_CHECK=~KPROBES ~RELAY
~DEBUG_FS
NONEED gnome-base/gnome-menus/gnome-menus-2.20.3.ebuild:
CONFIG_CHECK=~INOTIFY
MODULE media-sound/alsa-driver/alsa-driver-1.0.20-r1.ebuild: local
CONFIG_CHECK=!SND SOUND ${CHECK_PNP} ${CHECK_ISA} ${CHECK_FW} ${CHECK_PARPORT}
MODULE media-sound/alsa-driver/alsa-driver-.ebuild:local
CONFIG_CHECK=!SND SOUND ${CHECK_PNP} ${CHECK_ISA} ${CHECK_FW}
MODULE media-sound/line6usb/line6usb-0.7.3.ebuild:CONFIG_CHECK=USB SOUND
MODULE media-sound/line6usb/line6usb-0.8.1.ebuild:CONFIG_CHECK=USB SOUND
MODULE media-sound/xfi-drivers/xfi-drivers-1.00.ebuild:CONFIG_CHECK=SND SOUND
MODULE media-tv/ivtv/ivtv-1.0.1.ebuild:CONFIG_CHECK=EXPERIMENTAL KMOD
HAS_IOMEM FW_LOADER I2C I2C_ALGOBIT
MODULE media-tv/ivtv/ivtv-1.0.1.ebuild:
CONFIG_CHECK=${CONFIG_CHECK} FB FB_TRIDENT FRAMEBUFFER_CONSOLE FONTS
MODULE media-tv/ivtv/ivtv-1.0.2.ebuild:CONFIG_CHECK=EXPERIMENTAL KMOD
HAS_IOMEM FW_LOADER I2C I2C_ALGOBIT
MODULE media-tv/ivtv/ivtv-1.0.2.ebuild:
[gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
Seeing the debate raised in the udev thread about checking for the kernel, I'd like to propose that we revise the linux-info.eclass. linux-info already checks a number of locations: - KBUILD_OUTPUT, - KERNEL_DIR, which defaults to /usr/src/linux/ - OUTPUT_DIR, which defaults to /lib/modules/`uname -r`/build/ It does NOT check /proc/config.gz presently. The original logic against not checking /proc was that we were targeting the kernel being built, but that's moot given the use of `uname -r` in OUTPUT_DIR. Additionally, linux-info.eclass already has provisions for non-fatally checking for kernel config options, by prefixing them with a tilde. In parallel to what we actually check, we have the issue of systems that may not have ANY of the above, or ANY kernel sources whatsoever. The most common amongst these are: - Hardened systems - Virtual Machines Proposed solution: -- We need to be able to reduce user error, so we cannot simply make it trust the user by default. So I propose that we add an environment variable (I'm not set on the name yet), eg: EXTERNALLY_BUILT_KERNEL=1 This option will cause linux-info.eclass to consider ALL kernel option checks non-fatal. That way we still get the warnings and logs, but it does not stop the builds. When is the above NOT enough? - The only time that ANY kernel sources are required is when you are building an out-of-tree module. For this purpose, they must be configured. The check for having configured kernel sources must only be executed when the modules are about to be compiled. Putting it in pkg_preinst would block use of binpkgs on (related) machines. - If a package builds modules AND userspace, we should offer a way to build the userspace only, as the user can build their modules externally (or patch them into the kernel) [1] - For packages that ONLY build modules, and no userspace at all, having EXTERNALLY_BUILT_KERNEL=1 means that they should error out? [2] (this case might be thrown into the above one). Footnotes: -- 1. This has already been requested for ipset, bug #274577. 2. What about documentation? Is that enough of userspace still? -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgphnpgtI4Etg.pgp Description: PGP signature
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Robin H. Johnson wrote:
It does NOT check /proc/config.gz presently. The original logic against
not checking /proc was that we were targeting the kernel being built,
but that's moot given the use of `uname -r` in OUTPUT_DIR.
I might be reading the eclass wrong, but it looks as though OUTPUT_DIR
is only set to use uname -r if the KV_* variables match the output of
uname -r:
[ ${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA} == $(uname -r) ] \
OUTPUT_DIR=${OUTPUT_DIR:-/lib/modules/${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA}/build}
Otherwise it's taken from the KBUILD_DIR and then it even checks inside
the KV_DIR's Makefile to find one. So not checking /proc/config.gz
isn't a moot point anymore, which is not to say it's not a good idea,
but the reason against doing it still stands. To deal with running
kernel's, there's a get_running_version function, that will use uname to
fill out the KV_* variables appropriately.
Proposed solution:
--
We need to be able to reduce user error, so we cannot simply make it
trust the user by default. So I propose that we add an environment
variable (I'm not set on the name yet), eg:
EXTERNALLY_BUILT_KERNEL=1
This option will cause linux-info.eclass to consider ALL kernel option
checks non-fatal. That way we still get the warnings and logs, but it
does not stop the builds.
I'm not sure what this is solving? It doesn't seem to reduce user
error, it just allows users a way to override portage errors? It seems
unrelated to config.gz and the discussion going on in the previous
thread. I do like the idea of being able to override portage when it's
stopping us from building (incorrectly), but surely with the capability
to have non-fatal checks, we can try to minimize the times when
portage/the ebuild is wrong using that, instead of a whole new override?
When is the above NOT enough?
-
The only time that ANY kernel sources are required is when you are
building an out-of-tree module. For this purpose, they must be
configured.
There's a lot of ebuilds that use linux-info, and they can't all be
out-of-tree modules. Are they misusing linux-info, or are you saying
there's a specific instance in which linux-info requires configured
kernel sources?
The check for having configured kernel sources must only be executed
when the modules are about to be compiled. Putting it in pkg_preinst
would block use of binpkgs on (related) machines.
- If a package builds modules AND userspace, we should offer a way to
build the userspace only, as the user can build their modules
externally (or patch them into the kernel) [1]
- For packages that ONLY build modules, and no userspace at all, having
EXTERNALLY_BUILT_KERNEL=1 means that they should error out? [2]
(this case might be thrown into the above one).
That all seems fine, but again these just seem like standard guidelines.
Is there not already some how to write kernel module ebuilds page
somewhere that documents how you're supposed to use linux-info?
Mike 5:)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.11 (GNU/Linux)
iEYEARECAAYFAkqa30QACgkQu7rWomwgFXo99gCfd8NbK9QkxtJ9BKalq/n1iBxn
l0QAoLiBsvKXZRzR4Dp8HEtoxrsONCtc
=ZK9i
-END PGP SIGNATURE-
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
On Sun, Aug 30, 2009 at 09:21:24PM +0100, Mike Auty wrote:
[ ${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA} == $(uname -r) ] \
OUTPUT_DIR=${OUTPUT_DIR:-/lib/modules/${KV_MAJOR}.${KV_MINOR}.${KV_PATCH}${KV_EXTRA}/build}
This check is inside the get_version call, and is ultimately used to set
KV_OUT_DIR, which is in turn used for all the config checks.
The conditional part of the check means that it's only used if the
running kernel is also the one currently pointed to by the sources
variables (which were used further up in the function to populate KV_*).
I also realize on closer reading, that the above bit will never even be
reached unless the kernel sources are present.
Otherwise it's taken from the KBUILD_DIR and then it even checks inside
the KV_DIR's Makefile to find one. So not checking /proc/config.gz
isn't a moot point anymore, which is not to say it's not a good idea,
but the reason against doing it still stands. To deal with running
kernel's, there's a get_running_version function, that will use uname to
fill out the KV_* variables appropriately.
FYI:
get_running_version is used in one single ebuild, in the entire tree:
sys-fs/evms/evms-2.5.5-r10.ebuild
And there it's only for a warning.
Proposed solution:
--
We need to be able to reduce user error, so we cannot simply make it
trust the user by default. So I propose that we add an environment
variable (I'm not set on the name yet), eg:
EXTERNALLY_BUILT_KERNEL=1
This option will cause linux-info.eclass to consider ALL kernel option
checks non-fatal. That way we still get the warnings and logs, but it
does not stop the builds.
I'm not sure what this is solving? It doesn't seem to reduce user
error, it just allows users a way to override portage errors? It seems
unrelated to config.gz and the discussion going on in the previous
thread. I do like the idea of being able to override portage when it's
stopping us from building (incorrectly), but surely with the capability
to have non-fatal checks, we can try to minimize the times when
portage/the ebuild is wrong using that, instead of a whole new override?
The great majority of CONFIG_CHECK usage in the tree is fatal already.
It only actually needs to be fatal only when it's being used to build a
module.
However, past bugs have shown lots of users complaining that a package
doesn't work, simply because they did not have the relevant option
enabled in their kernel.
This leaves us between hand-holding the basic user's kernel configuration
(exiting if the kernel config option is not enabled), and changing all
non-module instances in the tree to be non-fatal.
If we make all the non-module uses non-fatal, there will be an increase of bugs
due to users not reading the warnings.
When is the above NOT enough?
-
The only time that ANY kernel sources are required is when you are
building an out-of-tree module. For this purpose, they must be
configured.
There's a lot of ebuilds that use linux-info, and they can't all be
out-of-tree modules. Are they misusing linux-info, or are you saying
there's a specific instance in which linux-info requires configured
kernel sources?
Here's a few that I know of, off the top of my head:
app-mobilephone/gnokii/gnokii-0.6.27-r4.ebuild:
CONFIG_CHECK=UNIX98_PTYS
/usr/portage/net-print/hplip/hplip-3.9.4b-r1.ebuild:
CONFIG_CHECK=PARPORT PPDEV
/usr/portage/net-firewall/conntrack-tools/conntrack-tools-0.9.13.ebuild:
CONFIG_CHECK=IP_NF_CONNTRACK_NETLINK
CONFIG_CHECK=NF_CT_NETLINK
CONFIG_CHECK=${CONFIG_CHECK} NF_CONNTRACK
linux_chkconfig_present NF_CONNTRACK_IPV4 || \
linux_chkconfig_present NF_CONNTRACK_IPV6 || \
If you don't have configured kernel sources, all of these ebuilds fail right
now. They are not building modules, simply userspace. The CONFIG_CHECK elements
are not prefixed with tilde to make them non-fatal.
conntrack-tools is actually even worse, as it has a kernel version check before
the config checks, which would also fail without kernel sources.
The check for having configured kernel sources must only be executed
when the modules are about to be compiled. Putting it in pkg_preinst
would block use of binpkgs on (related) machines.
- If a package builds modules AND userspace, we should offer a way to
build the userspace only, as the user can build their modules
externally (or patch them into the kernel) [1]
- For packages that ONLY build modules, and no userspace at all, having
EXTERNALLY_BUILT_KERNEL=1 means that they should error out? [2]
(this case might be thrown into the above one).
That all seems fine, but again these just seem like standard guidelines.
Is there not already some how to write kernel module ebuilds page
somewhere that documents how you're supposed to use linux-info?
If you're building modules, most of the time you're using linux-mod, not just
linux-info. There's no document or
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robin H. Johnson wrote: FYI: get_running_version is used in one single ebuild, in the entire tree: sys-fs/evms/evms-2.5.5-r10.ebuild And there it's only for a warning. Ok, I was just suggesting that if there was an intention to implement config.gz checks, they should only apply when people ask about the running version rather than the build version. Since that doesn't seem popular or even necessary, perhaps neither is the need to check config.gz? The great majority of CONFIG_CHECK usage in the tree is fatal already. It only actually needs to be fatal only when it's being used to build a module. Ok, I see what you're suggesting now. When people want to build packages, but portage knows their kernel isn't setup to run them properly, then it should still build them, but warn them strongly about it (as opposed to currently, where it'll just die). This leaves us between hand-holding the basic user's kernel configuration (exiting if the kernel config option is not enabled), and changing all non-module instances in the tree to be non-fatal. Ok, so then the question is do we sacrifice correctness for fewer (invalid) bugs? Seems like a judgement call. For what it's worth, I'm not sure adding extra plumbing to allow smart users to bypass the checks is the right middle ground. I'd either leave it as is, or change the ebuilds to accurately reflect whether the userspace will build or not. That all seems fine, but again these just seem like standard guidelines. Is there not already some how to write kernel module ebuilds page somewhere that documents how you're supposed to use linux-info? If you're building modules, most of the time you're using linux-mod, not just linux-info. There's no document or recommended behavior in the tree for the above actually, and I'd like to introduce one. Sounds like a good idea, it might also be worth adding to the quizes, if existing devs are asking how it should be done? I guess that's a call on how common it is to have kernel config requirements on userspace... Still, I think I'm on the right page, and even in agreement (which makes me happy). 5:) Thanks! Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) iEYEARECAAYFAkqa9gkACgkQu7rWomwgFXq1PwCfTbp8hqsGZjDmsxKE21gKe1Y8 lYYAoI2EBCn5KwQdlm6Xd8u0q7KGl7gI =Jrsa -END PGP SIGNATURE-
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
On Sun, Aug 30, 2009 at 10:58:33PM +0100, Mike Auty wrote: Robin H. Johnson wrote: FYI: get_running_version is used in one single ebuild, in the entire tree: sys-fs/evms/evms-2.5.5-r10.ebuild And there it's only for a warning. Ok, I was just suggesting that if there was an intention to implement config.gz checks, they should only apply when people ask about the running version rather than the build version. Since that doesn't seem popular or even necessary, perhaps neither is the need to check config.gz? The running version stuff may be needed to be used more than it is presently. If no kernel sources are present at all, all functions that check the kernel version presently bail out. get_version is also at the very top of linux-info's pkg_setup function, so it bails as well: * Determining the location of the kernel source code * Unable to find kernel sources at /usr/src/linux ... The existing state is: - Force the user to install sources Our choices are: - `uname -r` output. - Create an override environment variable for all the checks. /proc/config.gz comes back here again, in that, we can use it as a middle ground with `uname -r`, rather than having the environment variable. So from our discussion, I propose the following: Finding the kernel version: --- 0. (optional) give an env var to bypass entirely. 1. Use existing logic of /usr/src/linux, KERNEL_DIR et al. 2. Fall back to using the running version, with a warning. Checking a configuration option, for non-module use: 0. (optional) give an env var to make all checks non-fatal. 1. Use existing logic of .config from /usr/src/linux, KERNEL_DIR et al. 2. Fall back to /proc/config.gz if available. The great majority of CONFIG_CHECK usage in the tree is fatal already. It only actually needs to be fatal only when it's being used to build a module. Ok, I see what you're suggesting now. When people want to build packages, but portage knows their kernel isn't setup to run them properly, then it should still build them, but warn them strongly about it (as opposed to currently, where it'll just die). Two different cases here: 1. Portage knows their kernel is not setup correctly. 2. Portage cannot find out enough info. It's #2 that bites on virtual machines and hardened servers, and is what I'd like to fix. This leaves us between hand-holding the basic user's kernel configuration (exiting if the kernel config option is not enabled), and changing all non-module instances in the tree to be non-fatal. Ok, so then the question is do we sacrifice correctness for fewer (invalid) bugs? Seems like a judgement call. For what it's worth, I'm not sure adding extra plumbing to allow smart users to bypass the checks is the right middle ground. I'd either leave it as is, or change the ebuilds to accurately reflect whether the userspace will build or not. It's a one-liner to give an override making all checks non-fatal, with the downside that it can't differentiate why the checks are being made. So yes, in the case we should simply fix all of the ebuilds (after the kernel source check is fixed as well). If you're building modules, most of the time you're using linux-mod, not just linux-info. There's no document or recommended behavior in the tree for the above actually, and I'd like to introduce one. Sounds like a good idea, it might also be worth adding to the quizes, if existing devs are asking how it should be done? I guess that's a call on how common it is to have kernel config requirements on userspace... 142 packages inherit linux-mod. 130 packages use functions or the MODULE_NAMES code from linux-mod. 105 packages inherit linux-info. 11 packages inherit both linux-info AND linux-mod. So I propose this as resolutions from the above: 1. USE=modules added to the base profile. 2. Every package that builds kernel modules must offer USE=modules, which can be used to disable building the kernel modules, leaving a pure userspace build of that package. Most of the changes to the above can be done in the linux-mod eclass, so I don't think we'll have to touch that many packages directly. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpshiQ2Mw61x.pgp Description: PGP signature
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robin H. Johnson wrote: The existing state is: - Force the user to install sources Our choices are: - `uname -r` output. - Create an override environment variable for all the checks. /proc/config.gz comes back here again, in that, we can use it as a middle ground with `uname -r`, rather than having the environment variable. Ok, that seems a very reasonable use. So from our discussion, I propose the following: Finding the kernel version: --- 0. (optional) give an env var to bypass entirely. 1. Use existing logic of /usr/src/linux, KERNEL_DIR et al. 2. Fall back to using the running version, with a warning. Checking a configuration option, for non-module use: 0. (optional) give an env var to make all checks non-fatal. 1. Use existing logic of .config from /usr/src/linux, KERNEL_DIR et al. 2. Fall back to /proc/config.gz if available. Where 2 also issues a warning, presumably? I've had a think and can't see any repercussions in changing the default behaviour, but I'm assuming people would only generally hit 2 with virtual machines and/or hardened servers. Can you see either of these suffering from defaulting to the running kernel? Do you know of many circumstances where 2 might get hit by normal users other than those situations? Also (and potentially off-topic), if we're using linux-info to detect kernel sources, whether installed or not, should we also check the tree for ebuilds that require a package which PROVIDES=sources? I personally use a single git checkout, since I think (possibly mistakenly, I honestly haven't checked) that it will leave different checkouts of the kernel all over my /usr/src directory, whenever a new version's emerged. I've had to create a fake-sources package to trick ebuilds that need *some* sources installed, and I'm wondering if that should be necessary? Two different cases here: 1. Portage knows their kernel is not setup correctly. 2. Portage cannot find out enough info. It's #2 that bites on virtual machines and hardened servers, and is what I'd like to fix. Ok, I'm all for it, and the solution you've suggested sounds fine. It's a one-liner to give an override making all checks non-fatal, with the downside that it can't differentiate why the checks are being made. So yes, in the case we should simply fix all of the ebuilds (after the kernel source check is fixed as well). I'd definitely try and do things the right way, and a global override (whilst easy) doesn't sound like it. Fixing all the ebuilds (and the kernel source check) sounds like the way to go. So I propose this as resolutions from the above: 1. USE=modules added to the base profile. 2. Every package that builds kernel modules must offer USE=modules, which can be used to disable building the kernel modules, leaving a pure userspace build of that package. Most of the changes to the above can be done in the linux-mod eclass, so I don't think we'll have to touch that many packages directly. Yep, although if the changes are in linux-mod, then those packages that *only* provide kernel modules will need a way to not present that use flag (no point installing a package is USE=-modules and nothing gets installed). Also, I'd assume +modules would be added as a default. The whole plan sounds extremely sensible in general! 5:) Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) iEYEARECAAYFAkqbC3oACgkQu7rWomwgFXpwywCeKzcB0Znzuul3wq9U9ew5+SuX scQAnjShkQTVQQrFABb8u2t0RsP9K34z =LFlY -END PGP SIGNATURE-
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
On Mon, Aug 31, 2009 at 12:30:02AM +0100, Mike Auty wrote: Checking a configuration option, for non-module use: 0. (optional) give an env var to make all checks non-fatal. 1. Use existing logic of .config from /usr/src/linux, KERNEL_DIR et al. 2. Fall back to /proc/config.gz if available. Where 2 also issues a warning, presumably? I've had a think and can't see any repercussions in changing the default behaviour, but I'm assuming people would only generally hit 2 with virtual machines and/or hardened servers. Can you see either of these suffering from defaulting to the running kernel? Do you know of many circumstances where 2 might get hit by normal users other than those situations? Yes, a warning with using /proc/config.gz. I missed a bit for the config option. If there is NO source of the config data, what do we do? Error out or more warnings? Also (and potentially off-topic), if we're using linux-info to detect kernel sources, whether installed or not, should we also check the tree for ebuilds that require a package which PROVIDES=sources? I personally use a single git checkout, since I think (possibly mistakenly, I honestly haven't checked) that it will leave different checkouts of the kernel all over my /usr/src directory, whenever a new version's emerged. I've had to create a fake-sources package to trick ebuilds that need *some* sources installed, and I'm wondering if that should be necessary? This is what I use for my home workstation: /etc/portage/profile/virtuals:virtual/linux-sources sys-kernel/git-sources /etc/portage/profile/virtuals:virtual/alsa sys-kernel/git-sources /etc/portage/profile/package.provided:sys-kernel/git-sources-2.6.30 Saves having any fake package like that. It's a one-liner to give an override making all checks non-fatal, with the downside that it can't differentiate why the checks are being made. So yes, in the case we should simply fix all of the ebuilds (after the kernel source check is fixed as well). I'd definitely try and do things the right way, and a global override (whilst easy) doesn't sound like it. Fixing all the ebuilds (and the kernel source check) sounds like the way to go. Want to lend a hand fixing up the ebuilds then? I'll work on the eclass sources check. So I propose this as resolutions from the above: 1. USE=modules added to the base profile. 2. Every package that builds kernel modules must offer USE=modules, which can be used to disable building the kernel modules, leaving a pure userspace build of that package. Yep, although if the changes are in linux-mod, then those packages that *only* provide kernel modules will need a way to not present that use flag (no point installing a package is USE=-modules and nothing gets installed). There IS still a point of having the entirely empty kernel package installed: - Split userspace/kernel packages where the userspace package has a dependency on the module-providing package. - other packages that might have a dependency on the module-providing package. - /etc/modprobe.d/ files. USE=-modules will ONLY block files installed to /lib/modules/. Also, I'd assume +modules would be added as a default. I'm not sure we can get away with USE-defaults for this change, due to the amount of EAPI=0 stuff that will be changing, so it'll be in profiles/base/make.defaults instead. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robin H. Johnson wrote: Yes, a warning with using /proc/config.gz. I missed a bit for the config option. If there is NO source of the config data, what do we do? Error out or more warnings? Well, if we can't determine whether a config option's set or not, if it's not critical (ie, it's ~CHECK), then we warn. If there's any hard checks, then we error out I suppose. This is what I use for my home workstation: /etc/portage/profile/virtuals:virtual/linux-sources sys-kernel/git-sources /etc/portage/profile/virtuals:virtual/alsa sys-kernel/git-sources /etc/portage/profile/package.provided:sys-kernel/git-sources-2.6.30 Saves having any fake package like that. Ah cunning, I'll have to try that... Want to lend a hand fixing up the ebuilds then? I'll work on the eclass sources check. Sure, as long as I sure I know how we're fixing them, I'd be glad to help. 5:) I'd just be fixing the userspace ones to make sure they're ~CHECK or would I be doing something else as well? Also should I do that via bugs and a week's grace period, or would you recommend I just dive right in? There IS still a point of having the entirely empty kernel package installed: - Split userspace/kernel packages where the userspace package has a dependency on the module-providing package. - other packages that might have a dependency on the module-providing package. - /etc/modprobe.d/ files. USE=-modules will ONLY block files installed to /lib/modules/. Ok, presumably USE=-modules will also stop the kernel checks, otherwise the following could occur. Let's say the foobar package builds external modules and explicitly requires that foobar not be set in the kernel. With USE=-modules, there's no internal version and there's no external version, but the kernel package is installed, and the deps are happy, even though they'll bug out when it's showtime. Presumably there's also packages that aren't split, or depended on, where not installing the modules is never a good idea. Is there still a way to disable the USE flag, or will it be a requirement of the linux-mod package? I'm not sure we can get away with USE-defaults for this change, due to the amount of EAPI=0 stuff that will be changing, so it'll be in profiles/base/make.defaults instead. That's fine, just so long as it's on and not off, I don't mind what system we use to do it. 5:) Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) iEYEARECAAYFAkqbEqkACgkQu7rWomwgFXq53QCdFi8YlDrKwkh9b0g6EX2DBI35 0t0AnRRRV+oxqQaKPqzcWxGJDtW2lszQ =oia+ -END PGP SIGNATURE-
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
On Mon, Aug 31, 2009 at 01:00:41AM +0100, Mike Auty wrote: I missed a bit for the config option. If there is NO source of the config data, what do we do? Error out or more warnings? Well, if we can't determine whether a config option's set or not, if it's not critical (ie, it's ~CHECK), then we warn. If there's any hard checks, then we error out I suppose. +1. This is what I use for my home workstation: /etc/portage/profile/virtuals:virtual/linux-sources sys-kernel/git-sources /etc/portage/profile/virtuals:virtual/alsa sys-kernel/git-sources /etc/portage/profile/package.provided:sys-kernel/git-sources-2.6.30 Saves having any fake package like that. Ah cunning, I'll have to try that... We should document it in the kernel web pages. Want to lend a hand fixing up the ebuilds then? I'll work on the eclass sources check. Sure, as long as I sure I know how we're fixing them, I'd be glad to help. 5:) I'd just be fixing the userspace ones to make sure they're ~CHECK or would I be doing something else as well? Also should I do that via bugs and a week's grace period, or would you recommend I just dive right in? If you feel like reviewing ~140 packages and filing bugs for them, I won't stop you. But I'm just going to go and fix the ones that seem simple enough to me, and only file bugs for the complex ones. There IS still a point of having the entirely empty kernel package installed: - Split userspace/kernel packages where the userspace package has a dependency on the module-providing package. - other packages that might have a dependency on the module-providing package. - /etc/modprobe.d/ files. USE=-modules will ONLY block files installed to /lib/modules/. Ok, presumably USE=-modules will also stop the kernel checks, otherwise the following could occur. Let's say the foobar package builds external modules and explicitly requires that foobar not be set in the kernel. With USE=-modules, there's no internal version and there's no external version, but the kernel package is installed, and the deps are happy, even though they'll bug out when it's showtime. Err, I'm not following what you claim is the problem here? cat/foo-mod: inherit linux-mod CONFIG_CHECK=!... MODULE_NAMES= cat/foo-usr: RDEPEND=cat/foo And say foo-mod provides /dev/foo for some crazy hardware. The user sets USE=-modules because they have built cat/foo-mod on their own, into the kernel. foo-mod installs nothing, foo-usr installs fine. foo-usr runs, and /dev/foo already exists, so it gets used fine. Presumably there's also packages that aren't split, or depended on, where not installing the modules is never a good idea. Is there still a way to disable the USE flag, or will it be a requirement of the linux-mod package? It needs to be ALWAYS available. The ipset bug I linked earlier in the thread was an example of that. The userspace is useless without the kernel code, but there is nothing to stop the user patching it into their kernel and not having it as a module at all (as is the case on a couple of my work boxes, which is why the bug got filed). -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpGQavFipWmZ.pgp Description: PGP signature
Re: [gentoo-dev] linux-info.eclass: lacking sources, config checks and module building
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robin H. Johnson wrote: If you feel like reviewing ~140 packages and filing bugs for them, I won't stop you. But I'm just going to go and fix the ones that seem simple enough to me, and only file bugs for the complex ones. Ok, I'll do what I can tomorrow. I'm off to bed now... 5:) Err, I'm not following what you claim is the problem here? cat/foo-mod: inherit linux-mod CONFIG_CHECK=!option ... The user sets USE=-modules because they have built cat/foo-mod on their own, into the kernel. foo-mod installs nothing Just checking that it will actually install nothing, rather than failing out with an error because the kernel has option set. I'm just re-iterating that CONFIG_CHECK should only occur if USE=modules. If USE=-modules, then the CONFIG_CHECK options should probably be ignored shouldn't they? It needs to be ALWAYS available. The ipset bug I linked earlier in the thread was an example of that. The userspace is useless without the kernel code, but there is nothing to stop the user patching it into their kernel and not having it as a module at all (as is the case on a couple of my work boxes, which is why the bug got filed). Ok, fair enough. However, in that instance, ipset is a userspace dependency, I was talking about kernel modules that have no userspace dependencies. I suppose there could be a case where someone is trying to custom build a userspace tool that isn't in portage yet, but I still think it might cause confusion to allow USE=modules on kernel modules (without any dependencies) that then install nothing. I'm wondering which will cause more bugs to be filed? Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.11 (GNU/Linux) iEYEARECAAYFAkqbHEEACgkQu7rWomwgFXrEpQCeIMWofCtvwHKJoZsb3/qUyLcP tTUAoIb3Sr645x6NY82LXK6i/3g75NF5 =uaOB -END PGP SIGNATURE-
