Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Fri, 5 Jul 2013 09:38:10 +0100 Steven J. Long sl...@rathaus.eclipse.co.uk wrote: Tom Wijsman wrote: Yes, we currently have base and extras tarballs for genpatches; it is trivial to add a experimental tarball to this set, all the optional experimental patches will be in that tarball. This is also handy for maintainers of other distros whom use genpatches. That's cool. The bit about the user explicitly opting-in to 'fragile' patches still is of concern, however. Why is this still of concern? (See the next response before replying) There shouldn't be a problem here unless the user applies a lot of patches that could introduce a colliding patch; in this case the user either has to fix the conflicting patch or exclude ours. We can't know for ourselves which patches will be troublesome in this light; but well, I feel like this only happens on a very exceptional basis. If an user keeps this amount of patches, ours are probably not needed. No, the problem is as mentioned, with patches for which disabling the configure option, does not stop the patch from changing anything. Such as aufs, as has been outlined by Greg K-H earlier in the thread. My original post mentions 3) The patch should not affect the build by default., which I later clarified with It's just a matter of embedding each + block in the diff with a config check and updating the counts.; in detail we QA check whether all blocks contain such a config check. It does not change anything other than insert some code which won't be parsed if you don't enable the relevant option in the menu config. [... SNIP ...]; and can hardly be called Proper integration, imo. Why not? After all, these are experimental patches. If they don't play nicely, don't let them out of the sandbox, unless the user tells us to. It's the user that can let each individual patch out of its sandbox. Having a clear policy on that makes negotiating with patch authors much simpler too, and it's far more likely they'll fall into line where possible, just as upstreams are usually quite happy to apply portability patches: it means they get more users and feedback. Not sure what is meant by this; but given my above clarifications, I think you were missing a detail in the approach that is taken. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Thu, 4 Jul 2013 06:27:59 +0100 Steven J. Long sl...@rathaus.eclipse.co.uk wrote: Tom Wijsman wrote: Steven J. Long wrote: If it does [affect the build by default] then it should never be applied, unless the user specifically asks for it, imo, and the resultant kernel is labelled -exp as you suggest. Yes, we are going to introduce an experimental USE flag for this. That's for the over-arching set of patches isn't it? Which takes care of the labelling. Yes, we currently have base and extras tarballs for genpatches; it is trivial to add a experimental tarball to this set, all the optional experimental patches will be in that tarball. This is also handy for maintainers of other distros whom use genpatches. It's just a matter of embedding each + block in the diff with a config check and updating the counts. .. I can convert the original developer's patch whenever he updates it; or on top of that, write a script to generate the original patch back. Please, just keep a copy of the original patch as well as the modified output from the script, somewhere reasonable to you, if you are doing any editing. Traceability is essential here. But yes, some git branches can easily cover the editing part. I think a lot of these things are checks that should happen before patches are included, and on every upgrade. So we need to separate out what the developer/ebuild-maintainer has to do to prepare files/, and what needs to happen in the ebuild itself. Please note that this discussion is regarding genpatches; so, there is no files/ directory and the ebuild change is very minimal, changing one or two numbers to indicate the genpatches version to use. Every time I add a patch to genpatches I compile a test kernel using a test ebuild; I plan to add these checks to our genpatches scripts, then I can call the checks script from the ebuild in a QA way. Unless of course the user specifically requests it. This can be a simple variable with a list of required patches, or whatever. With USE=-experimental (which will be the default) they are excluded by default, after enabling that the user can exclude patches by setting UNIPATCH_EXCLUDE through the package.env mechanism. I'd feel happier if certain patches, the troublesome ones discussed, had to be explicity enabled, before the configuration options and the patch to kernel files took place. Perhaps via UNIPATCH_INCLUDE or USE=aufs. There shouldn't be a problem here unless the user applies a lot of patches that could introduce a colliding patch; in this case the user either has to fix the conflicting patch or exclude ours. We can't know for ourselves which patches will be troublesome in this light; but well, I feel like this only happens on a very exceptional basis. If an user keeps this amount of patches, ours are probably not needed. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
gentoo-checkconf script Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 07/01/2013 11:53 PM, Anthony G. Basile wrote: Now I'm confused because gentoo-sources is gentoo specific. It contains stuff that we need in gentoo but other distros do not need, like our end-to-end support for certain xattr namespaces. If you remove these then we must either 1) maintain a userland which is not in line with other distros or 2) give up on critical features we want in gentoo, like markings on elf object in user.pax.flags and certain caps, as well as in the future preserving selinux labels through emerge. Upstream will not accept them because of who needs that crap and we can't give them up without loosing core functionality. Feel free to review those patches but don't ask us to drop them from gentoo-sources because their not in upstream. What about a check-kernel-config-for-gentoo-compliance script for starterts? I manage a handfull of kernel configs over some years (laptop vs. server, graphics, firewalling capabilities) and was always tempted to write an script to check if the config meets a certain set of requirements. I think of xattr, selinux, gentoo-boot and so on, that can be expanded by users demand, like, CONFIG_CMDLINE should include and CONFIG_DEFAULT_HOSTNAME=x and all iptables target on. A additional make target in gentoo-sources could the warn about any missing feature, and ask for yes or wait some seconds. (I remember reaging some funny note about my kernel supporting x32 but by userland not, like that kernel build would run on that userland) == Merging a certain source does always imply to run it on that system. (diff-ing configs is really nasty since sub*module=N drops lines from the config) (and i got lazy on reading all the added features in subsystems [1]) Michael I can live with a lot of things, as long as I can configure/compile/update my kernel and the out-of-tree drivers when i want Weber [1] http://3.bp.blogspot.com/_rtOXMZlMTkg/RZWVjP3f49I/ADs/YpHlSwXpiUg/s400/drinking_bird.jpg - -- Michael Weber Gentoo Developer web: https://xmw.de/ mailto: Michael Weber x...@gentoo.org -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlHSj+IACgkQknrdDGLu8JD65AD+NHyGeFNQw4GceLp0g9ypik5j NzoEwKYztMCOwKcjbO4A/A1e/KQv4DabFoZA41kdPBH8DMOITWL7Jb3OHqewwpPL =OOdc -END PGP SIGNATURE-
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
Almost all of this portion of the thread is off-topic for gentoo-dev, so I'll leave it alone, and will be more than willing to take it up somewhere else it is on-topic for, like linux-kernel, if you want to. But, there is one thing I do want to ask/comment on, as it is relevant to users of Gentoo: On Mon, Jul 01, 2013 at 11:29:39PM -0400, Richard Yao wrote: 4. Risk of patent trolls I know of no kernel patches outside of the tree because of this, do you? There is compatibility code for DOS long filenames in fat that is no longer in-tree because of this. I remember the conversations that a number of us had a few years ago about this potential issue, but do not see any in-kernel changes that were ever made because of that. Could you point me at the changes that were made that were accepted into the kernel.org tree? thanks, gerg k-h
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 11:20 AM, Jeff Horelick wrote: On 1 July 2013 10:41, Tom Wijsman tom...@gentoo.org wrote: Hello Please reply to gentoo-dev in case ML daemon changes Reply-To. ### TL; DR ### By introducing feature patches which menu options are disabled by default to genpatches, we can deduplicate *-sources maintainers as well as large groups of users work. By introducing a distribution section in the menuconfig, we can provide options that enable minimal Gentoo requirements by default (DEVTMPFS, making Gentoo systems bootable since an udev release a long time ago) and other distribution stuff. I really like this idea. geek-sources has appealed to me massively the past few months, but i didn't want to risk stability and go with a kernel from a unofficial overlay. I can not wait to see this in gentoo-sources and I fully support this idea. Tom, you already know my opinion because we discussed it. I'm all for it. Just a reminder: there's always problems somewhere in the kernel which can be triggered by various options. The kernel is not one big take it or leave it chunk of code, but many chunks selectable by Kconfig with the exception of course of the core. The best we can do wrt to BFQ and other risky patches is mark these options as EXPERIMENTAL. I was going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated. See scripts/checkpatch.pl Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580; --Tony -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 1 July 2013 19:17, Greg KH gre...@gentoo.org wrote: greg stick to the vanilla-sources k-h Before these changes were introduced, the gentoo-sources and vanilla-sources were quite similar in the sense that genpatches used to contain patches already in Linus' tree. However, given that gentoo-sources will become heavily patched by external 3rd-party patches, it might makes more sense to recommend vanilla-sources to our users (handbook, etc). I certainly don't feel safe anymore running non-upstream code in production boxes. -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 11:17:49 -0700 Greg KH gre...@gentoo.org wrote: Meet CONFIG_DEVTMPFS; ... This is not the only build option that users must enable to get a booting system by far. So why single this one out? Because it is an example. Later on I explicitly mention There are a small set of other variables in this nature, ...; I'm talking about configuration options not present in the defconfig, specifically those that everyone who runs a Gentoo based system (@system set + stage3) wants to have enabled. Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. In other words, genpatches stay as stable as before; unless you explicitly toggle options that intentionally make it unstable. As pointed out above, this is not always true. Under the above condition (3), it is always true. Also, as others stated, the hardened patches also don't always only touch areas covered by non-config-selected portions of the kernel. Yes, it seems wise to keep hardened-sources separated. Mix and match your external kernel patches at your own risk, what you are suggesting does make it easy for users, but I bet it will be a huge support issue for the already-overworked gentoo kernel developers, the combinations just are too big to test and guarantee working. I'm waiting for you to push new kernels to kernel.org. Joking aside; users are doing this anyway, it is better to know about it. Who knows some of the bugs we have is the result of our unawareness. Agreed, we have to watch out how far we push this though; which is why we should start with just those patches that appear the most in other sources, carefully meeting our goal over time. Not mimic geek-sources all at once, I believe it took him some time to reach that point... -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 19:38:48 +0100 Markos Chandras hwoar...@gentoo.org wrote: I certainly don't feel safe anymore running non-upstream code in production boxes. You don't run it unless you explicitly tick on that you want experimental functionality _as well as_ the optional features in question; as I said earlier on chat, I don't understand your point here. If you don't enable them, genpatches is just like it is before; I'm not sure why the recommendations should change here, especially with vanilla-sources taking a further step away from Gentoo Security and QA. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 01 Jul 2013 14:30:51 -0400 Anthony G. Basile bluen...@gentoo.org wrote: I was going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated. See scripts/checkpatch.pl Yes, I think it wasn't clear from my first post; but the intention was to introduce such option under the Gentoo distribution menu section. I think I forgot to mention that near the end. I'm well aware, which is why we should proceed slowly; since I will have an away period soon, I don't plan to implement this this month and set up the organization before starting anything to ensure we do this properly. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 1, 2013 at 1:56 PM, Tom Wijsman tom...@gentoo.org wrote: On Mon, 1 Jul 2013 19:38:48 +0100 Markos Chandras hwoar...@gentoo.org wrote: I certainly don't feel safe anymore running non-upstream code in production boxes. You don't run it unless you explicitly tick on that you want experimental functionality _as well as_ the optional features in question; as I said earlier on chat, I don't understand your point here. If you don't enable them, genpatches is just like it is before; I'm not sure why the recommendations should change here, especially with vanilla-sources taking a further step away from Gentoo Security and QA. Tom, I think the point was well-made by grehkh. If the patchset patches the kernel's core, it doesn't matter what CONFIG_* option is set the core kernel code _has_now_been_changed_. This is the crux of the argument, I believe. AUFS simply being one example of this. I'm sure there are others. -- Matthew W. Summers Gentoo Foundation Inc. GPG: 111B C438 35FA EDB5 B5D3 736F 45EE 5DC0 0878 9D46
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. So be very careful here, you are messing with things that are rejected by upstream. greg k-h
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote: Tom, you already know my opinion because we discussed it. I'm all for it. Just a reminder: there's always problems somewhere in the kernel which can be triggered by various options. The kernel is not one big take it or leave it chunk of code, but many chunks selectable by Kconfig with the exception of course of the core. The best we can do wrt to BFQ and other risky patches is mark these options as EXPERIMENTAL. I was going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated. See scripts/checkpatch.pl Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580; It's flat out gone now in the 3.10 kernel release, so if you use it, your code just will never be enabled. greg k-h
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 14:09:57 -0500 Matthew Summers quantumsumm...@gentoo.org wrote: I think the point was well-made by grehkh. You missed my response to that point. If the patchset patches the kernel's core, it doesn't matter what CONFIG_* option is set the core kernel code _has_now_been_changed_. This is the crux of the argument, I believe. AUFS simply being one example of this. I'm sure there are others. As per my response to that point, this statement is no longer true. Let me re-iterate it here: Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 12:23:24 -0700 Greg KH gre...@gentoo.org wrote: Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. So be very careful here, you are messing with things that are rejected by upstream. It sounds to me like some of those developers don't care about providing an option yet; because they would introduce it as an afterthought, based on its need and not just because they can. I don't see why this would be problematic, embedding it in config checks as well as checking whether the patch introduces non-conditional code are trivial to implement; why do you think this is not always true? Thank you for making us aware, feedback on this is nice. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 01, 2013 at 09:25:42PM +0200, Tom Wijsman wrote: On Mon, 1 Jul 2013 14:09:57 -0500 Matthew Summers quantumsumm...@gentoo.org wrote: If the patchset patches the kernel's core, it doesn't matter what CONFIG_* option is set the core kernel code _has_now_been_changed_. This is the crux of the argument, I believe. AUFS simply being one example of this. I'm sure there are others. As per my response to that point, this statement is no longer true. Let me re-iterate it here: Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Wonderful, now you are maintaining a patch that looks nothing like the one created by the original developers, nor tested by anyone else other than gentoo developers. There's a reason that no other distro does this. Playing fast-and-loose with kernel patches is a fun thing to do, but really, why? Users love doing this type of thing, but the interactions of different kernel patches with core subsystems is almost always a non-trivial thing. I'm not saying not to do this, but consider this a friendly warning that this is going to be a MAJOR pain to maintain and debug over the long-run. But hey, what do I know? It's not like I've ever done this before and had the experience of the resulting fall-out that took years to recover from on user's production systems, causing a number of enterprise Linux companies to swear that they would never do this type of thing again... Personally, I wish you luck, it will push the sane users to the vanilla-sources tree, which I strongly encourage :) greg kids, get off my lawn! k-h
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 12:24:36 -0700 Greg KH gre...@gentoo.org wrote: On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote: Tom, you already know my opinion because we discussed it. I'm all for it. Just a reminder: there's always problems somewhere in the kernel which can be triggered by various options. The kernel is not one big take it or leave it chunk of code, but many chunks selectable by Kconfig with the exception of course of the core. The best we can do wrt to BFQ and other risky patches is mark these options as EXPERIMENTAL. I was going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated. See scripts/checkpatch.pl Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580; It's flat out gone now in the 3.10 kernel release, so if you use it, your code just will never be enabled. As I replied earlier and tried to make clear in my first post, this would be our own config variable; naming it CONFIG_GENTOO_EXPERIMENTAL for that matter as to not confuse people. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 12:33:30 -0700 Greg KH gre...@gentoo.org wrote: On Mon, Jul 01, 2013 at 09:25:42PM +0200, Tom Wijsman wrote: On Mon, 1 Jul 2013 14:09:57 -0500 Matthew Summers quantumsumm...@gentoo.org wrote: If the patchset patches the kernel's core, it doesn't matter what CONFIG_* option is set the core kernel code _has_now_been_changed_. This is the crux of the argument, I believe. AUFS simply being one example of this. I'm sure there are others. As per my response to that point, this statement is no longer true. Let me re-iterate it here: Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Wonderful, now you are maintaining a patch that looks nothing like the one created by the original developers, nor tested by anyone else other than gentoo developers. I can convert the original developer's patch whenever he updates it; or on top of that, write a script to generate the original patch back. Playing fast-and-loose with kernel patches is a fun thing to do, but really, why? Users love doing this type of thing, but the interactions of different kernel patches with core subsystems is almost always a non-trivial thing. Our main intent is to provide them and deduplicate work; if users have a problem with one of the patches and it isn't clear to us, we can forward them to the authors. Nothing in this proposal inherits that we must support the patches; especially not, since they are experimental. I'm not saying not to do this, but consider this a friendly warning that this is going to be a MAJOR pain to maintain and debug over the long-run. Maybe, maybe not; plan is to do a slow start, congestion avoidance. :) But hey, what do I know? It's not like I've ever done this before and had the experience of the resulting fall-out that took years to recover from on user's production systems, causing a number of enterprise Linux companies to swear that they would never do this type of thing again... I covered this in the conditions as well as the F.A.Q.; feel free to read about it again, this does not affect them unless they explicitly want them to enable CONFIG_GENTOO_EXPERIMENTAL which help includes a warning; thinking it through, we might even suffix -exp to version. Personally, I wish you luck, it will push the sane users to the vanilla-sources tree, which I strongly encourage :) I will push them to keep CONFIG_GENTOO_EXPERIMENTAL disabled. :) greg kids, get off my lawn! k-h Gentoo (Penguin) and Larry do not necessarily like grass; they might like ice, fire, water, earth, ... whatever their appetite craves. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. -- Fabio Erculiani
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
El lun, 01-07-2013 a las 21:55 +0200, Fabio Erculiani escribió: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. I don't see them exclusionary, look different issues to me :/ (with completely different solutions I think)
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 1, 2013 at 9:59 PM, Pacho Ramos pa...@gentoo.org wrote: El lun, 01-07-2013 a las 21:55 +0200, Fabio Erculiani escribió: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. I don't see them exclusionary, look different issues to me :/ (with completely different solutions I think) Of course I'm not saying that they're not orthogonal. OTOH I believe that the complexity of supporting something like this (the original proposal) is not linear. However, I don't see any problem with people implementing it btw... it's their time. -- Fabio Erculiani
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 21:55:23 +0200 Fabio Erculiani lx...@gentoo.org wrote: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. There's been a start on this, but we're looking at you now; feel free to communicate with Zero_Chaos about this to clarify why and how you do certain matters, make sure (just ping me on IRC after a conversation with him or so) I can follow along so I can more easily bring this into the Portage tree; as at least someone on the kernel herd needs to understand what is going on with all of that There is a bug filed for this with some work from him. https://bugs.gentoo.org/show_bug.cgi?id=472906 Separate from this whole thread; we definitely have some thoughts about this in mind, and I'll give you one cool thought: Having the kernel build the sources can allow the kernel to automatically rebuild out of kernel modules; eg. `emerge sys-kernel/gentoo-kernel` and it will rebuild nvidia-drivers, virtualbox, ... Of course, for this to work we would need to propagate sub slot rebuilds through virtuals. We thinked through some possible problems and solutions that appear; but I would like to keep them out of this ML thread; as to not mix a thought out idea with one that does some early circles in our minds. On a side note; I'm currently testing genpatches using an ebuild I hacked together myself, allows me to test the way eclass does the patches as well as include some neat local QA checks and testing magic. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 1 July 2013 20:09, Matthew Summers quantumsumm...@gentoo.org wrote: On Mon, Jul 1, 2013 at 1:56 PM, Tom Wijsman tom...@gentoo.org wrote: On Mon, 1 Jul 2013 19:38:48 +0100 Markos Chandras hwoar...@gentoo.org wrote: I certainly don't feel safe anymore running non-upstream code in production boxes. You don't run it unless you explicitly tick on that you want experimental functionality _as well as_ the optional features in question; as I said earlier on chat, I don't understand your point here. If you don't enable them, genpatches is just like it is before; I'm not sure why the recommendations should change here, especially with vanilla-sources taking a further step away from Gentoo Security and QA. Tom, I think the point was well-made by grehkh. If the patchset patches the kernel's core, it doesn't matter what CONFIG_* option is set the core kernel code _has_now_been_changed_. This is the crux of the argument, I believe. AUFS simply being one example of this. I'm sure there are others. -- Matthew W. Summers Gentoo Foundation Inc. GPG: 111B C438 35FA EDB5 B5D3 736F 45EE 5DC0 0878 9D46 And besides that, I am sure that 98% of our users out there do not know they run a (heavily?) modified upstream kernel when they emerge the official/supported gentoo-sources. The transition between the minimal genpatches to the new-shiny-feature-full was made behind the scenes. This should have been communicated earlier in time. If you ask me, I would prefer if you apply all the 3rd-party patches conditionally (use flag?, maybe a new gentoo-sources-ng ebuild?) It's really scary to have the BFQ in a stable gentoo-sources ebuild. -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
2013/7/1 Fabio Erculiani lx...@gentoo.org: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. +1 The binary use flag for sys-kernel/*-sources in Funtoo implements exactly that. -- Fabio Erculiani -- Christoph Junghans http://dev.gentoo.org/~ottxor/
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/01/2013 03:55 PM, Fabio Erculiani wrote: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. ++ - -ZC -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR0eW1AAoJEKXdFCfdEflKK4EP/jHyQsECDwVc27eiCglQQ5vV i+FP/cLYkzmNkH8AQwHFq37TVD00XWRiW4vTLu8FrEyu2EIUOze3IwDS9z988rhm TAOcCkoZPF0rA3NoRmapzccSwXMC6W8wOChXSXvRZx+FqYc2kOJ+Yjc7w552ISg3 DQbJlyS/HvspMfAa/zHQYUb2xDX/1HzH1qIdWCBVHo1RpfkmRoblkQ41e7lKnyfA 8oot1XGNc/8jTL5s1rRtpwt0R8Xw2gIEMjSK93B3QLSAzHVkpxmAQ4uELi5m5R5O l+07jit+vqw/cS3/2mSP5nJg8SfOIXNSHLexErUGYg92JsebrlqwmhZjYIHF+ldC h5CbRO30CrcF7kjAjP77gfp0r9tw0AAzOxBqB8cP7/YB62Ee7yAJjXOST7ip7KA5 Bvua4PkgUtyL2fs0orm/Vrkg5ckOVdP92E5oYKFh4gM7UeuDDN2ZJ7fuEKNL9pRC KyCIf3MkQKw18RxO+ecgJlxVN3aj8Qkt4sdJUCZWpiSpry77gVrnciORpHLguKVh yVqhDVkd9b3u2jsnlWs6SeqBQtsChx+xI3OK1lA8kEzNArF6yQxhFxrMeyLUu2Iq LWgBWTuilVIxPAiwDpiWOrXSMjWtjvYGH0vxmfX8wOlvx/L7PA+KEYv+jQeHo59m B1M0btSM6Ut5RJzeS7K8 =cRF9 -END PGP SIGNATURE-
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 1, 2013 at 10:14 PM, Markos Chandras hwoar...@gentoo.org wrote: [...] It's really scary to have the BFQ in a stable gentoo-sources ebuild. BFQ is not that scary, it's just an iosched (and it's quite easy to write an iosched), what could possibly go wrong? Jokes apart, I've been using it in production for almost 2 years now (can't remember exactly). -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang -- Fabio Erculiani
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 1, 2013 at 10:24 PM, Christoph Junghans ott...@gentoo.org wrote: 2013/7/1 Fabio Erculiani lx...@gentoo.org: I believe that end users would benefit more from kernel binary ebuilds (ebuilds building an actual kernel with an official config), rather than all this. +1 The binary use flag for sys-kernel/*-sources in Funtoo implements exactly that. [OT] And why should you throw kernel sources at people as well? Separate pkgs is much better. I don't want to have kernel sources on my servers (for the same reason I don't want to have a compiler, and I've been able to sort this out as well). [/OT] Sorry for the OT. -- Fabio Erculiani -- Christoph Junghans http://dev.gentoo.org/~ottxor/ -- Fabio Erculiani
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 21:14:45 +0100 Markos Chandras hwoar...@gentoo.org wrote: And besides that, I am sure that 98% of our users out there do not know they run a (heavily?) modified upstream kernel when they emerge the official/supported gentoo-sources. This point I do understand. The transition between the minimal genpatches to the new-shiny-feature-full was made behind the scenes. This should have been communicated earlier in time. Apart from the BFQ test in ~3 versions, there did not really happen a transition yet; unless you consider fbcondecor to be part of that, ... If you ask me, I would prefer if you apply all the 3rd-party patches conditionally (use flag?, maybe a new gentoo-sources-ng ebuild?) It's really scary to have the BFQ in a stable gentoo-sources ebuild. An USE flag sounds like something that would make sense; similarly, someone suggested in a mail to cover this as well under a new set in genpatches; currently we do base (Linux patches + fixes) and extra, we could extend this with experimental or something like that. An experimental USE flag definitely makes sense; and documents where users can find the introduced CONFIG_GENTOO_EXPERIMENTAL kernel option in its USE flag description, as well as a page documenting the patches present in the kernel as well as where they can be found in menuconfig. In the long end, this documentation could be generated from the patches. So, we introduce USE=-experimental in the eclass which users would have to enable to get these patches to even apply; that way users can still see and use a completely untampered kernel, in case they want to want to apply patches that fail against all the experimental patches. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( I'm not sure what you mean by distro-specific, but suppose people want BFQ? Why can't we have it in gentoo-sources. It is totally disabled by not selecting CONFIG_BFQ. Selecting it is no different than emerging pf-sources with the same other options ported over. By your logic, we should not distribut pf-sources either. The truth of the matter is, there are forks of the vanilla kernel out there. Are you suggesting we distribute none of them? NOTE: hardened-sources is its own world. There is not level of turning on/off options that get you back to a vanilla kernel. which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. So be very careful here, you are messing with things that are rejected by upstream. greg k-h -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 03:24 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 02:30:51PM -0400, Anthony G. Basile wrote: Tom, you already know my opinion because we discussed it. I'm all for it. Just a reminder: there's always problems somewhere in the kernel which can be triggered by various options. The kernel is not one big take it or leave it chunk of code, but many chunks selectable by Kconfig with the exception of course of the core. The best we can do wrt to BFQ and other risky patches is mark these options as EXPERIMENTAL. I was going to say depend on CONFIG_EXPERIMENTAL in Kconfig, but this is deprecated. See scripts/checkpatch.pl Use of CONFIG_EXPERIMENTAL is deprecated. For alternatives, see https://lkml.org/lkml/2012/10/23/580; It's flat out gone now in the 3.10 kernel release, so if you use it, your code just will never be enabled. greg k-h Yeah i noticed that right after sending it. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 04:25 PM, Fabio Erculiani wrote: On Mon, Jul 1, 2013 at 10:14 PM, Markos Chandras hwoar...@gentoo.org wrote: [...] It's really scary to have the BFQ in a stable gentoo-sources ebuild. BFQ is not that scary, it's just an iosched (and it's quite easy to write an iosched), what could possibly go wrong? Jokes apart, I've been using it in production for almost 2 years now (can't remember exactly). I'm pretty sure I hit a genuine deadlock with it. I've been trying to reproduce with debugging on but nothing yet. But, having said that: BFQ [Experimtental] This introduced an experimental io scheduler. Have fun with it, but don't dare use it in production else we will laugh. -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 01, 2013 at 05:17:07PM -0400, Anthony G. Basile wrote: On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( I'm not sure what you mean by distro-specific, See later mention of CONFIG_GENTOO_EXPERIMENTAL, that is what I was referring to. but suppose people want BFQ? Why can't we have it in gentoo-sources. It is totally disabled by not selecting CONFIG_BFQ. Selecting it is no different than emerging pf-sources with the same other options ported over. Until you run into a patch that modifies code outside of it's CONFIG_ option, like the aufs example I pointed out. By your logic, we should not distribut pf-sources either. The truth of the matter is, there are forks of the vanilla kernel out there. Are you suggesting we distribute none of them? That's a total false argument, the discussion here is about our main gentoo-kernel tree, not one of our many domain-specific kernel versions that are maintained separately. NOTE: hardened-sources is its own world. There is not level of turning on/off options that get you back to a vanilla kernel. Agreed, which keeps that from being merged into this tree, hopefully :) thanks, greg k-h
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 1, 2013 at 11:26 PM, Anthony G. Basile bluen...@gentoo.org wrote: I'm pretty sure I hit a genuine deadlock with it. I've been trying to reproduce with debugging on but nothing yet. But, having said that: BFQ [Experimtental] This introduced an experimental io scheduler. Have fun with it, but don't dare use it in production else we will laugh. Don't trust outdated documentation. http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php There is nothing about it in the latest BFQ patchset. -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA -- Fabio Erculiani
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 05:24 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 05:17:07PM -0400, Anthony G. Basile wrote: On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( I'm not sure what you mean by distro-specific, See later mention of CONFIG_GENTOO_EXPERIMENTAL, that is what I was referring to. but suppose people want BFQ? Why can't we have it in gentoo-sources. It is totally disabled by not selecting CONFIG_BFQ. Selecting it is no different than emerging pf-sources with the same other options ported over. Until you run into a patch that modifies code outside of it's CONFIG_ option, like the aufs example I pointed out. Yeah, that's the situation with hardened-sources and then we are in agreement. If its orthogonal to the rest of the kernel, I maintain that it can safely be included with the appropriate warnings. By your logic, we should not distribut pf-sources either. The truth of the matter is, there are forks of the vanilla kernel out there. Are you suggesting we distribute none of them? That's a total false argument, the discussion here is about our main gentoo-kernel tree, not one of our many domain-specific kernel versions that are maintained separately. Now I'm confused because gentoo-sources is gentoo specific. It contains stuff that we need in gentoo but other distros do not need, like our end-to-end support for certain xattr namespaces. If you remove these then we must either 1) maintain a userland which is not in line with other distros or 2) give up on critical features we want in gentoo, like markings on elf object in user.pax.flags and certain caps, as well as in the future preserving selinux labels through emerge. Upstream will not accept them because of who needs that crap and we can't give them up without loosing core functionality. Feel free to review those patches but don't ask us to drop them from gentoo-sources because their not in upstream. Only vanilla-sources should be exactly that. upstream vanilla with nothing else. period. NOTE: hardened-sources is its own world. There is not level of turning on/off options that get you back to a vanilla kernel. Agreed, which keeps that from being merged into this tree, hopefully :) Yeah I think everyone is in agreement with that. But it also fits my point about orthogonality above. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 05:30 PM, Fabio Erculiani wrote: On Mon, Jul 1, 2013 at 11:26 PM, Anthony G. Basile bluen...@gentoo.org wrote: I'm pretty sure I hit a genuine deadlock with it. I've been trying to reproduce with debugging on but nothing yet. But, having said that: BFQ [Experimtental] This introduced an experimental io scheduler. Have fun with it, but don't dare use it in production else we will laugh. Don't trust outdated documentation. http://algo.ing.unimo.it/people/paolo/disk_sched/sources.php There is nothing about it in the latest BFQ patchset. No, I hit it. Not read about it. Hit it. I can't be 100% sure, but next time it happens, I will be ready with the smoking gun. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, 1 Jul 2013 14:24:54 -0700 Greg KH gre...@gentoo.org wrote: but suppose people want BFQ? Why can't we have it in gentoo-sources. It is totally disabled by not selecting CONFIG_BFQ. Selecting it is no different than emerging pf-sources with the same other options ported over. Until you run into a patch that modifies code outside of it's CONFIG_ option, like the aufs example I pointed out. It would be policy to not add such patches, unless wrapped with config checks by a script; further more, I discussed USE=-experimental with mpagano and he found this separation a good idea, we can split this into a third experimental tarball to not surprise non-Gentoo users as well. mpagano as well as I stand completely behind that gentoo-sources must remain usable for production servers; which this USE flag fulfills, as well as separate from all of this to use live ebuilds in our testing to avoid surprises that even our non-experimental genpatches could bring. (For those in #gentoo-kernel, that conversation happened there) -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? I am accquainted with the developer of a stackable filesystem developer. According to what he has told me in person offline, the developers on the LKML cannot decide on how a stackable filesystem should be implemented. I was told three different variations on the design that some people liked and others didn't, which ultimately kept the upstream kernel from adopting anything. I specifically recall two variations, which were doing it as part of the VFS and doing it as part of ext4. If you want to criticize stackable filesystems, would you lay out a groundwork for getting one implemented upon which people will agree? The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. I was under the impression that there were several reasons for patches not being merged upstream: 1. Lack of signed-off 2. Code drop that no one will maintain 3. Subsystem maintainers saying no simply because they do not like insert non-technical reason here. 4. Risk of patent trolls 5. Actual technical reasons So be very careful here, you are messing with things that are rejected by upstream. greg k-h Only some of the patches were rejected. Others were never submitted. The PaX/GrSecurity developers prefer their code to stay out-of-tree. As one of the people hacking on ZFSOnLinux, I prefer that the code be out-of-tree. That is because fixes for other filesystems are either held back by a lack of system kernel updates or held hostage by regressions in newer kernels on certain hardware. With that said, being in Linus' tree does not make code fall under some golden standard for quality. There are many significant issues in code committed to Linus' the kernel, some of which have been problems for years. Just to name a few: 1. Doing `rm -r /dir` on a directory tree containing millions of inodes (e.g. ccache) on an ext4 filesystem mounted with discard with the CFQ IO elevator will cause a system to hang for hours on pre-SATA 3.1 hardware. This is because TRIM is a non-queued command and is being interleaved with writes for fairness. Incidentally, using noop turns a multiple hour hang into a laggy experience of a few minutes. 2. aio_sync() is unimplemented, which means that there is no sane way for userland software like QEMU and TGT to be both fast and guarantee data integrity. A single crash and your guest is corrupted. It would have been better had AIO never been implemented. 3. dm-crypt will reorder write requests across flushes. That is because upon seeing a write, it sends it to a work queue to be processed asynchronously and upon seeing a flush, it immediately processes it. A single kernel panic or sudden power loss can damage filesystems stored on it. 4. Under low memory conditions with hundreds of concurrent threads (e.g. package builds), every thread will enter direct reclaim and there will be a remarkable drop in system throughput, assuming that the system does not lockup. There is a fairly substantial amount of time wasted after one thread finishes direct reclaim in other threads because they will still be performing direct reclaim afterward. 5. The Linux 3.7 nouveau rewrite broke kexec support. The graphics hardware will not reinitialize properly. 6. A throttle mechanism introduced for memory cgroups can cause the system to deadlock whenever it is holding a lock needed for swap and enters direct reclaim with a significant number of dirty pages.
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 09:36 PM, Richard Yao wrote: On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? I am accquainted with the developer of a stackable filesystem developer. I should probably proofread multiple times before I send emails. Anyway, that should have been: I am acquainted with the developer of a stackable filesystem. According to what he has told me in person offline, the developers on the LKML cannot decide on how a stackable filesystem should be implemented. I was told three different variations on the design that some people liked and others didn't, which ultimately kept the upstream kernel from adopting anything. I specifically recall two variations, which were doing it as part of the VFS and doing it as part of ext4. If you want to criticize stackable filesystems, would you lay out a groundwork for getting one implemented upon which people will agree? The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. I was under the impression that there were several reasons for patches not being merged upstream: 1. Lack of signed-off 2. Code drop that no one will maintain 3. Subsystem maintainers saying no simply because they do not like insert non-technical reason here. 4. Risk of patent trolls 5. Actual technical reasons So be very careful here, you are messing with things that are rejected by upstream. greg k-h Only some of the patches were rejected. Others were never submitted. The PaX/GrSecurity developers prefer their code to stay out-of-tree. As one of the people hacking on ZFSOnLinux, I prefer that the code be out-of-tree. That is because fixes for other filesystems are either held back by a lack of system kernel updates or held hostage by regressions in newer kernels on certain hardware. With that said, being in Linus' tree does not make code fall under some golden standard for quality. There are many significant issues in code committed to Linus' the kernel, some of which have been problems for years. Just to name a few: 1. Doing `rm -r /dir` on a directory tree containing millions of inodes (e.g. ccache) on an ext4 filesystem mounted with discard with the CFQ IO elevator will cause a system to hang for hours on pre-SATA 3.1 hardware. This is because TRIM is a non-queued command and is being interleaved with writes for fairness. Incidentally, using noop turns a multiple hour hang into a laggy experience of a few minutes. 2. aio_sync() is unimplemented, which means that there is no sane way for userland software like QEMU and TGT to be both fast and guarantee data integrity. A single crash and your guest is corrupted. It would have been better had AIO never been implemented. 3. dm-crypt will reorder write requests across flushes. That is because upon seeing a write, it sends it to a work queue to be processed asynchronously and upon seeing a flush, it immediately processes it. A single kernel panic or sudden power loss can damage filesystems stored on it. 4. Under low memory conditions with hundreds of concurrent threads (e.g. package builds), every thread will enter direct reclaim and there will be a remarkable drop in system throughput, assuming that the system does not lockup. There is a fairly substantial amount of time wasted after one thread finishes direct reclaim in other threads because they will still be performing direct reclaim afterward. 5. The Linux 3.7 nouveau rewrite broke kexec
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On Mon, Jul 01, 2013 at 09:36:21PM -0400, Richard Yao wrote: On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? I am accquainted with the developer of a stackable filesystem developer. According to what he has told me in person offline, the developers on the LKML cannot decide on how a stackable filesystem should be implemented. I was told three different variations on the design that some people liked and others didn't, which ultimately kept the upstream kernel from adopting anything. I specifically recall two variations, which were doing it as part of the VFS and doing it as part of ext4. If you want to criticize stackable filesystems, would you lay out a groundwork for getting one implemented upon which people will agree? I was not criticising stackable filesystems at all, nor the aufs code, which I personally run on some of my own systems. I agree that the upstream kernel development of stackable filesystems has been all over the place, with seemingly not much guidance on how to get things merged properly. Being that I am not part of the subset of the kernel development community, I am in no position to lay any groundwork out at all. And note, this is totally off-topic from this thread. My main point is that if Gentoo wants to start maintaining out-of-tree kernel patches, and modifying them, the maintenance nightmare is going to be huge. Been there, done that, have way too many t-shirts commemorating it, and never want to do it again. The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. I was under the impression that there were several reasons for patches not being merged upstream: 1. Lack of signed-off No distro will accept code that does not have a signed-off-by line, otherwise they might get into trouble, as that implies the patch is not properly licensed, right? 2. Code drop that no one will maintain Agreed. 3. Subsystem maintainers saying no simply because they do not like insert non-technical reason here. That is very rare and usually the subsystem maintainer can be poked to resolve this. 4. Risk of patent trolls I know of no kernel patches outside of the tree because of this, do you? 5. Actual technical reasons That's the majority of the reasons patches aren't accepted. Only some of the patches were rejected. Others were never submitted. The PaX/GrSecurity developers prefer their code to stay out-of-tree. As one of the people hacking on ZFSOnLinux, I prefer that the code be out-of-tree. There's also a small legal issue with the zfs code that might prevent it from being merged :) That is because fixes for other filesystems are either held back by a lack of system kernel updates or held hostage by regressions in newer kernels on certain hardware. I have never heard this being a reason for keeping code upstream, what do you mean by it? With that said, being in Linus' tree does not make code fall under some golden standard for quality. There are many significant issues in code committed to Linus' the kernel, some of which have been problems for years. Just to name a few: bug reports snipped The fact that bugs happen in 16 million lines of kernel code, moving at a rate that is orders of magnitude faster than any other software development project, is not news to anyone, it's a fact of life. The fact that we fix them when they are found is the important thing, don't you agree? And of
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 09:56 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 09:36:21PM -0400, Richard Yao wrote: On 07/01/2013 03:23 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 08:45:16PM +0200, Tom Wijsman wrote: Q: What about my stable server? I really don't want to run this stuff! A: These options would depend on !CONFIG_VANILLA or CONFIG_EXPERIMENTAL What is CONFIG_VANILLA? I don't see that in the upstream kernel tree at all. CONFIG_EXPERIMENTAL is now gone from upstream, so you are going to have a problem with this. Earlier I mentioned 2) These feature should depend on a non-vanilla / experimental option. which is an option we would introduce under the Gentoo distribution menu section. Distro-specific config options, great :( which would be disabled by default, therefore if you keep this option the way it is on your stable server; it won't affect you. Not always true. Look at aufs as an example. It patches the core kernel code in ways that are _not_ accepted upstream yet. Now you all are running that modified code, even if you don't want aufs. Earlier I mentioned 3) The patch should not affect the build by default.; if it does, we have to adjust it to not do that, this is something that can be easily scripted. It's just a matter of embedding each + block in the diff with a config check and updating the counts. Look at aufs as a specific example of why you can't do that, otherwise, don't you think that the aufs developer(s) wouldn't have done so? I am accquainted with the developer of a stackable filesystem developer. According to what he has told me in person offline, the developers on the LKML cannot decide on how a stackable filesystem should be implemented. I was told three different variations on the design that some people liked and others didn't, which ultimately kept the upstream kernel from adopting anything. I specifically recall two variations, which were doing it as part of the VFS and doing it as part of ext4. If you want to criticize stackable filesystems, would you lay out a groundwork for getting one implemented upon which people will agree? I was not criticising stackable filesystems at all, nor the aufs code, which I personally run on some of my own systems. I agree that the upstream kernel development of stackable filesystems has been all over the place, with seemingly not much guidance on how to get things merged properly. Being that I am not part of the subset of the kernel development community, I am in no position to lay any groundwork out at all. And note, this is totally off-topic from this thread. My main point is that if Gentoo wants to start maintaining out-of-tree kernel patches, and modifying them, the maintenance nightmare is going to be huge. Been there, done that, have way too many t-shirts commemorating it, and never want to do it again. The goal of don't touch any other kernel code is a very good one, but not always true for these huge out-of-tree kernel patches. Usually that is the main reason why these patches aren't merged upstream, because those changes are not acceptable. I was under the impression that there were several reasons for patches not being merged upstream: 1. Lack of signed-off No distro will accept code that does not have a signed-off-by line, otherwise they might get into trouble, as that implies the patch is not properly licensed, right? That is wrong. Signed-off is an affirmation that the code is under the license, but the absence of signed-off does not imply that the code is not under the license. For example, I doubt you will receive signed-off for PaX/GrSecurity, but is there any doubt that code is under the GPL? To give another example, I doubt that you will receive signed off for code from BSD UNIX, but is there any doubt that code is under the 3-clause BSD license? 2. Code drop that no one will maintain Agreed. 3. Subsystem maintainers saying no simply because they do not like insert non-technical reason here. That is very rare and usually the subsystem maintainer can be poked to resolve this. Past conversations with you have made me reluctant to try, but the next time I see something like this, I will send you an email. 4. Risk of patent trolls I know of no kernel patches outside of the tree because of this, do you? There is compatibility code for DOS long filenames in fat that is no longer in-tree because of this. 5. Actual technical reasons That's the majority of the reasons patches aren't accepted. Not all patches that aren't accepted are submitted to be rejected. Only some of the patches were rejected. Others were never submitted. The PaX/GrSecurity developers prefer their code to stay out-of-tree. As one of the people hacking on ZFSOnLinux, I prefer that the code be out-of-tree. There's also a small legal issue with the zfs code that might prevent it from being merged :) To summarize an email from Linus, the reason people who want ZFS
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
Furthermore, should the kernel kernel choose to engage that out-of-tree code, my expectation is that its quality will improve as they do testing and write patches. What do you mean by this? I probably should have clarified that there was a typo in that. I meant the kernel team, not the kernel kernel. I was referring to the Gentoo kernel team. I sent an email to the list with the correction, but I neglected to include you on the CC list by mistake. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: [gentoo-kernel] Proper distribution integration of kernel *-sources, patches and configuration.
On 07/01/2013 11:29 PM, Richard Yao wrote: On 07/01/2013 09:56 PM, Greg KH wrote: On Mon, Jul 01, 2013 at 09:36:21PM -0400, Richard Yao wrote: That is because fixes for other filesystems are either held back by a lack of system kernel updates or held hostage by regressions in newer kernels on certain hardware. I have never heard this being a reason for keeping code upstream, what do you mean by it? This is an issue that end users tend to encounter where changes in a newer kernel break stuff that they need. One example is nouveau kexec support. Another is that the nouveau in the first two RCs of Linux 3.7 (if I recall) broke my display completely. I probably should clarify that this issue keeps some users from obtaining bug fixes in key components (e.g. their filesystem). That is why I prefer the situation where code lives out-of-tree and works on a variety of kernels over the situation where it is bundled with the kernel for in-tree builds. signature.asc Description: OpenPGP digital signature
