Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
В Вск, 02/11/2008 в 12:11 -0700, Gordon Malm пишет: You can cry abuse all you want. You FAIL to offer any alternatives or solutions. I'll ask again, how do you detect that you are compiling code destined to be run in-kernel from within gcc without checking for the __KERNEL__ macro? I think we both know the answer. Gordon, as far as I see our linux-mod.eclass builds each kernel module separately and it is possible to modify it to pass CFLAGS you wish for kernel modules only and as such avoid using __KERNEL__ macro for this... -- Peter.
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Monday, November 3, 2008 02:22:12 Peter Volkov wrote:
В Вск, 02/11/2008 в 12:11 -0700, Gordon Malm пишет:
You can cry abuse all you want. You FAIL to offer any alternatives or
solutions. I'll ask again, how do you detect that you are compiling code
destined to be run in-kernel from within gcc without checking for the
__KERNEL__ macro? I think we both know the answer.
Gordon, as far as I see our linux-mod.eclass builds each kernel module
separately and it is possible to modify it to pass CFLAGS you wish for
kernel modules only and as such avoid using __KERNEL__ macro for this...
Peter, with much respect, this doesn't really address the question. Here is
the relevant snippet of hardened's gcc specs:
%(cc1_cpu) %{profile:-p} %{!D__KERNEL__: %{!static: %{!fno-PIC: %{!fno-pic:
%{!shared: %{!nostdlib: %{!nostartfiles: %{!fno-PIE: %{!fno-pie: %{!nopie:
%{!fPIC:
%{!fpic:-fPIE}}} } } } } } } } } %{!nostdlib: %{!fno-stack-protector:
-fstack-protector %{!D_LIBC: %{!D_LIBC_REENTRANT: %{!fno-stack-protector-all:
-fstack-protector-all} } } } } }
As you can see the __KERNEL__ macro is checked to decide which gcc specs to
apply any time hardened's gcc compiles anything. Patching linux-mod.eclass
isn't going to do anything to break our reliance on the __KERNEL__ macro.
What patching linux-mod.eclass *would* allow us to do is not patch distcc to
pass -D__KERNEL__ (which I have already done in the case of USE=hardened,
so really this is solved). The limitations of the linux-mod.eclass patching
approach is that it wouldn't work for any modules that use a standalone build
system (dumb as it that might be) instead of kbuild or any module being
compiled outside of portage (kbuild or not).
The linux-mod.eclass hacking could get pretty ugly. I haven't looked deeply
into it but KBUILD_CFLAGS is not yet defined at the time emake is called that
I can tell, so I don't think it will be as simple as adding something like
KBUILD_CFLAGS=${KBUILD_CFLAGS} -fmy -fflags -fhere. If I'm wrong please
let me know, I'd be grateful. :)
There's potentially a lot of conditionals involved as well as far as what
flags to add for a diversity of ARCHes, what the available out-of-tree
modules want for themselves, etc. Overall, linux-mod.eclass could start to
look really nasty and I think it would take quite awhile (for me alone at
least) to figure out a workable solution, if even possible. When accounting
that this would only address the problem for modules using Kbuild and built
within portage - I don't think its worth it for a partial solution.
Patching KBUILD_CFLAGS in the kernel sources themselves works, but if we don't
do that for every kernel portage we restrict users to using hardened-sources
when running a hardened toolchain w/o the hardened kernel is a perfectly
valid configuration. They wouldn't be able to use a non-gentoo provided
kernel without patching it as well. Plus, we're just *not* going to have
every kernel in the tree have a patch conditional on USE=hardened (ick :).
Best regards,
Gordon Malm (gengor)
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sat, 1 Nov 2008 18:29:03 -0700 Gordon Malm [EMAIL PROTECTED] wrote: You're the one assuming the only purpose would be to mask parallel make problems. Apparently it does have a purpose though since avidemux builds fine in parallel but NOT via distcc. Have you conclusively established that they do build fine in parallel? If so, how? And why do they break in parallel only under distcc? Given how distcc works, this strikes me as somewhat implausible... Back to your most recent post *If* there's a legitimate use for RESTRICT=distcc then I am entirely in favour of it. But it looks like there isn't, with every issue being either a parallelism issue (which RESTRICT=distcc won't fix), a user configuration issue (which RESTRICT=distcc won't fix) or a hardened toolchain bug (for which RESTRICT=distcc is massive overkill, and thus the wrong solution). You've decided upon a solution before you've worked out what the problem is. You assumed it is a parallelism issue that people are trying to solve. I haven't pointed to any user configuration issues. Using RESTRICT=distcc on kernel modules is hardly overkill. This isn't openoffice. I know exactly what the problem is, but since you have such a better grasp on it RESTRICT=distcc on kernel modules is unnecessary for the large proportion of users who don't use hardened. RESTRICTs can't be set dependent upon whether or not something like hardenened is enabled; other mechanisms are available that can be. So why aren't you considering those other mechanisms? We have to build using different specs sets for kernel code than userland. If we can't depend on the __KERNEL__ macro for detection, how else do you propose one detect if gcc is building kernel code or not? A macro is just a macro. All it does is affect the preprocessor stage. Hardened is trying to abuse it for something else entirely, which is why you're encountering problems. What makes it any better than a simple option to RESTRICT distcc? You still appear to be confusing RESTRICT distcc and provide a mechanism for selectively disabling distcc. They are entirely different things. I guess the larger question in all this is why does a third party who has demonstrated his anti-Hardened (and anti-Gentoo) slant on multiple occasions define what goes in our PMS? Uh, that's your answer to technical objections to a proposal? You aren't prepared to defend your proposal on its merits? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sun, 2 Nov 2008 12:11:10 -0700 Gordon Malm [EMAIL PROTECTED] wrote: Have you conclusively established that they do build fine in parallel? If so, how? And why do they break in parallel only under distcc? Given how distcc works, this strikes me as somewhat implausible... Yes it builds in parallel. By compiling it in parallel. I don't know why and don't care to investigate for you. It's not my package, I'm not QA and my team is short staffed enough as it is. We have to take care of our own backyard. Maybe you could investigate the answers to your own questions? How about you at least feign attempt at answering my questions for a change? If you think compiling it in parallel confirms that it builds in parallel, you're reaffirming my growing suspicion that you're entirely wrong about distcc being responsible for anything other than hardened brokenness... RESTRICT=distcc on kernel modules is unnecessary for the large proportion of users who don't use hardened. RESTRICTs can't be set dependent upon whether or not something like hardenened is enabled; other mechanisms are available that can be. So why aren't you considering those other mechanisms? I agree with the first sentence, never said otherwise. Who says I am not considering them? In fact, I have stated that I am. They are just more hackish than adding the ability to RESTRICT distcc, hence my reason for soliciting such a feature. I'm surprised that you'd suggest this after debasing a RESTRICT option on the same grounds. And that brings us to the other thing you don't understand: RESTRICT is a global, system independent, invariant metadata key. Things in RESTRICT are in RESTRICT because they have to be known in situations where those constraints are in effect. Things that do not have to be known at the metadata level shouldn't be in metadata. A macro is just a macro. All it does is affect the preprocessor stage. Hardened is trying to abuse it for something else entirely, which is why you're encountering problems. You can cry abuse all you want. You FAIL to offer any alternatives or solutions. I'll ask again, how do you detect that you are compiling code destined to be run in-kernel from within gcc without checking for the __KERNEL__ macro? I think we both know the answer. I suggest you talk to hardened upstream and see what they have to say about it. What have they offered as justification for using -D in a way in which no-one else uses it? What are their objections to switching to an approach that doesn't break the preprocessor model? Sorry to everyone who e-mailed me who thought RESTRICT=distcc would be somewhat useful. Those people are mistaken. Uh, that's your answer to technical objections to a proposal? You aren't prepared to defend your proposal on its merits? Why are you assuming this point is *at all* related to my proposal? It's not. It's about Gentoo. But I stand corrected, a bunch of people rushed to tell me you don't actually determine anything. ;) So you don't care about whether your solution is right? You are proposing to add a metadata invariant option for a condition that isn't metadata invariant and that, by being made metadata invariant, means it's being disabled for everyone rather than by the small number of users who might need it. In addition, you can't demonstrate any cases where this option is genuinely useful, other than as a workaround for a hardened bug that you haven't contacted upstream about, and when used to work around said hardened bug the scope of the change isn't limited to hardened. You still haven't explained why you don't do something like: broken_hardened_compiler export DISTCC_HOSTS=localhost -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sat, 1 Nov 2008 13:57:17 -0700 Gordon Malm [EMAIL PROTECTED] wrote: But in the case of out-of-tree kernel modules the idea of distributing compile jobs to other machines is fundamentally flawed IMO. Why? And how are out of tree kernel modules in any way special when it comes to distcc? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
If you're compiling an out-of-tree module that requires the kernel be compiled with support for a particular item and the distccd host's kernel does not have that support compiles fail. Reference bug #120001 (though I know that it was properly diagnosed there). Then there's also this doozie. - #167844. Thanks, Gordon Malm (gengor) On Saturday, November 1, 2008 14:00:17 Ciaran McCreesh wrote: On Sat, 1 Nov 2008 13:57:17 -0700 Gordon Malm [EMAIL PROTECTED] wrote: But in the case of out-of-tree kernel modules the idea of distributing compile jobs to other machines is fundamentally flawed IMO. Why? And how are out of tree kernel modules in any way special when it comes to distcc?
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sat, 1 Nov 2008 14:21:43 -0700 Gordon Malm [EMAIL PROTECTED] wrote: If you're compiling an out-of-tree module that requires the kernel be compiled with support for a particular item and the distccd host's kernel does not have that support compiles fail. Reference bug #120001 (though I know that it was properly diagnosed there). Then there's also this doozie. - #167844. So far as I can see, when the correct options are passed to a build system, the only issue is hardened being broken. There doesn't seem to be any fundamental reason why distcc with a sane compiler can't be used with kernel modules. If this is the case, wouldn't it be better to get the hardened compiler fixed? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Saturday, November 1, 2008 14:28:06 Ciaran McCreesh wrote:
On Sat, 1 Nov 2008 14:21:43 -0700
Gordon Malm [EMAIL PROTECTED] wrote:
If you're compiling an out-of-tree module that requires the kernel be
compiled with support for a particular item and the distccd host's
kernel does not have that support compiles fail. Reference bug
#120001 (though I know that it was properly diagnosed there).
Then there's also this doozie. - #167844.
So far as I can see, when the correct options are passed to a build
system, the only issue is hardened being broken. There doesn't seem to
be any fundamental reason why distcc with a sane compiler can't be used
with kernel modules. If this is the case, wouldn't it be better to get
the hardened compiler fixed?
I use madwifi-ng extensively and have experienced the same issue with
madwifi-ng as stated in that bug. For bug #167844, please read comment #13
and http://code.google.com/p/distcc/issues/detail?id=25. There's nothing to
fix, hardened is doing the right thing as is distcc. The proper approach is
to not distribute compiles of kernel modules.
To not get too stuck on a single scenario, I think this would be a useful
feature and desireable vs. the alternative of FEATURES=${FEATURES/distcc/}.
This is not the first time its been requested, a simple search reveals bugs
#28300, #80894.
A quick grep shows these packages at minimum have to do some FEATURES checking
and either pass a config option (if provided) or die disable distcc and try
again.
dev-python/pyqwt
media-gfx/sam2p
media-tv/mythtv
media-video/avidemux
Gordon Malm (gengor)
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sat, 1 Nov 2008 14:58:39 -0700
Gordon Malm [EMAIL PROTECTED] wrote:
I use madwifi-ng extensively and have experienced the same issue with
madwifi-ng as stated in that bug. For bug #167844, please read
comment #13 and http://code.google.com/p/distcc/issues/detail?id=25.
There's nothing to fix, hardened is doing the right thing as is
distcc. The proper approach is to not distribute compiles of kernel
modules.
It looks to me like hardened is doing entirely the wrong thing. Thus,
the proper fix is to make hardened behave itself.
To not get too stuck on a single scenario, I think this would be a
useful feature and desireable vs. the alternative of
FEATURES=${FEATURES/distcc/}.
This is not the first time its been requested, a simple search
reveals bugs #28300, #80894.
It looks a lot like people are blaming distcc for parallelisation bugs
that aren't distcc related but that happen to be easier to reproduce
when distcc is enabled. Do you have any examples of problems that are
definitely caused by distcc, rather than general parallelisation bugs
or user misconfigurations? RESTRICTing distcc doesn't seem to be an
actual fix for anything...
--
Ciaran McCreesh
signature.asc
Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Saturday, November 1, 2008 15:11:16 Ciaran McCreesh wrote:
On Sat, 1 Nov 2008 14:58:39 -0700
Gordon Malm [EMAIL PROTECTED] wrote:
I use madwifi-ng extensively and have experienced the same issue with
madwifi-ng as stated in that bug. For bug #167844, please read
comment #13 and http://code.google.com/p/distcc/issues/detail?id=25.
There's nothing to fix, hardened is doing the right thing as is
distcc. The proper approach is to not distribute compiles of kernel
modules.
It looks to me like hardened is doing entirely the wrong thing. Thus,
the proper fix is to make hardened behave itself.
It looks to me like you've already made up your mind. How is hardened doing
the entirely wrong thing? What do you propose can be done to fix the
hardened compiler? What about madwifi-ng? You are getting increasingly
narrow in your points of objection.
To not get too stuck on a single scenario, I think this would be a
useful feature and desireable vs. the alternative of
FEATURES=${FEATURES/distcc/}.
This is not the first time its been requested, a simple search
reveals bugs #28300, #80894.
It looks a lot like people are blaming distcc for parallelisation bugs
that aren't distcc related but that happen to be easier to reproduce
when distcc is enabled. Do you have any examples of problems that are
definitely caused by distcc, rather than general parallelisation bugs
or user misconfigurations? RESTRICTing distcc doesn't seem to be an
actual fix for anything...
As I said in my first post:
'It is true that RESTRICT=distcc is usually not the proper solution to
problems.'
Parallel building problems can often and should be addressed properly. I
don't want the answer to every one that comes along to be to add
RESTRICT=distcc. This is something to be addressed through developer
documentation that using RESTRICT=distcc should be a last resort.
However, in practice making a package parallel-make safe isn't always trivial.
So what happens in these cases is FEATURES=distcc die check is put in
place killing the emerge chain and requiring user intervention. Either that
or the bug just lingers and the compile fails somewhere in the middle...
I don't know about palaudis but this is like a one line patch to portage. But
silly me, I thought the package manager was there to support the
distribution.
Gordon Malm (gengor)
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Sat, 1 Nov 2008 15:47:09 -0700 Gordon Malm [EMAIL PROTECTED] wrote: It looks to me like hardened is doing entirely the wrong thing. Thus, the proper fix is to make hardened behave itself. It looks to me like you've already made up your mind. How is hardened doing the entirely wrong thing? What do you propose can be done to fix the hardened compiler? What about madwifi-ng? You are getting increasingly narrow in your points of objection. I suggest you get the hardened upstream people to stop abusing the -D switch to gcc. The distcc people suggest the same. Parallel building problems can often and should be addressed properly. I don't want the answer to every one that comes along to be to add RESTRICT=distcc. This is something to be addressed through developer documentation that using RESTRICT=distcc should be a last resort. Uh, RESTRICT=distcc won't even fix parallel make problems. It'll just make them harder to reproduce on some systems. However, in practice making a package parallel-make safe isn't always trivial. So what happens in these cases is FEATURES=distcc die check is put in place killing the emerge chain and requiring user intervention. Either that or the bug just lingers and the compile fails somewhere in the middle... ...or you could use -j1, which whilst being horrible will at least work. I don't know about palaudis but this is like a one line patch to portage. But silly me, I thought the package manager was there to support the distribution. You have yet to demonstrate how RESTRICT=distcc will help. In fact, you seem to be demonstrating that all it'll do is make a few people apply a 'fix' that won't reliably fix anything. *If* there's a legitimate use for RESTRICT=distcc then I am entirely in favour of it. But it looks like there isn't, with every issue being either a parallelism issue (which RESTRICT=distcc won't fix), a user configuration issue (which RESTRICT=distcc won't fix) or a hardened toolchain bug (for which RESTRICT=distcc is massive overkill, and thus the wrong solution). You've decided upon a solution before you've worked out what the problem is. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
Gordon Malm wrote: It looks to me like you've already made up your mind. How is hardened doing the entirely wrong thing? From the page [1] you mentioned: If so, that seems to me like an abuse of the -D option. The abuse is in changing the compiler behavior based on -D options. What do you propose can be done to fix the hardened compiler? From the same page: It would be better for you to remove the patch from gcc where it makes -D__KERNEL__ imply -nossp -nopie, and to instead patch the Linux kernel build system (Makefiles, etc.) so that it passes -D__KERNEL__ -nossp -nopie rather than -D__KERNEL__. [1] http://code.google.com/p/distcc/issues/detail?id=25 Cheers, -jkt -- cd /local/pub more beer /dev/mouth signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Saturday 01 November 2008 20:57:17 Gordon Malm wrote: I'd like to get distcc added as one of the FEATURES we are able to RESTRICT. Regardless of whether it's a good idea or not, does it fix all the known issues if the ebuild sets DISTCC_HOSTS=localhost in the environment?
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
On Saturday, November 1, 2008 15:57:52 Ciaran McCreesh wrote: Parallel building problems can often and should be addressed properly. I don't want the answer to every one that comes along to be to add RESTRICT=distcc. This is something to be addressed through developer documentation that using RESTRICT=distcc should be a last resort. Uh, RESTRICT=distcc won't even fix parallel make problems. It'll just make them harder to reproduce on some systems. Why don't you post the whole context? Here, I'll do it for you: On Saturday, November 1, 2008 15:11:16 Ciaran McCreesh wrote: It looks a lot like people are blaming distcc for parallelisation bugs that aren't distcc related but that happen to be easier to reproduce when distcc is enabled. Do you have any examples of problems that are definitely caused by distcc, rather than general parallelisation bugs or user misconfigurations? RESTRICTing distcc doesn't seem to be an actual fix for anything... And my full response... As I said in my first post: 'It is true that RESTRICT=distcc is usually not the proper solution to problems.' Parallel building problems can often and should be addressed properly. I don't want the answer to every one that comes along to be to add RESTRICT=distcc. This is something to be addressed through developer documentation that using RESTRICT=distcc should be a last resort. You're the one assuming the only purpose would be to mask parallel make problems. Apparently it does have a purpose though since avidemux builds fine in parallel but NOT via distcc. Back to your most recent post *If* there's a legitimate use for RESTRICT=distcc then I am entirely in favour of it. But it looks like there isn't, with every issue being either a parallelism issue (which RESTRICT=distcc won't fix), a user configuration issue (which RESTRICT=distcc won't fix) or a hardened toolchain bug (for which RESTRICT=distcc is massive overkill, and thus the wrong solution). You've decided upon a solution before you've worked out what the problem is. You assumed it is a parallelism issue that people are trying to solve. I haven't pointed to any user configuration issues. Using RESTRICT=distcc on kernel modules is hardly overkill. This isn't openoffice. I know exactly what the problem is, but since you have such a better grasp on it On Saturday, November 1, 2008 15:57:52 Ciaran McCreesh wrote: On Sat, 1 Nov 2008 15:47:09 -0700 Gordon Malm [EMAIL PROTECTED] wrote: It looks to me like hardened is doing entirely the wrong thing. Thus, the proper fix is to make hardened behave itself. It looks to me like you've already made up your mind. How is hardened doing the entirely wrong thing? What do you propose can be done to fix the hardened compiler? What about madwifi-ng? You are getting increasingly narrow in your points of objection. I suggest you get the hardened upstream people to stop abusing the -D switch to gcc. The distcc people suggest the same. On Saturday, November 1, 2008 16:28:17 Jan Kundrát wrote: Gordon Malm wrote: It looks to me like you've already made up your mind. How is hardened doing the entirely wrong thing? From the page [1] you mentioned: If so, that seems to me like an abuse of the -D option. The abuse is in changing the compiler behavior based on -D options. What do you propose can be done to fix the hardened compiler? From the same page: It would be better for you to remove the patch from gcc where it makes -D__KERNEL__ imply -nossp -nopie, and to instead patch the Linux kernel build system (Makefiles, etc.) so that it passes -D__KERNEL__ -nossp -nopie rather than -D__KERNEL__. We have to build using different specs sets for kernel code than userland. If we can't depend on the __KERNEL__ macro for detection, how else do you propose one detect if gcc is building kernel code or not? Patching an out-of-tree module's build system to pass -fno-PIE works on x86, I don't know how this will affect other ARCHes, do either of you? This could potentially get really tricky. If it can't be done nicely in the eclass for every possible kernel release and out-of-tree module, then we get into patching *everything* and having to maintain it forward. So this is just a different workaround, a rather heavy-handed and high-maintenance one at that. What makes it any better than a simple option to RESTRICT distcc? I guess the larger question in all this is why does a third party who has demonstrated his anti-Hardened (and anti-Gentoo) slant on multiple occasions define what goes in our PMS? Gordon Malm (gengor)
Re: [gentoo-dev] [PMS] Add RESTRICT=distcc capability
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gordon Malm wrote: snip a lot of exchanges All the technical discussion on the above is perfectly fine, but the way the arguments are being presented and the tone used by both sides is getting arguments into a thin line from becoming flames. Please step back before turning this into another flame festival. I guess the larger question in all this is why does a third party who has demonstrated his anti-Hardened (and anti-Gentoo) slant on multiple occasions define what goes in our PMS? It's not up to a 3rd party to define what will be or not in PMS. Any 3rd party is free to contribute to the discussion (within boundaries), but it's up to the PMs folks to reach agreements about PMS and, or if they can't, up to the Gentoo Council. Gordon Malm (gengor) - -- Regards, Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org Gentoo- forums / Userrel / Devrel / SPARC / KDE -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkNG6IACgkQcAWygvVEyAKOrgCghGO8W2839jXMaMq9DN3DpWbM JJMAn0PhLDiKoBayr1juI48tzwa8j8Wc =EXnX -END PGP SIGNATURE-
