[gentoo-dev] Packages up for grabs
Dear all, since I dont have the time anymore, I've re-assigned the following packages to maintainer-needed. Please go get them if interested. app-admin/collectd (*) app-arch/arj app-editors/dhex dev-libs/libcaldav dev-libs/libofx dev-util/nvidia-cuda-npp kde-misc/kcollectd media-gfx/argyllcms media-gfx/brlcad (*) media-video/bombono-dvd net-firewall/fwbuilder net-libs/liboping net-wireless/libertas-firmware sci-misc/gcam (*) Collectd and BRLcad have users as proxy maintainers. Cheers, Andreas -- Andreas K. Huettel Gentoo Linux developer kde (ex team lead), sci, tex, arm, printing dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)
On 9/10/2012 10:39 PM, Duncan wrote: Gregory M. Turner posted on Mon, 10 Sep 2012 20:29:53 -0700 as excerpted: However, IIRC, /etc/make.conf is just ignored by portage if /etc/portage/make.conf is present, so symlinking, or even better, if possible, hardlinking those files would probably do the right thing for legacy tools that don't know about the new location... unless I'm mistaken, which is always plausible :) Thanks. Reasonable approach and good to know. Well, I did warn about the likelihood I was wrong :) Pretty clever 'cause that way even when I'm wrong (as turns out to have been the case), I'm still right :P In all seriousness, if both of them are sourced, then could one get away with something like this? /etc/make.conf: source /etc/portage/make.conf /etc/portage/make.conf: if [[ __GENTOO_MAKE_CONF_ONCE == gotit ]] ; then __GENTOO_MAKE_CONF_ONCE=gotit . . . endif or are conditionals disallowed? As Zac mentions, hopefully it's harmless to duplicate things, but, personally, I would worry about the effect of duplicates on performance, and also in PORTDIR_OVERLAY. Plus, it just seems dirty. -gmt
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/11/2012 9:54 AM, Ian Stakenvicius wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 12:43 PM, Zac Medico wrote: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. I personally wonder about why this would be necessary from the perspective of the user; if the user's system at emerge time can handle X concurrent processes per emerge-job , i don't see why it would matter what phase these jobs would be launched from. At the ebuild level, certainly, but that's one of the reasons for EJOBS in the first place, so that it can be overridden consistently within a phase, if necessary for the ebuild (regardless of build system type), right? I would hope so. There are definitely real-world reasons to want to restrict jobs -- usually to just one -- in a particular phase... several ebuilds, notably several in @system, do this already by injecting -j1 somewhere. To say it's all about performance seems to be forgetting a major reason somebody might want to do this as a user or an ebuild author. Hmm... I should preface this by saying that it really pains me to say this, because now this just starts to seem so fucking complicated -- but now that I think about it, this really seems to highlight a kind of semantic discrepancy that this thread has mostly dusted under the rug. There are really TWO things we seem to care about -- one is some kind of global build-parallelism frob-set, and the other is the ability to turn off all parallelism for certain parts of an ebuild and those are actually kind-of orthogonal. In other words, imagine if we had some kind of global parallellism arbiter thingy that looked at EJOBS or ELOADAVG or maybe some other things we haven't thought of yet, and decided things like: o should portage start a new parallel emerge process? o What is the appropriate MAKEOPTS for the emake that foo-1.0.ebuild just issued? or even, one could imagine, such things as: o should portage decide that parallelism has gotten out of control somehow and suspend or kill a running ebuild for subsequent resuming or restarting? o should portage re-nice some of its processes? Note that, effectively, we have this already, and it's called portage. But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. Not sure what I'm suggesting we do about it, exactly, but just pointing out that maybe a completely correct solution requires a much more elaborate implementation than just a bunch of syntactic sugar around what we currently call MAKEOPTS. Whether or not Gentoo wants to take that all on, right now, as part of the next EAPI, is certainly debatable -- in fact, I'm inclined to say maybe it's not the best idea. Perhaps we should really be asking: is the status-quo really so problematic/inelegant that it needs fixing? -- before we decide how best to fix it. -gmt
Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)
On 09/12/2012 02:16 AM, Gregory M. Turner wrote: In all seriousness, if both of them are sourced, then could one get away with something like this? /etc/make.conf: source /etc/portage/make.conf /etc/portage/make.conf: if [[ __GENTOO_MAKE_CONF_ONCE == gotit ]] ; then __GENTOO_MAKE_CONF_ONCE=gotit . . . endif or are conditionals disallowed? Yes, conditionals are not allowed. The parser only supports variable assingments and source commands. As Zac mentions, hopefully it's harmless to duplicate things, but, personally, I would worry about the effect of duplicates on performance, and also in PORTDIR_OVERLAY. Plus, it just seems dirty. I would recommend to use /etc/make.conf alone, until whatever tools you use have been updated to support /etc/portage/make.conf. There's no need for any of these compatibility workarounds that you and Duncan have been suggesting. -- Thanks, Zac
Re: [gentoo-dev] rfc: netplugd and ifplugd support in OpenRc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 06:23 PM, Rich Freeman wrote: On Tue, Sep 11, 2012 at 5:01 PM, William Hubbs willi...@gentoo.org wrote: I can agree that a server would probably want a static configuration, but all work stations do not use gnome, kde, etc. Most do not run unix, but at work I can't think of any servers that are using static configurations. They might be assigned static IPs, but they'll use DHCP just the same. I deploy my network at home in the same way - most of my standing PCs have DNS and static IP assignments, but they still use DHCP. This way I can still utilize PXE for backups/etc, and adjust things at any time fairly easily. Rich Some of my servers have a back-end nic that has static config defined, and i use netplug or ifplugd for that to bring the iface up and down based on link -- link detection != auto-configuration, imo. Suffice to say, networkmanager is not a reasonable means to determine link state for oldnet. Newnet, well, all of that gets handled outside of openrc's config anyhow so... But this is getting a little beyond the original point of the thread.. :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQg74ACgkQ2ugaI38ACPAL5gEArhYn4Zzas8k6ijddwP9F4q+F 95hFTjw9h0lp0nIrrDIA/R6twsmCnPt1T7yYN6jr0ZcjBonZgI+pk3TZS9qnLa/U =dJUq -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 12/09/12 05:55 AM, Gregory M. Turner wrote: Note that, effectively, we have this already, and it's called portage. But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Advanced HPC systems (sys-cluster/torque along with an appropriate scheduler, for instance) can do such things with their jobs when the jobs are properly built; I could see portage being able to handle this as well given most of what is necessary is already known (ebuild phases, build system type (via eclass), etc). However, given the limitations already put on parallelism in terms of emerge order, etc, I could see this solution needing to be -very- complex and integration needing to occur on multiple levels. We'd also need to consider distcc (and other cluster-shared compilation methods if there are any??).. It would be an interesting project, though. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQhwAACgkQ2ugaI38ACPCKrgD+JNlHPUl7ETYDC6u3lYWRSz8J fpWC/puDfCYu51yNOVIA/0E+U6x9Ds8GV8r/RinkTqss3/fcd06w24GRvZOda3Mj =XONZ -END PGP SIGNATURE-
Re: [gentoo-dev] EJOBS variable for EAPI 5?
Il 11/09/2012 18:43, Zac Medico ha scritto: On 09/11/2012 09:36 AM, viv...@gmail.com wrote: Dunno where to place this request, but if we go for something like EJOBS could we also make it phase specific? So compile, install and test could have a different number of jobs running. Possibly three different variables that override a predefined EJOBS. Per-phase sounds a little to fine-grained. Instead, I'd suggest to add an ELOADAVG variable that's analogous to make's --load-average option. That should be enough to compensate for any differences between phases. ok, but in my experience load-average really is too limited so I relaunch with the ability to control the following: - disk io - network - memory - cpu - jobs just tough that being able to control just jobs in a phase specific manner could have been sufficed ;-) Also this seem is a good job for containers, already implemented in the linux kernel, but will let someone with experience with them comment on the mattter.
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Hans signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On Wed, Sep 12, 2012 at 12:33 PM, Hans de Graaff gra...@gentoo.org wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. I've been thinking about this, but that only works until you get to the huge link step of, e.g. chromium, firefox, libreoffice. I've had programs with memory leaks in the past, but I've never seen a program validly consume as much memory as ld during those builds. To cover something like that, you would need to be able to freeze and swap out an entire process (such as ld) to allow something else to complete quickly...but there's no good way I can think of to prioritize sanely between the one big process and the few dozen smaller ones which might be allowed to spawn and complete first. -- :wq
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 09/12/2012 09:33 AM, Hans de Graaff wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Well, I think it's still good to limit the number of jobs at least, since otherwise you could become overloaded with processes that don't consume a lot of memory at first but by the time they complete they have consumed much more memory than desired (using swap). -- Thanks, Zac
Re: [gentoo-dev] EJOBS variable for EAPI 5?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 12/09/12 12:58 PM, Zac Medico wrote: On 09/12/2012 09:33 AM, Hans de Graaff wrote: On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: So essentially what you're saying here is that it might be worthwhile to look into parallelism as a whole and possibly come up with a solution that combines 'emerge --jobs' and build-system parallelism together to maximum benefit? Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the end. Once you hit swap overall throughput has to go down quite a bit. Well, I think it's still good to limit the number of jobs at least, since otherwise you could become overloaded with processes that don't consume a lot of memory at first but by the time they complete they have consumed much more memory than desired (using swap). I think this would need to be dealt with by having the parent emerge process monitor all children and specifically block individual processes (ie, 'make' , 'ld' , etc) once resources are unavailable until they become so. Swap may be hit by the big processes but they wouldn't continue to be processed while in swap, at least. I don't have a solution to the potential 'thrashing' issue, though, which i expect would be a problem even if there's enough memory. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQwFoACgkQ2ugaI38ACPAiAwD/foU8Xw1BQM3jeO6IiVfCGOnw xHtIufwVmMpsGVdJQRIA/3W7Utg92foSc6ZtKMzBP5Fj0qB2BXMt/RS2I4pLsCQm =gy9K -END PGP SIGNATURE-
Re: [gentoo-dev] rfc: netplugd and ifplugd support in OpenRc
El mié, 12-09-2012 a las 08:44 -0400, Ian Stakenvicius escribió: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/09/12 06:23 PM, Rich Freeman wrote: On Tue, Sep 11, 2012 at 5:01 PM, William Hubbs willi...@gentoo.org wrote: I can agree that a server would probably want a static configuration, but all work stations do not use gnome, kde, etc. Most do not run unix, but at work I can't think of any servers that are using static configurations. They might be assigned static IPs, but they'll use DHCP just the same. I deploy my network at home in the same way - most of my standing PCs have DNS and static IP assignments, but they still use DHCP. This way I can still utilize PXE for backups/etc, and adjust things at any time fairly easily. Rich Some of my servers have a back-end nic that has static config defined, and i use netplug or ifplugd for that to bring the iface up and down based on link -- link detection != auto-configuration, imo. Suffice to say, networkmanager is not a reasonable means to determine link state for oldnet. Newnet, well, all of that gets handled outside of openrc's config anyhow so... But this is getting a little beyond the original point of the thread.. :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQg74ACgkQ2ugaI38ACPAL5gEArhYn4Zzas8k6ijddwP9F4q+F 95hFTjw9h0lp0nIrrDIA/R6twsmCnPt1T7yYN6jr0ZcjBonZgI+pk3TZS9qnLa/U =dJUq -END PGP SIGNATURE- Also my father has two machines in his job with static config and ifplugd to detect link up/down. I also remember to migrate years ago from netplugd to ifplugd for some reason I can't remember now, probably ifplugd worked better :/ signature.asc Description: This is a digitally signed message part
[gentoo-dev] About changing security policy to unCC maintainers when their are not needed
Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. What do you think? Thanks for your thoughts signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
On Wed, 12 Sep 2012 19:59:01 +0200 Pacho Ramos pa...@gentoo.org wrote: Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. So you would want to be re-CC'd when it is time to remove the vulnerable versions, I guess. Also, I have problems with stating getting too much mail as the actual problem. Perhaps your brain or your computer can smartly filter them out? Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. You can un-CC yourself. I don't see why security@ should be doing the legwork. jer
[gentoo-dev] Re: About changing security policy to unCC maintainers when their are not needed
On 2012-09-13 03:59, Pacho Ramos wrote: Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. What do you think? Thanks for your thoughts Hello, Is the policy you describe officially documented, or just current behaviour? In KDE and Qt herds for example, we usually just unCC ourselves when we've taken the required action. Best regards, Michael
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
On Wed, Sep 12, 2012 at 2:29 PM, Jeroen Roovers j...@gentoo.org wrote: So you would want to be re-CC'd when it is time to remove the vulnerable versions, I guess. Isn't this done shortly after keywording is complete? I think the concern is more about issuing GLSAs/etc, which apparently can happen months or years after the vulnerable versions were removed judging by recent chromium@ mail. You can un-CC yourself. I don't see why security@ should be doing the legwork. I see no issue with that. Rich
Re: [gentoo-dev] EJOBS variable for EAPI 5?
On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 12/09/12 05:55 AM, Gregory M. Turner wrote:
Note that, effectively, we have this already, and it's called
portage. But one could certainly make a case for modularizing it
better, since, in truth, we are talking about a very common, very
abstract problem here which portage shares with any number of
batch-build systems.
Such an engine could very well do exactly the right thing if it
were faced with a constraint that a certain part of a certain build
needed to proceed without parallelism due to limitations coming
from the build.
Also, there are very large parts of most builds -- configure comes
to mind -- that don't parallelize even if, perhaps, they should.
In such cases, a really smart global parallelism arbiter could
easily respond by spawning more jobs from other builds.
So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?
Yeah, couldn't have said it better myself ... apparently :)
Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc). However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels. We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??).. It would be an interesting project, though.
ACK all of the above.
Tempting to think more deeply about this but probably the last thing I
need to do right now is to talk myself into another speculative project.
I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S
Only a few major sources of parallelism exist in portage: --jobs /
--load-average in emerge opts, multiprocessing eclass equiv. ebuild
helper, distcc, and make... Infrastructure is already in place for all
of those, so perhaps a good holistic solution exists that isn't /too/
complicated.
...OK another f!#!%$^ brainstorm incoming :)
For JOBS syntax... what really seems missing in portage are:
o a clean way to say dont parallelize this particular make
invocation in ebuilds
o a clean way to globally say try to use this parallelization
strategy when emerging.
So what about something like:
o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
--jobs and --load-average emerge options
o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars
o If the latter are not specified, they are copied respectively from
the former (debatable for *_JOBS, since now we get 16 processes when
we asked for four).
o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
if only if -j|--jobs|-l|--load-average options aren't provided in
make.conf/profile/envvar MAKEOPTS
o however, if MAKEOPTS override EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
issue a conspicuous yellow-stars warning
o extend emake to accept a --non-parallel option which will
strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
with support for override in profile.bashrc. Don't warn about this
overriding EBUILD_JOBS -- treat as SOP.
o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
or, perhaps, something like:
EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}
could be used to distinguish between don't use any parallelism
and don't use GNU's make parallelism in emake. Also maybe a
better name exists that doesn't use double-negatives.
?
Seems to me something vaguely like the above would provide
o backward compatibility for ebuilds and make.conf
o not so vastly different than what we have
o a decent way to specify what we really want globally;
insofar as portage doesn't do the best job effecting the requested
parallelization strategy, more ambitious tactics could be
implemented later, hopefully without huge interface revisions.
-gmt
P.S.:
(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into
EBUILD_JOBS when only the former is specified, and then let
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS) but maybe
too much automagic for this to be a good idea.)
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
El mié, 12-09-2012 a las 20:29 +0200, Jeroen Roovers escribió: On Wed, 12 Sep 2012 19:59:01 +0200 Pacho Ramos pa...@gentoo.org wrote: Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. So you would want to be re-CC'd when it is time to remove the vulnerable versions, I guess. Personally, I have never been asked by them to remove old vulnerable versions (and this refers to bugs I get from gnome and dotnet herds) Also, I have problems with stating getting too much mail as the actual problem. The problem is that one and, also, getting a comment months after the fixed version was stabilized with a comment like GLSA vote = no or similar. That comment is only useful to security team. Perhaps your brain or your computer can smartly filter them out? Perhaps things can be enhanced to not send useless mails that will need to get removed just after they are get, this is pretty annoying when I fetch a ton of mails after being out during August. Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. You can un-CC yourself. I don't see why security@ should be doing the legwork. It shouldn't be so hard to do, they can do it just when they CC arches, instead of relaying some random team member to do it himself once a useless message is received jer signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: About changing security policy to unCC maintainers when their are not needed
El jue, 13-09-2012 a las 04:30 +1000, Michael Palimaka escribió: On 2012-09-13 03:59, Pacho Ramos wrote: Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. What do you think? Thanks for your thoughts Hello, Is the policy you describe officially documented, or just current behaviour? I don't know, at least it's the current behavior, but I don't know if it's a policy :/ In KDE and Qt herds for example, we usually just unCC ourselves when we've taken the required action. Best regards, Michael signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
El mié, 12-09-2012 a las 14:42 -0400, Rich Freeman escribió: On Wed, Sep 12, 2012 at 2:29 PM, Jeroen Roovers j...@gentoo.org wrote: So you would want to be re-CC'd when it is time to remove the vulnerable versions, I guess. Isn't this done shortly after keywording is complete? I think the concern is more about issuing GLSAs/etc, which apparently can happen months or years after the vulnerable versions were removed judging by recent chromium@ mail. Yes, I am referring to that GLSA messages that are received months later and are useless to maintainers You can un-CC yourself. I don't see why security@ should be doing the legwork. I see no issue with that. Rich signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [RFC] new vala.eclass
On Sun, 2012-09-09 at 22:09 -0400, Alexandre Rostovtsev wrote:
Revised proposal with suggestions from Nirbheek. VALA_API_VERSION has
been split into max and min to make it easier for packages to depend on
a range of vala slots.
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: vala.eclass
# @MAINTAINER:
# gn...@gentoo.org
# @AUTHOR:
# Alexandre Rostovtsev tetrom...@gentoo.org
# @BLURB: Sets up the environment for using a specific version of vala.
# @DESCRIPTION:
# This eclass sets up commonly used environment variables for using a specific
# version of dev-lang/vala to configure and build a package. It is needed for
# packages whose build systems assume the existence of certain unversioned
vala
# executables, pkgconfig files, etc., which Gentoo does not provide.
#
# This eclass provides one phase function: src_prepare.
inherit multilib
case ${EAPI:-0} in
0) die EAPI=0 is not supported ;;
1) ;;
*) EXPORT_FUNCTIONS src_prepare ;;
esac
# @ECLASS-VARIABLE: VALA_MIN_API_VERSION
# @DEFAULT_UNSET
# @DESCRIPTION:
# Minimum vala API version (e.g. 0.16).
VALA_MIN_API_VERSION=${VALA_MIN_API_VERSION:-0.10}
# @ECLASS-VARIABLE: VALA_MAX_API_VERSION
# @DEFAULT_UNSET
# @DESCRIPTION:
# Maximum vala API version (e.g. 0.18).
VALA_MAX_API_VERSION=${VALA_MAX_API_VERSION:-0.18}
# @ECLASS-VARIABLE: VALA_USE_DEPEND
# @DEFAULT_UNSET
# @DESCRIPTION:
# USE dependencies that vala must be built with (e.g. vapigen).
# @FUNCTION: vala_api_versions
# @DESCRIPTION:
# Outputs a list of vala API versions from VALA_MAX_API_VERSION down to
# VALA_MIN_API_VERSION.
vala_api_versions() {
eval echo
0.{${VALA_MAX_API_VERSION#0.}..${VALA_MIN_API_VERSION#0.}..2}
}
# @FUNCTION: vala_depend
# @DESCRIPTION:
# Outputs a ||-dependency string on vala from VALA_MAX_API_VERSION down to
# VALA_MIN_API_VERSION
vala_depend() {
local u v versions=$(vala_api_versions)
[[ ${VALA_USE_DEPEND} ]] u=[${VALA_USE_DEPEND}]
echo -n || (
for v in ${versions}; do
echo -n dev-lang/vala:${v}${u}
done
echo )
}
# @FUNCTION: vala_best_api_version
# @DESCRIPTION:
# Returns the highest installed vala API version satisfying
# VALA_MAX_API_VERSION, VALA_MIN_API_VERSION, and VALA_USE_DEPEND.
vala_best_api_version() {
local u v
[[ ${VALA_USE_DEPEND} ]] u=[${VALA_USE_DEPEND}]
for v in $(vala_api_versions); do
has_version dev-lang/vala:${v}${u} echo ${v} return
done
}
# @FUNCTION: vala_src_prepare
# @USAGE: [--vala-api-version api_version]
# @DESCRIPTION:
# Sets up the environment variables and pkgconfig files for the
# specified API version, or, if no version is specified, for the
# highest installed vala API version satisfying
# VALA_MIN_API_VERSION, VALA_MIN_API_VERSION, and VALA_USE_DEPEND.
vala_src_prepare() {
local p d valafoo version
if [[ $1 = --vala-api-version ]]; then
version=$2
[[ ${version} ]] || die '--vala-api-version' option requires
API version parameter.
else
version=$(vala_best_api_version)
[[ ${version} ]] || die No installed vala in $(vala_depend)
fi
export VALAC=$(type -P valac-${version})
valafoo=$(type -P vala-gen-introspect-${VALA_API_VERSION})
[[ ${valafoo} ]] export VALA_GEN_INTROSPECT=$(type -P
vala-gen-introspect-${version})
valafoo=$(type -P vapigen-${VALA_API_VERSION})
[[ ${valafoo} ]] export VAPIGEN=${valafoo}
valafoo=${EPREFIX}/usr/share/vala/Makefile.vapigen
[[ -e ${valafoo} ]] export VAPIGEN_MAKEFILE=${valafoo}
export VAPIGEN_VAPIDIR=${EPREFIX}/usr/share/vala/vapi
mkdir -p ${T}/pkgconfig || die mkdir failed
for p in libvala vapigen; do
for d in ${EPREFIX}/usr/$(get_libdir)/pkgconfig
${EPREFIX}/usr/share/pkgconfig; do
if [[ -e ${d}/${p}-${VALA_API_VERSION}.pc ]]; then
ln -s ${d}/${p}-${VALA_API_VERSION}.pc
${T}/pkgconfig/${p}.pc || die ln failed
break
fi
done
done
:
${PKG_CONFIG_PATH:=${EPREFIX}/usr/$(get_libdir)/pkgconfig:${EPREFIX}/usr/share/pkgconfig}
export PKG_CONFIG_PATH=${T}/pkgconfig:${PKG_CONFIG_PATH}
}
Now in portage.
[gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
Hola folks.
Currently portage exposes a fair amount of it's internal
implementation via vars/funcs into the ebulid env; this frankly makes
it easier for ebuilds/eclasses to localize themselves to portage
(rather than PMS), leading to breakage.
Thus a proposal for EAPI5 has been made, banning ebuilds/eclasses from
using/accessing __, and requiring the PM to store it's internals in
that namespace.
Basically, instead of portage doing thus:
is_function dyn_pkg_preinst dyn_pkg_preinst
It does thus:
__is_function __dyn_pkg_preinst __dyn_pkg_preinst.
Aside from avoiding accidental conflicts/usage, the standardized
namespacing makes it a helluva lot easier to have repoman/qa tools
look for bad usage.
Currently, there is a minor amount of ebuild/eclass usage of things
named __*; ~90% of it is 'import once' eclass code like the following:
if [[ ${___ECLASS_ONCE_LIBTOOL} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_LIBTOOL=recur -_+^+_- spank
Converting that is easy enough, and I'll be doing that work for
gentoo-x86 if folks don't have an issue.
Note there is a few vars we need to exempt; that list is currently
SANDBOX_* and FEATURES. FEATURES is fine to exempt from this rule.
For SANDBOX_*, while that's a PM internal, that's a bit of a grey
zone; regardless, we can actually address that via extending the
sandbox functions a bit:
addwrite [-r|--remove] pathway # for example, to do a removal.
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
which is just
sandboxless() {
SANDBOX_ON=0 $@
}
or SYDBOX_ON=0 (or their equivalent var) for sydbox usage.
Comments?
~harring
Re: [gentoo-dev] Re: About changing security policy to unCC maintainers when their are not needed
On 09/12/2012 02:54 PM, Pacho Ramos wrote: El jue, 13-09-2012 a las 04:30 +1000, Michael Palimaka escribió: On 2012-09-13 03:59, Pacho Ramos wrote: Hello Currently, package maintainers are CCed to security bugs when their are needed. The problem is that, once maintainers add a fixed version and tell security team they are ok to get it stabilized, maintainers are kept CCed until bug is closed by security team. This usually means getting a lot of mail after some time when security team discuss if a GLSA should be filled or not, if security bot adds some comment... some of that comments are applied to really old bugs that need no action from maintainers. Our discussion is very minimal. There will typically never be any more than 3 comments discussing whether to have a GLSA or not -- in the event that 2 security team members disagree and a 3rd has to break the tie. Some bugs have been receiving more spam than usual (lately, from GLSAMaker/CVETool bot) as we have been trying to clean-up old CVE entries in the tool and old bugs. It would be nice if maintainers would follow-up on security bugs in [upstream], [ebuild], [stable], and [cleanup] to get those bugs closed as soon as possible. You are welcome to join the security team to help us keep bugs up-to-date and work on the backlog of GLSAs. :D Maybe would be interesting to change the policy to unCC maintainers again when their action is no longer required. What do you think? Thanks for your thoughts Hello, Is the policy you describe officially documented, or just current behaviour? I don't know, at least it's the current behavior, but I don't know if it's a policy :/ Yes, this is part of the Vulnerability Treatment Policy [1], listed under the Security Bug Wrangler role in Chapter 3. In KDE and Qt herds for example, we usually just unCC ourselves when we've taken the required action. Best regards, Michael The security bug process [2] involves removing the vulnerable versions from the tree after all arches are finished stabilizing. This is to ensure that users do not accidentally install vulnerable software. Many maintainers do not do this and I think that all of us on the security team are guilty of not always following up to ensure the vulnerable versions are dropped. As Jeroen mentioned, how will maintainers know when to remove the vulnerable versions if they are not current on the bug? If stabilization is complete and the maintainers have removed vulnerable versions from the tree, there is typically no issue with unCC'ing themselves like KDE/Qt herds do. Arches sometimes run into minor issues that don't warrant opening a new bug - they should be able to get help from maintainers without re-CC'ing them. If a decision were made to unCC maintainers, there would probably be some maintainers/herds that want to be left on the CC list and the security team does not have the capacity right now to keep up with exceptions. (Strictly my opinions, not that of the whole security team) [1] http://www.gentoo.org/security/en/vulnerability-policy.xml#doc_chap3 [2] http://www.gentoo.org/security/en/coordinator_guide.xml#doc_chap3 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
On Wed, 12 Sep 2012 20:53:20 +0200 Pacho Ramos pa...@gentoo.org wrote: You can un-CC yourself. I don't see why security@ should be doing the legwork. It shouldn't be so hard to do, they can do it just when they CC arches, instead of relaying some random team member to do it himself once a useless message is received It does become a chore when you have to check a list to match various CC'd people's preferences and decide whether to un-CC them based on that, the way they were CC'd (did they do it themselves, were they CC'd by security, and so on) and perhaps some other factors someone will no doubt soon propose in this thread. Basically you are saying, why doesn't anyone else do my volunteer work for me. jer
Re: [gentoo-dev] About changing security policy to unCC maintainers when their are not needed
On 13 September 2012 09:43, Jeroen Roovers j...@gentoo.org wrote: On Wed, 12 Sep 2012 20:53:20 +0200 Pacho Ramos pa...@gentoo.org wrote: You can un-CC yourself. I don't see why security@ should be doing the legwork. It shouldn't be so hard to do, they can do it just when they CC arches, instead of relaying some random team member to do it himself once a useless message is received It does become a chore when you have to check a list to match various CC'd people's preferences and decide whether to un-CC them based on that, the way they were CC'd (did they do it themselves, were they CC'd by security, and so on) and perhaps some other factors someone will no doubt soon propose in this thread. Basically you are saying, why doesn't anyone else do my volunteer work for me. jer I don't mind getting the odd security bug mail. It's relatively low volume, and I like to know what's happening to packages I maintain. What irks me much more is that it can take half an eternity for security bugs to get addressed properly. Especially minor arches can stretch out the stabilization process for months or years. Recently we (Qt team) had to push really hard and punish lagging minor arches with hard-masking Qt libs and all reverse dependencies in order to get an ancient version with several open security bugs removed from the tree (because they hadn't keyworded/stabilized newer versions and were unresponsive to our requests). I think we should adopt a policy that we set a hard limit of 3 months in which arches can address stabilization requests before we just drop keywords. Even that is in my opinion an awfully long time to leave vulnerable versions in the tree. -- Cheers, Ben | yngwin Gentoo developer Gentoo Qt project lead, Gentoo Wiki admin
Re: [gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
On 13 September 2012 04:36, Brian Harring ferri...@gmail.com wrote:
Hola folks.
Currently portage exposes a fair amount of it's internal
implementation via vars/funcs into the ebulid env; this frankly makes
it easier for ebuilds/eclasses to localize themselves to portage
(rather than PMS), leading to breakage.
Thus a proposal for EAPI5 has been made, banning ebuilds/eclasses from
using/accessing __, and requiring the PM to store it's internals in
that namespace.
Basically, instead of portage doing thus:
is_function dyn_pkg_preinst dyn_pkg_preinst
It does thus:
__is_function __dyn_pkg_preinst __dyn_pkg_preinst.
Aside from avoiding accidental conflicts/usage, the standardized
namespacing makes it a helluva lot easier to have repoman/qa tools
look for bad usage.
Currently, there is a minor amount of ebuild/eclass usage of things
named __*; ~90% of it is 'import once' eclass code like the following:
if [[ ${___ECLASS_ONCE_LIBTOOL} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_LIBTOOL=recur -_+^+_- spank
Converting that is easy enough, and I'll be doing that work for
gentoo-x86 if folks don't have an issue.
Note there is a few vars we need to exempt; that list is currently
SANDBOX_* and FEATURES. FEATURES is fine to exempt from this rule.
For SANDBOX_*, while that's a PM internal, that's a bit of a grey
zone; regardless, we can actually address that via extending the
sandbox functions a bit:
addwrite [-r|--remove] pathway # for example, to do a removal.
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
which is just
sandboxless() {
SANDBOX_ON=0 $@
}
or SYDBOX_ON=0 (or their equivalent var) for sydbox usage.
Comments?
~harring
I support this. It seems very sane.
--
Cheers,
Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
[gentoo-dev] Let's populate IUSE_IMPLICIT in the base profile
Hi, The council has approved [1] Profile IUSE injection [2] for inclusion in EAPI 5, and in latest Portage we have experimental EAPI 5_pre2 [3] which implements all of the approved features. So, now would be a good time to start populating IUSE_IMPLICIT with whatever values may be appropriate. What values belong there? Some of the flags that appear in profiles/base/use.mask might make good candidates, such as prefix and selinux. How about other special flags like bootstrap, build, and test? I've already populated the variables for the ARCH, ELIBC, KERNEL, and USERLAND flags that we can probably all agree on [4] (oops, I screwed up the commit message). The resulting injected list of IUSE_EFFECTIVE flags looks like this: alpha amd64 amd64-fbsd amd64-linux arm arm-linux elibc_AIX elibc_Cygwin elibc_Darwin elibc_FreeBSD elibc_HPUX elibc_Interix elibc_NetBSD elibc_OpenBSD elibc_SunOS elibc_Winnt elibc_glibc elibc_mintlib elibc_uclibc hppa hppa-hpux ia64 ia64-hpux ia64-linux kernel_AIX kernel_Cygwin kernel_Darwin kernel_FreeBSD kernel_HPUX kernel_Interix kernel_NetBSD kernel_OpenBSD kernel_SunOS kernel_Winnt kernel_freemint kernel_linux m68k m68k-mint mips ppc ppc-aix ppc-macos ppc-openbsd ppc64 ppc64-linux s390 sh sparc sparc-fbsd sparc-solaris sparc64-freebsd sparc64-solaris userland_BSD userland_GNU x64-freebsd x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-freebsd x86-interix x86-linux x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt [1] http://thread.gmane.org/gmane.linux.gentoo.pms/653 [2] http://dev.gentoo.org/~zmedico/portage/doc/portage.html#package-ebuild-eapi-5_pre2-profile-iuse-injection [3] http://blogs.gentoo.org/zmedico/2012/09/12/experimental-eapi-5_pre2/ [4] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/profiles/base/make.defaults?view=log#rev1.84 -- Thanks, Zac
Re: [gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
On Thu, 13 Sep 2012, Brian Harring wrote:
Note there is a few vars we need to exempt; that list is currently
SANDBOX_* and FEATURES. FEATURES is fine to exempt from this rule.
For SANDBOX_*, while that's a PM internal, that's a bit of a grey
zone; regardless, we can actually address that via extending the
sandbox functions a bit:
addwrite [-r|--remove] pathway # for example, to do a removal.
As already discussed on IRC, a -r option may have undesired effects.
For example:
addwrite /foo/bar
# some commands writing to /foo/bar here
addwrite -r /foo/bar # try to restore previous state
Now if /foo/bar was in SANDBOX_WRITE before, it's now removed from it.
Maybe it's better to add a --{save,restore} option pair:
addwrite --save /foo/bar
# some commands writing to /foo/bar here
addwrite --restore # restore last saved state
or --{push,pop} to allow for nested calls, but maybe that would be
overkill.
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
To start the bikeshedding: For some reason I associate less (the
pager) in a sandbox with this. ;-) Maybe nosandbox or sandboxoff?
Ulrich
Re: [gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
On Wed, Sep 12, 2012 at 4:36 PM, Brian Harring wrote:
Currently, there is a minor amount of ebuild/eclass usage of things
named __*; ~90% of it is 'import once' eclass code like the following:
if [[ ${___ECLASS_ONCE_LIBTOOL} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_LIBTOOL=recur -_+^+_- spank
Converting that is easy enough, and I'll be doing that work for
gentoo-x86 if folks don't have an issue.
pick a funner prefix and i approve
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
fine idea
-mike
Re: [gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
On Thu, Sep 13, 2012 at 1:48 AM, Ulrich Mueller wrote:
On Thu, 13 Sep 2012, Brian Harring wrote:
Note there is a few vars we need to exempt; that list is currently
SANDBOX_* and FEATURES. FEATURES is fine to exempt from this rule.
For SANDBOX_*, while that's a PM internal, that's a bit of a grey
zone; regardless, we can actually address that via extending the
sandbox functions a bit:
addwrite [-r|--remove] pathway # for example, to do a removal.
As already discussed on IRC, a -r option may have undesired effects.
For example:
addwrite /foo/bar
# some commands writing to /foo/bar here
addwrite -r /foo/bar # try to restore previous state
Now if /foo/bar was in SANDBOX_WRITE before, it's now removed from it.
Maybe it's better to add a --{save,restore} option pair:
addwrite --save /foo/bar
# some commands writing to /foo/bar here
addwrite --restore # restore last saved state
or --{push,pop} to allow for nested calls, but maybe that would be
overkill.
not sure how your --save/--restore isn't a --push/--pop already
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
To start the bikeshedding: For some reason I associate less (the
pager) in a sandbox with this. ;-) Maybe nosandbox or sandboxoff?
sandboxoff
-mike
[gentoo-portage-dev] Re: app-shells/gentoo-bashcomp needs some portage expert lovin'
On Tue, 11 Sep 2012 14:33:51 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Could a number of folks familiar with both portage AND bash completion help out the shell-tools herd (no specific maintainer specified in metadata) with app-shell/gentoo-bashcomp bugs? It seems to have accumulated quite a number (13) of portage-completion (and gentoolkit-completion) related bugs, including bug #424777 filed on July 4th by jer, mentioning that it it breaks with make.conf in /etc/ portage, which is kinda critical right about now. =:^( https://bugs.gentoo.org/show_bug.cgi?id=424777 And here's the list of all open bugs for it. As you can see if you look, all or nearly all of the currently 13 open bugs are portage or gentoolkit related, and it really looks like the package could use some help from those who know those tools. https://bugs.gentoo.org/buglist.cgi?quicksearch=app-shells%2Fgentoo-bashcomp (I said on the devlist thread I'd report bugs, but this one's been open for over two months and is directly related to the make.conf move, so it's a big one, simple fix proposed, but no actual fixes in-tree yet, plus seeing all those others related to portage and gentoolkit, thus the request here.) I was planning on working on this. I just need to get off my butt and find someone to give me write access to the overlay. -- gcc-porting toolchain, wxwidgets we were never more here, expanse getting broader @ gentoo.org but bigger boats been done by less water signature.asc Description: PGP signature
Re: [gentoo-portage-dev] How to prevent dispatch-conf from reverting valid changes
El jue, 02-08-2012 a las 12:34 -0700, Zac Medico escribió: On 08/01/2012 11:36 PM, Pacho Ramos wrote: El mié, 01-08-2012 a las 16:14 -0700, Zac Medico escribió: On 08/01/2012 03:19 AM, Pacho Ramos wrote: On every openrc update I get dispatch-conf wanting to revert all my changes in /etc/conf.d files, like KEYMAP, clock... Is there any way to prevent it from doing that? Thanks a lot for the info Maybe we can trace the behavior back to the diff3 command that it's using. Inside /usr/lib/portage/pym/portage/dispatch_conf.py we have this command: DIFF3_MERGE = diff3 -mE '%s' '%s' '%s' '%s' Are you able to reproduce the problem by running this command manually? Something like this: diff3 -mR /etc/conf.d/hostname \ /etc/config-archive/etc/conf.d/hostname.dist \ /etc/conf.d/._cfg_hostname /tmp/mrgconf I will probably need to reemerge it as I already merged changes and re-edited config files. Anyway, diff3 looks to not admit -R option: $ diff3 -mR /etc/conf.d/hostname /etc/config-archive/etc/conf.d/hostname.dist /etc/conf.d/._cfg_hostname diff3: opción inválida -- R diff3: Pruebe `diff3 --help' para más información. Sorry, that -mR a typo. As you can see in the DIFF3_MERGE value above, it's diff -mE. Will try on next openrc update :) signature.asc Description: This is a digitally signed message part
