Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds
Philippe Trottier wrote:
Lisa Seelye wrote:
On Thu, 2006-01-12 at 00:18 +, Ferris McCormick wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 11 Jan 2006, Lisa Seelye wrote:
On Wed, 2006-01-11 at 14:51 -0800, Robin H. Johnson wrote:
I've been cleaning up media-fonts/ to work with modular-X, and I see a
lot of ebuilds with stuff like this:
for font in *.bdf; do
/usr/X11R6/bin/bdftopcf ${font} `basename $font .bdf`.pcf
done
gzip *.pcf
For having 100 files in *bdf, this is so serial it's painful.
And here I was hoping Distcc would get some usage. :(
Distcc gets lots of usage with modular X. But for the fonts? :)
Time for distfont? ;)
Make this distributed tool for tar zip bzip2 and gzip and I'm in, I
don't think it would be useful with anything else than Gigabit Ethernet.
We might want to have in the make.conf 2 separate variables, one of them
saying how many threads can be run on the machine, then How many
threads/process across a cluster.
For example, my Dual Xeon EM64T file server can do make -j4 locally,
like in make install, make docs etc etc, But for compiling I can use
-j20, really not useful over -j8 anyway. But the point is, it would be
usefully to separate the load distribution on the local machine and
cluster nodes.
As the discusison started...
I would like to be able to limit the -jN when there is no distcc host
available or when compiling c++ code, otherwise my poor laptop is dead with
-j5 compiling pwlib when the network is down
It is particular example, but being able to limit portage in some way as
total CPU, total MEM might be interesting (just nice-ing is not enough)
Kalin.
--
|[ ~~ ]|
+- http://ThinRope.net/ -+
|[ __ ]|
--
gentoo-dev@gentoo.org mailing list
[gentoo-dev] pending dooooooom of use.defaults
as one of the new sane features of the next portage-2.1_pre release, we're looking to cut out use.defaults support existing stable users wont be affected as the 2.0.x versions will continue to carry support for this, but some of you stable users may notice some USE flags suddenly disappearing to recap, use.defaults inserts USE flags for you based upon what packages you have installed when you havent declared a preference. for example, if you have neither '-cups' or 'cups' in your USE (either in your make.conf, profile, env, whatever), but you do have the net-print/cups package installed, portage will add 'cups' to your USE -mike -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Re: Parallizing ebuilds - 'trivial' ebuilds
Patrick Lauer posted [EMAIL PROTECTED], excerpted below, on Fri, 13 Jan 2006 12:06:28 +0100: Very difficult - usually gcc uses ~25M per process (small source files), but I've seen 100M (most larger C++ files) and heard of ~600M per process for MySQL Limiting that is beyond the scope of portage. There's one point in the kmail/kdepim (split/monolithic) build where with USE=kdeenablefinal on AMD64, a single process takes 700 meg, based on my results. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman in http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Monthly Gentoo Council Reminder for January
On Thursday 05 January 2006 17:20, Patrick Lauer wrote: But it's already getting too bureaucratic ;-) It's getting more and more difficult to get things done, more and more people / groups / herds to wait on to decide obvious things. They shouldn't. If there is anything I learned is that a mailing list never comes to a decision. At some point the principal stakeholder (the person waiting for the decision) must make a conclusion, and get to work. It works. The support was there, people will follow, end else there is repoman to force them to ;-). For example - our baselayout supports UML and vServer (almost fully) native. Most of you won't see that, but to those that do it's something that's really nice. One of the reasons that gentoo is still my favourite distro. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgp5goZauTR0f.pgp Description: PGP signature
Re: [gentoo-dev] Monthly Gentoo Council Reminder for January
On Thursday 05 January 2006 18:03, Patrick Lauer wrote: Exactly :-) But I guess many among us have become a bit disillusioned and try to stay away from what is perceived as useless trolling and silly infights. So things either stall in discussion or get implemented with the obvious flawed approach (early webapp-config and portage are good examples) and then take a long time to become fixed. There's still a lot of good stuff happening, but as someone else said in this thread, we suck at execution :-( I guess, the council should be more brave, and make decisions like rejecting flawed approaches. Even when discussions have not been thrown up and re-eaten again. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpcPZHhp4dPQ.pgp Description: PGP signature
Re: [gentoo-dev] Projects and simple guides
On Tue, 10 Jan 2006 15:12:27 -0600 Lance Albertson [EMAIL PROTECTED] wrote: | Last I knew, its not a simple task for generating those nice looking | html pages that ciaranm made a while back for the developer docs. | When I asked him about (he can probably provide more detail), It took | a lot of processing time and wasn't that scalable. Now, I'm not sure | if anything has changed since then. If you're using docutils, then yes, it's reaallly slow. I've got a (very fast) parser that handles a decent subset of the RST spec written, but getting it converted to be usable in a general kind of way isn't too high up my list of priorities... The thing is... If you're trying to do RST - GuideXML, you'll run into all kinds of weirdness because of the GuideXML heading structure. You'll also run into a load more weirdness because about half of the GLEPs currently massively abuse blockquotes (in all but one case accidentally). See, this is a list in RST: * one * two And this is a list inside a blockquote: * one * two Very easy to screw up, especially since docutils goes to great lengths to create output even if the input is highly weird. My own parser moans on anything like that -- it disallows most nested structure markup -- which means it's useless on most GLEPs unless someone goes through and does some serious whitespace cleanups... -- Ciaran McCreesh : Gentoo Developer (King of all Londinium) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds
Patrick Lauer wrote: On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote: Make this distributed tool for tar zip bzip2 and gzip and I'm in, I don't think it would be useful with anything else than Gigabit Ethernet. One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I can tell. Although the speedups won't be extreme it could just work. We might want to have in the make.conf 2 separate variables, one of them saying how many threads can be run on the machine, then How many threads/process across a cluster. For example, my Dual Xeon EM64T file server can do make -j4 locally, like in make install, make docs etc etc, But for compiling I can use -j20, really not useful over -j8 anyway. But the point is, it would be usefully to separate the load distribution on the local machine and cluster nodes. As the discusison started... I would like to be able to limit the -jN when there is no distcc host available or when compiling c++ code, otherwise my poor laptop is dead with -j5 compiling pwlib when the network is down As far as I can tell distcc isn't smart enough for dynamic load balancing. One could hack portage to test each server in the distcc host list and remove missing servers for each run - doesn't look elegant to me. Yes, might be a solution, even if not elegant. I am thinking also of automating distcc configuration (i.e. no need to run --set-hosts) and one idea is to use DNS with some TXT record, but that is just an idea - no patching is done yet. Not sure if distcc has local limiter, i.e. if it it set with localhost/2 and portage user (or some other user != root) tries to start 3 processes, the 3rd just blocks (and not take memory). I think not, so this thing might be interesting to implement (for old laptops with less memory). I think I should resubscribe to the distcc list :-) It is particular example, but being able to limit portage in some way as total CPU, total MEM might be interesting (just nice-ing is not enough) Very difficult - usually gcc uses ~25M per process (small source files), but I've seen 100M (most larger C++ files) and heard of ~600M per process for MySQL Limiting that is beyond the scope of portage. Hmm, may be not limiting the total usage, but more like just adjusting MAKEOPTS='-j1' in some cases (NOTE: to /me, define some cases). Implementing the above Local limiter in distcc will solve that automagically. Kalin. -- |[ ~~ ]| +- http://ThinRope.net/ -+ |[ __ ]| -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] pending dooooooom of use.defaults
Mike Frysinger wrote:
as one of the new sane features of the next portage-2.1_pre release, we're
looking to cut out use.defaults support
existing stable users wont be affected as the 2.0.x versions will continue to
carry support for this, but some of you stable users may notice some USE
flags suddenly disappearing
to recap, use.defaults inserts USE flags for you based upon what packages you
have installed when you havent declared a preference. for example, if you
have neither '-cups' or 'cups' in your USE (either in your make.conf,
profile, env, whatever), but you do have the net-print/cups package
installed, portage will add 'cups' to your USE
Can I just ask, since when is this feature on? I have never run into it...
Or is it because I always had:
USE=-* ${MY_USE}
in /etc/make.conf?
Is -* counted as preference? I thought that is ignoring just the ones in
the profile (just is plain wrong, as I didn't even feel there were other
useflags :-)
Kalin
--
|[ ~~ ]|
+- http://ThinRope.net/ -+
|[ __ ]|
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] pending dooooooom of use.defaults
On Fri, Jan 13, 2006 at 06:57:24AM -0500, Mike Frysinger wrote: as one of the new sane features of the next portage-2.1_pre release, we're looking to cut out use.defaults support existing stable users wont be affected as the 2.0.x versions will continue to carry support for this, but some of you stable users may notice some USE ^^ flags suddenly disappearing I'm a bit confused, existing stable users won't be affected, but they will notice use flags disappearing? Wouldn't that mean they are affected or did you simply mistype and mean unstable? to recap, use.defaults inserts USE flags for you based upon what packages you have installed when you havent declared a preference. for example, if you have neither '-cups' or 'cups' in your USE (either in your make.conf, profile, env, whatever), but you do have the net-print/cups package installed, portage will add 'cups' to your USE That's the current behaviour in stable 2.0.x and will be gone with 2.1, right? I'm a little confused now, could you clarify this? cheers, Wernfried -- Wernfried Haas (amne) - amne at gentoo dot org Gentoo Forums: http://forums.gentoo.org IRC: #gentoo-forums on freenode - email: forum-mods at gentoo dot org pgpEq7ORbfSdJ.pgp Description: PGP signature
Re: [gentoo-dev] pending dooooooom of use.defaults
On Friday 13 January 2006 11:15, Kalin KOZHUHAROV wrote:
Or is it because I always had:
USE=-* ${MY_USE}
in /etc/make.conf?
yes
-mike
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] pending dooooooom of use.defaults
On Fri, Jan 13, 2006 at 06:57:24AM -0500, Mike Frysinger wrote: as one of the new sane features of the next portage-2.1_pre release, we're looking to cut out use.defaults support Could you add a USE_ORDER without auto to /etc/make.globals for that release, please, or alternatively provide some other way of checking whether use.defaults is read? This would greatly help me out with ufed, which currently has no way to check this, and instead has to hardcode env:pkg:conf:auto:defaults as the default USE_ORDER just like portage does. pgpx4vERZRK22.pgp Description: PGP signature
Re: [gentoo-dev] pending dooooooom of use.defaults
Can we get this on the website/announce? I agree that auto-use is the
suck and that it needs to die a long excrutiating death, but I think a
lot of users will be like wtf when 2.1 hits stable and --newuse turns up
a massive crapload of packages.
Whether this announced now, or when portage-2.1 hits stable, or both, I
don't really care. If you need a ditty to post about it we can probably
whip one up.
Mike Frysinger wrote:
On Friday 13 January 2006 11:15, Kalin KOZHUHAROV wrote:
Or is it because I always had:
USE=-* ${MY_USE}
in /etc/make.conf?
yes
-mike
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Parallizing ebuilds - 'trivial' ebuilds
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kalin KOZHUHAROV wrote: Patrick Lauer wrote: On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote: Make this distributed tool for tar zip bzip2 and gzip and I'm in, I don't think it would be useful with anything else than Gigabit Ethernet. One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I can tell. Although the speedups won't be extreme it could just work. We might want to have in the make.conf 2 separate variables, one of them saying how many threads can be run on the machine, then How many threads/process across a cluster. For example, my Dual Xeon EM64T file server can do make -j4 locally, like in make install, make docs etc etc, But for compiling I can use -j20, really not useful over -j8 anyway. But the point is, it would be usefully to separate the load distribution on the local machine and cluster nodes. As the discusison started... I would like to be able to limit the -jN when there is no distcc host available or when compiling c++ code, otherwise my poor laptop is dead with -j5 compiling pwlib when the network is down As far as I can tell distcc isn't smart enough for dynamic load balancing. One could hack portage to test each server in the distcc host list and remove missing servers for each run - doesn't look elegant to me. Yes, might be a solution, even if not elegant. I am thinking also of automating distcc configuration (i.e. no need to run --set-hosts) and one idea is to use DNS with some TXT record, but that is just an idea - no patching is done yet. Recipe for disaster, specially in a place like mine where sparc, alpha, x86_64 and ppc32/64 mix... not counting ia64 for a test run soon... If you really want to do this, someone has to make a rendezvous a la Apple. Where not only distcc says I am available but I am also doing the right stuff. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFDx/nsP0/FkJ0eBc0RAn8aAJ9h8C1QFO0HyQmBINl2erPljgaEYwCfcOBF yPpDJ/KElWfotWqNvWYPq3s= =XM29 -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] pending dooooooom of use.defaults
On Fri, 2006-01-13 at 06:57 -0500, Mike Frysinger wrote: as one of the new sane features of the next portage-2.1_pre release, we're looking to cut out use.defaults support I see this as a good and bad thing. Good in one hand that less autojunk would be enabled like python/perl bindings not being added to every program on your system that supports it. Bad in the other hand I see the state of profiles getting worse=more bloated. The autouse itself is not a bad feature or idea if it were used properly. Problem is that it's not been used properly. If it were limited to simple things like just X and the things that actually make sense then it would even be fine to keep and would allow some of the more bloated (default-linux) profiles to be cleaned up. Shrug. I like the existing behavior and the power of deciding for myself when and where I want to take advantage of USE_ORDER= existing stable users wont be affected as the 2.0.x versions will continue to carry support for this, but some of you stable users may notice some USE flags suddenly disappearing to recap, use.defaults inserts USE flags for you based upon what packages you have installed when you havent declared a preference. for example, if you have neither '-cups' or 'cups' in your USE (either in your make.conf, profile, env, whatever), but you do have the net-print/cups package installed, portage will add 'cups' to your USE -mike -- solar [EMAIL PROTECTED] Gentoo Linux -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] pending dooooooom of use.defaults
On Fri, 13 Jan 2006 15:13:02 -0500 solar [EMAIL PROTECTED] wrote: | The autouse itself is not a bad feature or idea if it were used properly. | Problem is that it's not been used properly. No, it's bad. It's another thing that makes correct dependency resolution impossible. -- Ciaran McCreesh : Gentoo Developer (King of all Londinium) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] pending dooooooom of use.defaults
On Friday 13 January 2006 15:13, solar wrote: On Fri, 2006-01-13 at 06:57 -0500, Mike Frysinger wrote: as one of the new sane features of the next portage-2.1_pre release, we're looking to cut out use.defaults support I see this as a good and bad thing. Good in one hand that less autojunk would be enabled like python/perl bindings not being added to every program on your system that supports it. Bad in the other hand I see the state of profiles getting worse=more bloated. i dont really see the profiles getting any more USE flags than they already have ... as for bloated, i see it as being a more-than-worth-it trade off when it comes to stability a profile-based USE will always stay the same while a autouse-based USE may fluctuate greatly based upon what the user emerges from day to day The autouse itself is not a bad feature or idea if it were used properly. there is no used properly or improperly when it comes to use.defaults -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: pending dooooooom of use.defaults
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 snip According to previous posts, USE_ORDER will be going away with use.defaults, because that was really the only reason it was there in the first place as there's no other sane ordering possible, if it is removed. There are other sane orderings possible, one being pkg:env:conf:defaults so that USE=xxx emerge -NpDuv world will show exactly what adding xxx to make.conf will do. I don't recall where I saw this, unfortunately, but I do know that some people actually use it for this. (Okay, maybe that's really the only other sane ordering.) I would prefer to keep USE_ORDER for now, since I was going to replace the auto dict with the default-iuse which means you can choose not to stack these new flags. Although it may be a hack, we have no better way of managing use flag stacks at the moment. - -Alec Warner -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQ8h0XGzglR5RwbyYAQJ5rRAAie3nIGMJQQQuy/tI9as+jniZ2BJ5fDSj Z0sMS/qdPUItdfMQ6WLAthG7Y7RBA3u8rKuYMEDGUZ4ile7s43s7YPpCUXuwwmF8 a4uiX2TY16yE+1I9W7Agp1Bl3fmtL79vYg+39BQooFq2kmcWMFmTO+dE/jbO2bxI 2ttPVERaKHJn+Nu44otqQ84hQ2bQ2tJtLBlmUvCwz5RBVhN9Jj8G/3ZOc/hMuUDo W/ytxHr4AJ/fpMa4ksRBI4jz2KjDbBurYH5tyI3ZRNqVGxzB4Oj4CzakRHevHGs1 h7NeJM2ZGI2r6K5iZlzPMUO2UWaO17KUo1C2w1Oa7Xyl5Zhd8UDphsf/xXCsr13d Goyg/zwCJUIp3cSvp3ajQgxVCDQtHVcT4TKz/6ph7A/UTRn67Byl80cm8rHc7Xl+ Bi6X7UHPURzYEqxqshbfL0eEoOjqxqgQTl9i9lAmiKBIjlgPxH8DuxNOJx1JmcTV tFAa3ktDRWJMjuSPN+9dn4P+9msdlhB1OVQtM9sEvb4Wu2TQzzPWijn42bsWLwxB EBESmuXyycrwuaJD2YC+bdHsFEsdNssKeM1rU7w332fMefQu4ttXYaoAh2HtR9Pf 5lzB2aOp5iL9CCXL+SIQzsRcoIkSva+4xUGMgsRW4ulpVsy5sy9gOO5CHY0s4YjQ jEiI5ljabUo= =dbSA -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Re: Re: pending dooooooom of use.defaults
Alec Warner posted [EMAIL PROTECTED], excerpted below, on Fri, 13 Jan 2006 22:47:40 -0500: snip According to previous posts, USE_ORDER will be going away with use.defaults, because that was really the only reason it was there in the first place as there's no other sane ordering possible, if it is removed. There are other sane orderings possible, one being pkg:env:conf:defaults so that USE=xxx emerge -NpDuv world will show exactly what adding xxx to make.conf will do. I don't recall where I saw this, unfortunately, but I do know that some people actually use it for this. (Okay, maybe that's really the only other sane ordering.) I would prefer to keep USE_ORDER for now, since I was going to replace the auto dict with the default-iuse which means you can choose not to stack these new flags. Although it may be a hack, we have no better way of managing use flag stacks at the moment. I was wondering... but nobody challenged it at the time the plan was mentioned (the previous posts I referred to), and that's one thing I haven't messed with (yet?), so /I/ was keeping quiet. Maybe I misunderstood the entire thing, but I don't think so because I remember being rather unconfortable with it just being outright dismissed like that. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman in http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html -- gentoo-dev@gentoo.org mailing list
