Re: [gentoo-user] Are multiple emerges safe?
On Mon, 2006-01-30 at 13:24 +0100, Huib van Wees wrote: > > On 1/30/06, Neil Bothwick <[EMAIL PROTECTED]> wrote: > On Sun, 29 Jan 2006 19:48:34 +0200, Uwe Thiem wrote: > > > Now I have got a related question: Imagine you have got one > portage > > tree on one box but several boxes that NFS mount it with > different > > world files. Will several parallel "emerge --fetchonly" > processes on > > those different boxes still be safe? > > Yes, because portage uses lockfiles in $DISTDIR. Once one > emerge begins > downloading a file, any others wanting that file will wait for > it. > > I can remember that export the portage dir for other host wasn't > supported and can cause problems Or did I missread/misinterpert > the documentation? > > > -- > Met vriendelijke groet / With kind regards, > > H. van Wees > --- > The official Gentoo motto is, "If it moves, compile it." I've exported /usr/portage through NFS from my server box to two client boxes for a couple of years now, and I've never had any problems with it... -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Mon, 30 Jan 2006 12:41:37 +0100 Benno Schulenberg <[EMAIL PROTECTED]> wrote: | Ciaran McCreesh wrote: | > the many ways in which parallel merges can still break. | | Please give us one example. You install a package that runs autotools and a package that provides an aclocal entry at the same time. You're unlucky, and an aclocal directory has its contents changed whilst autotools is running. -- Ciaran McCreesh : Gentoo Developer (King of all Londinium) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
On Mon, 30 Jan 2006 13:24:14 +0100, Huib van Wees wrote: > > Yes, because portage uses lockfiles in $DISTDIR. Once one emerge > > begins downloading a file, any others wanting that file will wait for > > it. > > > > I can remember that export the portage dir for other host wasn't > > supported > and can cause problems Or did I missread/misinterpert the > documentation? I don't share the portage tree, although many do, but I do have a shared $DISTDIR and it works with the locks exactly as advertised. -- Neil Bothwick Only an idiot actually READS taglines. signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
On 1/30/06, Neil Bothwick <[EMAIL PROTECTED]> wrote: On Sun, 29 Jan 2006 19:48:34 +0200, Uwe Thiem wrote:> Now I have got a related question: Imagine you have got one portage> tree on one box but several boxes that NFS mount it with different> world files. Will several parallel "emerge --fetchonly" processes on > those different boxes still be safe?Yes, because portage uses lockfiles in $DISTDIR. Once one emerge beginsdownloading a file, any others wanting that file will wait for it. I can remember that export the portage dir for other host wasn't supported and can cause problems Or did I missread/misinterpert the documentation?-- Met vriendelijke groet / With kind regards, H. van Wees---The official Gentoo motto is, "If it moves, compile it."
Re: [gentoo-user] Are multiple emerges safe?
Ciaran McCreesh wrote: > the many ways in which parallel merges can still break. Please give us one example. Benno -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sun, 29 Jan 2006 19:48:34 +0200, Uwe Thiem wrote: > Now I have got a related question: Imagine you have got one portage > tree on one box but several boxes that NFS mount it with different > world files. Will several parallel "emerge --fetchonly" processes on > those different boxes still be safe? Yes, because portage uses lockfiles in $DISTDIR. Once one emerge begins downloading a file, any others wanting that file will wait for it. -- Neil Bothwick We've all heard that a million monkeys banging on a million typewriters will eventually reproduce the works of Shakespeare. Now, thanks to the Internet, we know this is not true. signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
On Sun, 29 Jan 2006 22:33:01 -1000 "Beau E. Cox" <[EMAIL PROTECTED]> wrote: | On Sunday 29 January 2006 09:59 pm, Ciaran McCreesh wrote: | > On Sun, 29 Jan 2006 00:51:32 -1000 "Beau E. Cox" | > | > <[EMAIL PROTECTED]> wrote: | > | Can I safely emerge (different packages, non-interdependent) in | > | different terminal sessions at the same time? | > | > No. | | Many have said "Yes." Can you explain? Sure. Those people are optimists who just happen to have not hit any of the many ways in which parallel merges can still break. -- Ciaran McCreesh : Gentoo Developer (King of all Londinium) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
On Sunday 29 January 2006 09:59 pm, Ciaran McCreesh wrote: > On Sun, 29 Jan 2006 00:51:32 -1000 "Beau E. Cox" > > <[EMAIL PROTECTED]> wrote: > | Can I safely emerge (different packages, non-interdependent) in > | different terminal sessions at the same time? > > No. Many have said "Yes." Can you explain? -- Aloha => Beau; -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sun, 29 Jan 2006 00:51:32 -1000 "Beau E. Cox" <[EMAIL PROTECTED]> wrote: | Can I safely emerge (different packages, non-interdependent) in | different terminal sessions at the same time? No. -- Ciaran McCreesh : Gentoo Developer (King of all Londinium) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
On 29 January 2006 18:53, Michael Sullivan wrote: > Keep in mind though that multiple emerges will share resources, so each > process may lose speed... True, but "--fetchonly" emerge can pull in sources as fast as possible while the other emerge compiles stuff. That makes sense if you are paying through your nose for bandwidth. Now I have got a related question: Imagine you have got one portage tree on one box but several boxes that NFS mount it with different world files. Will several parallel "emerge --fetchonly" processes on those different boxes still be safe? Uwe -- Unix is sexy: who | grep -i blonde | date cd ~; unzip; touch; strip; finger mount; gasp; yes; uptime; umount sleep -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
Alexander Skwar wrote: Something that I always wondered about - does it actually *lose* speed? IOW: Is it, *IN* *TOTAL* slower to do multiple emerges in parallel compared to doing them sequentially? I would expect the answer to this lies in how you set your MAKEOPTS. -- gentoo-user@gentoo.org mailing list
Re[2]: [gentoo-user] Are multiple emerges safe?
Hello, Alexander. You wrote: >> Keep in mind though that multiple emerges will share resources, so each >> process may lose speed... > Something that I always wondered about - does it > actually *lose* speed? IOW: Is it, *IN* *TOTAL* > slower to do multiple emerges in parallel compared > to doing them sequentially? Given that you have enough RAM, it may speed up things somewhat. If you do some cpu load monitoring (e.g., using 'top') you should see that cpu load during emerge process does not stay constant. There are 99% periods (usually when compiling something), and there are 1% periods (sh, grep, ldconfig, emerge itself and most notably configure). If you are installing something large (say, KDE) you can get your cpu working at 100% when one instance of emerge configures and other does actual compilation. Mind that gcc can hog up to 150-200 megabytes of RAM per source file when compiling large c++ programs (e.g., kde with kdeenablefinal USE flag), so trying parallel emerge on systems with less than 512 megs of RAM space is rather pointless. -- WBR, Alexander mailto:[EMAIL PROTECTED] -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
At Sun, 29 Jan 2006 13:03:19 -0600 Dale <[EMAIL PROTECTED]> wrote: > Alexander Skwar wrote: > >>Something that I always wondered about - does it >>actually *lose* speed? IOW: Is it, *IN* *TOTAL* >>slower to do multiple emerges in parallel compared >>to doing them sequentially? >> > I'm no guru for sure but here is my $.02 worth. Let's say emerge one > takes ten minutes and emerge two takes 30 minutes. If you have a single > CPU machine I would think that if both were started at the same time, > the both of them would take 40 minutes. So in my theory, "emerge one && > emerge two" should be the same as doing seperately. It is more complicated than that. 1. A single CPU machine most likely has a DMA I/O controller and hence more than one action can be concurrent. 2. I/O itself causes long delays while the disk is active, with a second emerge (or any other processor) the CPU can be concurrent with this delay. These two argue for the concurrent emerges to be faster, but 3. The context switching between processes is not free and adds CPU (at least) requirements to the concurrent emerge situation. 4. The concurrent invocation increases the cache pressure (i.e., demand) which most likely increases the cache miss rate. 5. Similarly for the buffer cache. etc allan -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sun, 29 Jan 2006 19:26:41 +0100, Alexander Skwar wrote: > Something that I always wondered about - does it > actually *lose* speed? IOW: Is it, *IN* *TOTAL* > slower to do multiple emerges in parallel compared > to doing them sequentially? It could be faster, when one emerge is in a CPU-intensive stage while another is in an I/O-intensive stage. At that point, the computer would be working harder than on either emerge individually. -- Neil Bothwick I have plenty of talent and vision. I just don't give a damn. signature.asc Description: PGP signature
Re: [gentoo-user] Are multiple emerges safe?
Alexander Skwar wrote: >Michael Sullivan wrote: > > > >>Keep in mind though that multiple emerges will share resources, so each >>process may lose speed... >> >> > >Something that I always wondered about - does it >actually *lose* speed? IOW: Is it, *IN* *TOTAL* >slower to do multiple emerges in parallel compared >to doing them sequentially? > >Alexander Skwar > > I'm no guru for sure but here is my $.02 worth. Let's say emerge one takes ten minutes and emerge two takes 30 minutes. If you have a single CPU machine I would think that if both were started at the same time, the both of them would take 40 minutes. So in my theory, "emerge one && emerge two" should be the same as doing seperately. Of course, emerge one would finish first but while emerge one is working it will slow down emerge two as well. This is assuming they have the same nice settings of course. Now if you have a dually or quad CPU rig, that may vary depending on the compile times and a few other things. I would also think that one emerge would speed up the compile for example when the other is waiting on the data to/from a drive as well. So it may make it a little faster. I wonder of there is a good way to measure this. My worthless $0.02 worth. Dale :-) -- To err is human, I'm most certainly human. I have four rigs: 1: Home built; Abit NF7 ver 2.0 w/ AMD 2500+ CPU, 1GB of ram and right now two 80GB hard drives. Named Smoker 2: Home built; Iwill KK266-R w/ AMD 1GHz CPU, 256MBs of ram and a 4GB drive. Named Swifty 3: Home built; Gigabyte GA-71XE4 w/ 800MHz CPU, 224MBs of ram and a 2.5GB drive. Named Pokey 4: Compaq Proliant 6000 Server w/ Quad 200MHz CPUs, 128MBs of ram and a 4.3GB SCSI drive. Named Putput All run Gentoo Linux, all run folding. #1 is my desktop, 2, 3, and 4 are set up as servers. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
Michael Sullivan wrote: > Keep in mind though that multiple emerges will share resources, so each > process may lose speed... Something that I always wondered about - does it actually *lose* speed? IOW: Is it, *IN* *TOTAL* slower to do multiple emerges in parallel compared to doing them sequentially? Alexander Skwar -- The savior becomes the victim. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sunday 29 January 2006 06:53 am, Michael Sullivan wrote: > On Sun, 2006-01-29 at 01:02 -1000, Beau E. Cox wrote: > > On Sunday 29 January 2006 12:56 am, Alexander Skwar wrote: > > > Beau E. Cox wrote: > > > > Can I safely emerge (different packages, non-interdependent) in > > > > different terminal sessions at the same time? > > > > > > Yes, I do all that all the time. > > > > > > Alexander Skwar > > > -- > > > we need to split main into"core" and "wtf-uses-this" > > > > Thanks. I've been waiting on a veeery long emerge. > > > > -- > > Aloha => Beau; > > Keep in mind though that multiple emerges will share resources, so each > process may lose speed... Yes. But when an emerge (ghc - haskell - in my case) is taking *hours* to finish, it's nice to know that I can fire up another one... :) -- Aloha => Beau; -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sun, 2006-01-29 at 01:02 -1000, Beau E. Cox wrote: > On Sunday 29 January 2006 12:56 am, Alexander Skwar wrote: > > Beau E. Cox wrote: > > > Can I safely emerge (different packages, non-interdependent) in > > > different terminal sessions at the same time? > > > > Yes, I do all that all the time. > > > > Alexander Skwar > > -- > > we need to split main into"core" and "wtf-uses-this" > > Thanks. I've been waiting on a veeery long emerge. > > -- > Aloha => Beau; Keep in mind though that multiple emerges will share resources, so each process may lose speed... -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sunday 29 January 2006 01:04 am, Andrei Slavoiu wrote: > --- "Beau E. Cox" <[EMAIL PROTECTED]> wrote: > > Hi - > > > > Can I safely emerge (different packages, > > non-interdependent) in > > different terminal sessions at the same time? > > > > -- > > Aloha => Beau; > > > > -- > > gentoo-user@gentoo.org mailing list > > Yes, you can start as many sesions of emerge as you > like, even for the same package, portage uses locks to > make sure they do not conflict with each other. > A very useful application for this is to do something > like this: in one console start `emerge -f kde` and in > another `emerge kde`. The first will just download the > files needed and the second will compile them. That's good to know. I used Sorcerer for about two years and was burned by trying multiple 'casts' (the Sorcerer emerge) several times, so I just wanted to check. > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com -- Aloha => Beau; -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
On Sunday 29 January 2006 12:56 am, Alexander Skwar wrote: > Beau E. Cox wrote: > > Can I safely emerge (different packages, non-interdependent) in > > different terminal sessions at the same time? > > Yes, I do all that all the time. > > Alexander Skwar > -- > we need to split main into"core" and "wtf-uses-this" Thanks. I've been waiting on a veeery long emerge. -- Aloha => Beau; -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
--- "Beau E. Cox" <[EMAIL PROTECTED]> wrote: > Hi - > > Can I safely emerge (different packages, > non-interdependent) in > different terminal sessions at the same time? > > -- > Aloha => Beau; > > -- > gentoo-user@gentoo.org mailing list > > Yes, you can start as many sesions of emerge as you like, even for the same package, portage uses locks to make sure they do not conflict with each other. A very useful application for this is to do something like this: in one console start `emerge -f kde` and in another `emerge kde`. The first will just download the files needed and the second will compile them. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Are multiple emerges safe?
Beau E. Cox wrote: > Can I safely emerge (different packages, non-interdependent) in > different terminal sessions at the same time? Yes, I do all that all the time. Alexander Skwar -- we need to split main into"core" and "wtf-uses-this" -- gentoo-user@gentoo.org mailing list
[gentoo-user] Are multiple emerges safe?
Hi - Can I safely emerge (different packages, non-interdependent) in different terminal sessions at the same time? -- Aloha => Beau; -- gentoo-user@gentoo.org mailing list
