Re: [gentoo-user] Genlop oddity
On Monday 01 Aug 2016 10:03:53 John Blinka wrote: > Glad to see that someone else has experienced the same thing, and I'm not > going crazy (although some might argue this is hardly proof...) No, but it'll do as a bit of evidence pro tem. :) -- Rgds Peter
Re: [gentoo-user] Genlop oddity
On Sun, Jul 31, 2016 at 6:47 AM, Peter Humphrey wrote: > > How is it possible for genlop's reported ETA to increase while its time > spent so far also increases? Could the concurrent gnutls merging have > affected it? Surely not. I've noticed the same oddity recently while building a couple of new amd64 boxes. Unfortunately, I can't document my observations with precise numbers, but I have noticed it on multiple occasions when simultaneously emerging, say, two packages that each take hours to build. As an example, If the estimate for the remaining time was another 1/2-1 hr while the 2d build was sharing system resources with the 1st, I've seen the 2d build estimate jump to 1-2 hr when having exclusive access to system resources after the 1st finishes. It's almost as if somewhere in the code there's logic that estimates time remaining as proportional to available system resources available instead of inversely proportional. This isn't a subtle anomaly, in my opinion. I've used genlop in this way to monitor build times for ages and I've never seen it behave this way before. And I haven't changed my build options by fooling around with -jN or by trying to do other things on the system while emerging. New behavior for genlop, or a 32/64 bit thing? (I've recently gone 64 bit and that's where I see the discrepancy.) Glad to see that someone else has experienced the same thing, and I'm not going crazy (although some might argue this is hardly proof...) John Blinka
Re: [gentoo-user] Genlop oddity
On Sunday 31 Jul 2016 18:52:23 Fernando Rodriguez wrote: --->8 > Just out of curiosity what are the differences between the original genlop > calculation and yours, and how long did it actually take? and what is the > output of 'genlop -t '. $ genlop -t gcc -f /mnt/rescue/var/log/emerge.log using logfile /mnt/rescue/var/log/emerge.log * sys-devel/gcc Sat Apr 16 10:07:24 2016 >>> sys-devel/gcc-4.9.3 merge time: 25 minutes and 41 seconds. Sun May 15 10:03:34 2016 >>> sys-devel/gcc-4.9.3 merge time: 12 minutes and 19 seconds. Sun Jul 31 11:33:57 2016 >>> sys-devel/gcc-4.9.3 merge time: 17 minutes and 58 seconds. This is a fairly new box, so there aren't many records yet. The last one above is the one I was talking about. I can't tell you how long I thought it would take - sorry. > I think it would be inaccurate in most cases (at least it is for me) but > if you think the calculation is wrong you should file a bug. For me it > seems reasonably accurate por packages with consistent build times. Just those three samples are enough to confirm the inaccuracy, but I don't really expect anything much better given all the factors we've mentioned. I was just curious. Thanks for your interest though. -- Rgds Peter
Re: [gentoo-user] Genlop oddity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 07/31/2016 12:14 PM, Peter Humphrey wrote: > On Sunday 31 Jul 2016 10:48:33 Fernando Rodriguez wrote: >> On 07/31/2016 06:47 AM, Peter Humphrey wrote: >>> I've just encountered something I can't explain. Can anyone here? >>> >>> ~ $ genlop -c -f /mnt/rescue/var/log/emerge.log >>> using logfile /mnt/rescue/var/log/emerge.log >>> Currently merging 281 out of 287 >>> * net-libs/gnutls-3.3.24 >>>current merge time: 1 minute and 44 seconds. >>>ETA: less than a minute. >>> Currently merging 264 out of 287 >>> >>> * sys-devel/gcc-4.9.3 >>>current merge time: 7 minutes and 12 seconds. >>>ETA: 3 minutes and 13 seconds. >>> >>> $ genlop -c -f /mnt/rescue/var/log/emerge.log >>> using logfile /mnt/rescue/var/log/emerge.log >>> Currently merging 264 out of 287 >>> * sys-devel/gcc-4.9.3 >>>current merge time: 15 minutes and 19 seconds. >>>ETA: 3 minutes and 41 seconds. >>> >>> How is it possible for genlop's reported ETA to increase while its time >>> spent so far also increases? > > ...or to change at all, for that matter. > >> It is an estimate (a prediction) so it's subject to change. > > Yes, but the estimate is derived from an average of the durations found in > the log file, so it can't change until the emerge is complete. I haven't looked at the code but if it changes I think it's doing more than just averaging previous builds. It can and should try better than that. Even if all it has to work with is the logs a simple algorithm could (once it starts taking longer) look at the build time differences and come up with a better guess. My guess it's doing something along those lines. After a bit it just says "anytimme now." >>> Could the concurrent gnutls merging have affected it? Surely not. >> >> Probably, I don't know if genlop takes that into account. But there's >> countless factors that affect build time that it's practically impossible >> to predict accurately. So it means nothing. > > I think that's too pessimistic. If all your emerges have -j1, the accuracy > is pretty good, or it used to be in the days when I did that. The major > factor affecting the durations nowadays is -j > 1, so any other packages > could be emerging at the same time, thus skewing the log record. > > Another, even bigger log skewer is my practice, if I need an emerge -e, of > doing it in two stages: -eB first, then reboot to a minimal system and -eK. > That avoids things like kdelibs being different when I come to reboot the > system next time and hanging up during shutdown. > There's the -j option, there's distcc, ccache, etc. Many system settings, the amount of junk in /tmp if it's tmpfs, etc etc. Even in the simplest of cases using -j1 and controlling everything else some versions of the same package take significantly longer than others, what you have installed on your system can affect compile times, some compiler versions are faster, etc. etc. For me they're totally useless. gcc compile time ranges from 24mins-13hrs. Firefox 24mins-1day 3hrs. LibreOfffice 5hrs-14hrs. >> Does that estimate even look reasonable to you? >> gnutls compiles in a few minutes but gcc takes significantly longer for >> me. > No, as I said, genlop's own estimates are poor nowadays, but I can make a > rough estimate myself from comparing "genlop -c" with "genlop -t | > grep minutes". Just out of curiosity what are the differences between the original genlop calculation and yours, and how long did it actually take? and what is the output of 'genlop -t '. I think it would be inaccurate in most cases (at least it is for me) but if you think the calculation is wrong you should file a bug. For me it seems reasonably accurate por packages with consistent build times. - -- Fernando Rodriguez -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJXnoEnAAoJEPbOFX/5Ulwc3c4P/i26UPhwodsQ2TMS28fUyv4q BAyMvxGZG/nACbmzAYwPPbT8cZwIQomXhfuozokWGFsRzWLFdNms73HQ0YoQokeV Q8fHBolQdT3UyNgoROcDn7MJopr0QXWcZ2sWSpLFm2jZVmzz01d4ZkrQ2rXD22q/ 6qK3293svTfHpsft0h9ADeFW5r8BJ8BuLT+Q1TNHFQETILUHxmB34R7cHyJ6eMYX B0vGUXVSb1yYPReNKM3g+ok0KHMh5Q8iUGW44pzXLEg2Iw1Rnk2nwb7SYj0FjyzX rFreLC5iiVVPRLUkI9NKn6k3Mg3y3TIn8fNqI44r8RMMt8o96sXKGsUIsWamXBzp Sgorb+hvzTmiffD9Lk+1/reA5ne9nWfXGYlNfJY+JKgF0peGSOiWQ90w/T1x97si tYs1MqqMnvna+vkDW+lZZcditRE1NnLpmSIkHtvBuPi0XcjeiRNlmUEI6s6k9ZPO vTn0obIyT9f6XvUDNWZw5XlAZaP+w0Tf9r2mV6gJrx11OpjiKlEvWSN/F+rvRlsh 2Ki+QdrCjHpfnPbyQirUzfMeY2Ch4qzo8Ucnz/mJntE/4D9jPcmXisHV9k6KpYuX phE6m+DmU48QRys1pbh9prQTtvIPDm9LkuG2Hc5XJq+q+nAGeMUIH6SskCtE1Lsy 5UFWcFsuzZLAm7ShTAuZ =TiEU -END PGP SIGNATURE-
Re: [gentoo-user] Genlop oddity
On Sunday 31 Jul 2016 10:48:33 Fernando Rodriguez wrote: > On 07/31/2016 06:47 AM, Peter Humphrey wrote: > > I've just encountered something I can't explain. Can anyone here? > > > > ~ $ genlop -c -f /mnt/rescue/var/log/emerge.log > > using logfile /mnt/rescue/var/log/emerge.log > > Currently merging 281 out of 287 > > * net-libs/gnutls-3.3.24 > >current merge time: 1 minute and 44 seconds. > >ETA: less than a minute. > > Currently merging 264 out of 287 > > > > * sys-devel/gcc-4.9.3 > >current merge time: 7 minutes and 12 seconds. > >ETA: 3 minutes and 13 seconds. > > > > $ genlop -c -f /mnt/rescue/var/log/emerge.log > > using logfile /mnt/rescue/var/log/emerge.log > > Currently merging 264 out of 287 > > * sys-devel/gcc-4.9.3 > >current merge time: 15 minutes and 19 seconds. > >ETA: 3 minutes and 41 seconds. > > > > How is it possible for genlop's reported ETA to increase while its time > > spent so far also increases? ...or to change at all, for that matter. > It is an estimate (a prediction) so it's subject to change. Yes, but the estimate is derived from an average of the durations found in the log file, so it can't change until the emerge is complete. > > Could the concurrent gnutls merging have affected it? Surely not. > > Probably, I don't know if genlop takes that into account. But there's > countless factors that affect build time that it's practically impossible > to predict accurately. So it means nothing. I think that's too pessimistic. If all your emerges have -j1, the accuracy is pretty good, or it used to be in the days when I did that. The major factor affecting the durations nowadays is -j > 1, so any other packages could be emerging at the same time, thus skewing the log record. Another, even bigger log skewer is my practice, if I need an emerge -e, of doing it in two stages: -eB first, then reboot to a minimal system and -eK. That avoids things like kdelibs being different when I come to reboot the system next time and hanging up during shutdown. > Does that estimate even look reasonable to you? > gnutls compiles in a few minutes but gcc takes significantly longer for > me. No, as I said, genlop's own estimates are poor nowadays, but I can make a rough estimate myself from comparing "genlop -c" with "genlop -t | grep minutes". -- Rgds Peter
Re: [gentoo-user] Genlop oddity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 07/31/2016 06:47 AM, Peter Humphrey wrote: > Hello list, > > I've just encountered something I can't explain. Can anyone here? > > ~ $ genlop -c -f /mnt/rescue/var/log/emerge.log > using logfile /mnt/rescue/var/log/emerge.log > > Currently merging 281 out of 287 > > * net-libs/gnutls-3.3.24 > >current merge time: 1 minute and 44 seconds. >ETA: less than a minute. > > Currently merging 264 out of 287 > > * sys-devel/gcc-4.9.3 > >current merge time: 7 minutes and 12 seconds. >ETA: 3 minutes and 13 seconds. > > $ genlop -c -f /mnt/rescue/var/log/emerge.log > using logfile /mnt/rescue/var/log/emerge.log > > Currently merging 264 out of 287 > > * sys-devel/gcc-4.9.3 > >current merge time: 15 minutes and 19 seconds. >ETA: 3 minutes and 41 seconds. > > How is it possible for genlop's reported ETA to increase while its time > spent so far also increases? It is an estimate (a prediction) so it's subject to change. > Could the concurrent gnutls merging have > affected it? Surely not. Probably, I don't know if genlop takes that into account. But there's countless factors that affect build time that it's practically impossible to predict accurately. So it means nothing. Does that estimate even look reasonable to you? gnutls compiles in a few minutes but gcc takes significantly longer for me. > > Nothing else was running, apart from some file copying between external > disks. > - -- Fernando Rodriguez -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJXng/BAAoJEPbOFX/5Ulwcaw0P/07UJDhxAOhNp3NAf8awSMcR L8dIzDmssiABhyCH5GLCafSDujNelooOxvyy+Skf2BbNMSQk2gOiAOxjMQZSOmzC Sa8zh4PhfyV7jtsKLzjFFpuDHMJTH5Px9N3tvUfRuPSh6gtsmb1W+sD6AZr82KQI VwgH+QDLlO3jYkZGgyryIJXcL69QP1pspfBJ08iJk8y8hXbPla55aEmmrEI5HUsT 4LdineirvKYgDpMpErOoMgkZGy4BrUvPwVKyFmsHiB8byHl/Hcvsnb8tcyrrMpJ3 PU2QPpa/EcBiljlBiiE+ApSJHwOuv/ZMhltfYki/W1kdQHB9THN7sEdpDcjZUCs6 lKEAPe35nOizNoCMhHo1zFOB/5K6f2oLKdz/WP910eJ4CZkgWYyHvRCZJMN6jWZ1 AkwG9hcSQ+N7FAxXIQw15NLA9uky/vFOueu/NupUR6cJH1CmIBKZYe63PuTik2o6 3GIes2lHRUHroyCLnsfc9hxILXrzKkHEwFlgrtr6PhPPkchgqxBSHuB5hY5HfM1P cLgj9bj92sZLqwmImK/IRLn964119me2VEg8m1zYEAZLKuDsuAPrw2QtwO1W1APr ztok+sHWu51y36iC0oFBDuWKZG+612J1RfRZGgFXh0Vanh/dRjhLuUY8yuHB+Cka yEPhTh2DnZ8/MViBt1kt =a3yM -END PGP SIGNATURE-
