On 11/14/2011 03:17 PM, Duncan wrote: > Zac Medico posted on Mon, 14 Nov 2011 07:22:19 -0800 as excerpted: > >> On 11/14/2011 12:47 AM, Dirkjan Ochtman wrote: >>> On Mon, Nov 14, 2011 at 02:59, Zac Medico <[email protected]> wrote: >>>> Well, it's much easier to gather interest and get feedback if we >>>> deploy the change and ask questions later. >>> >>> I like the new output, but find it kind of annoying that there's very >>> little feedback on how far the progress is within a single job. Perhaps >>> we could show the currently executing ebuild phase in order to give a >>> little more feedback? Maybe most packages are completely dominated by >>> src_compile(), but for smaller packages it would be helpful, IMO. >> >> I think that would be an interesting option. I imagine that it would be >> too much information for many people, so I don't think that we'd want to >> enable it by default. > > Thanks for thinking "out of the box". =:^) > > This is one thing that I've found frustrating with parallel emerging, > myself. > > The first thing I discovered were that some phases (at least the merge > phase, and I believe install) aren't counted, so the numbers don't > entirely add up (complete + running + not-started + failed <> total). > Thus, some packages appear to be simply sitting there doing nothing.
This behavior is discussed in the "When packages are built in parallel with the --jobs option, why aren't some packages installed immediately after they have finished building? They are installed only after a long delay." section of the FAQ [1]. > When a whole bunch of packages are in that state, it appears emerge is > doing nothing but consuming cycles and real-time, for no visible reason > at all. I haven't noticed anything like that. You can send a SIGUSR1 signal to the emerge process and that will cause it drop to a pdb shell so you can poke around and see what's happening. > Since for a number of packages (I seem to notice it most on kde packages, > but perhaps it's all C++ or most CMAKE or some such, plus of course > primarily data packages that don't have a significant build phase but > install many or large files) the install phase can be as intense as the > build phase if not more so, so if some phases aren't counted in > "running", then I'd definitely prefer they be counted /somewhere/. The src_install phase is counted along with all of the earlier phases. > Of course, listing the number of packages in each phase (which in single- > mode would by definition list the phase the single package is in) would > cure the above issue at the same time. > > Another problem, but one I'm not sure how to fix (tho the interactive > display would offer opportunities here), is that it'd be nice to get a > list of completed packages, in-process packages, and still not started > packages. That doesn't so easily fit in a neat summary, but as noted, > the interactive proposal introduces quite some opportunity, here. > > Alternatively, since often the real info desired is the status of a > particular package, it'd be useful to have either an interactive command > or a simple separate querying command, that can be run to query the > status of a particular package. > > *** Which brings up an alternative to the whole interactive emerges idea > as well -- what about a separate command, say "emerging", that when run, > would simply output a bunch of detail on any pending emerges? > > This would certainly offer a safer initial implementation, as well, since > it's less likely to break the current setup due to bugs, etc. Another > noted benefit is that it leaves the existing defaults alone, so it's not > going to be threatening anyone's sacred cows (or scripted assumptions) in > terms of how portage has always behaved. =:^) > > > So, perhaps improve the current "quiet" display marginally, either > including all phases in "running" or adding another listing so the > numbers add up, but more importantly... > > *** Add a new "emerging" (name up for bikeshedding) command, that > displays a nice multiline summary, perhaps something like this (the > second emerge command was for a failure in the first, with a second > attempt made before completion of the main emerge @world has completed): > > Current emerge status: > > 2 emerge command(s) running. > 7 emerge jobs outstanding. > > Commands: > > ** emerge --upgrade --deep --newuse --keep-going @world > > Status: > > 123 of 257 jobs complete, 1 failed, 6 running, 120 not yet started, > 7 dependencies removed due to failures. > > Running (6): > > 1 pkgsetup (cat-egory/package-ver) > > 1 configure (cat-egory/package-ver, cat-egory/package-ver) > > 2 build (cat-egory/package-ver, cat-egory/package-ver) > > 1 install (cat-egory/package-ver) > > 1 merge (cat-egory/package-ver) > > Failed (1): > > dev-libs/boost-1.47.0-r1 > > Complete (123): > > <list in nice columns> > ... > > Not started (120): > > <list in nice columns> > ... > > Removed dependencies due to failure (7): > > <list in nice columns> > ... > > > ** emerge -u1 boost > > Status: > > 0 of 1 jobs complete, 1 running. > > Running (1): > > 1 build (dev-libs/boost-1.47.0-r1) > Obviously the first implementation might not have all the information > above, and there might have been some nice information I missed and other > information that could be displayed better, but it's a very cool idea > even if I /do/ say so myself. =:^) > > Note that zeroed-out listings aren't displayed, so the the --oneshot > doesn't list failed, not yet started, etc, and inactive phases are also > not displayed. > > Also note how easy this sort of detailed display would make it to retry > failed jobs and the resulting removed dependencies, once the failure has > been resolved. > > "emerging" could have a "repeat" mode as well as one-shot, with a polling- > delay parameter set to something sane like 30 or 60 seconds by default. > That way, I wouldn't have to feed it to "watch" =:^) I'm not sure how many people would use an interface like that in practice, but I wouldn't be opposed to adding support for something like that. Since I'm not really interested in using an interface like that myself, so I don't feel inspired to implement it myself. > Because this is a separate command, worries about TMI/TLI should be > eliminated. Additionally, it should go some way toward easing the whole > quiet/noisy debate as well, since there's now another command available > with an intermediate level of information. > > Something like this: > > edisplaylog > If we then throw in one more tool, I'll call it edisplaylog for lack of a > better name, that reads make.conf to find the elog dir, and can > automatically pick out the latest log for a package, thus eliminating the > trouble of doing it manually, that should help eliminate another > objection to quiet-by-default. So... > > edisplaylog boost > > ... would find the last elog for boost and display it. > > To make things easy for the user, simply ... > > edisplaylog > > ... could be made to filetime search and display the last active log in > tail -f mode. If after say a couple seconds of no activity, it re-polled > the log dir and switched to displaying a different log (have it list the > log it's displaying at the top) if it was newer, that could pretty much > replace "noisy" mode entirely. =:^) > > Obviously running simply "edisplaylog" by itself during a parallel-jobs > emerge would be nearly as confusing as portage not automatically going > into quiet mode for parallel emerging would be, tho not quite since it > would display the logfile name at the top, but "edisplaylog <pkg>" would > still be very useful, ESPECIALLY for those using --keep-going, so they > could track down and fix whatever failed a build before the parallel > emerge spit out the results at the end. > > Talking about which... an "edisplaylog failed" mode, which would > automatically find the last failure and displayed the log for it, could > be very helpful as well. =:^) That sounds pretty handy. > Obviously all these ideas entail a lot of coding, and I realize it's easy > to sit here and make requests without doing the coding, but implementing > them as separate commands first and incremental implementation should > help a lot, and I hadn't seen anything like this proposed yet, so... > > I'll close with a thanks to Zac and everyone else who has already made > portage the great tool it is today, because without them, we'd obviously > not be having this discussion in the first place. =:^) > [1] http://www.gentoo.org/proj/en/portage/doc/faq.xml#doc_chap1_sect9 -- Thanks, Zac
