Re: [gentoo-dev] Things one could be upset about
On 01/22/2015 10:19, Peter Stuge wrote: > Joshua Kinard wrote: >> Using seed stage3 stages I built 6 months ago (but never released due >> to getting sidetracked), I run into errors like this: >> >> !!! Multiple package instances within a single package slot have been pulled >> !!! into the dependency graph, resulting in a slot conflict: >> >> dev-lang/perl:0 >> >> (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) >> pulled in by >> =dev-lang/perl-5.20* required by >> (virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for >> merge) >> ^ ^ >> (and 16 more with the same problem) >> >> (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) >> pulled in by >> dev-lang/perl:0/5.18=[-build(-)] required by >> (dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed) >> >> =dev-lang/perl-5.18* required by >> (virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed) >> ^ ^ >> (and 2 more with the same problems) >> >> It's hard to read mess like that and trace down the offending package, >> fix it, and make catalyst happy. > > Lots of dev-perl packages have specific minor version dependencies on > dev-lang/perl, maybe because sometimes the package is included in perl > and sometimes not. It's a f*ing mess. You have to look up all your > installed dev-perl packages manually and find which ones are either > too old to know about perl-5.20 or not compatible with it, and then > you have to unmerge those manually. In the past, it's been possible to have Portage deal with the updates to Perl, but only as long as you hit all of the packages in the same update run to satisfy the dependency chain. Newer portage seems to not do that anymore. But that output is horrible. Even with the color coding, it's not directly apparent which package is the problem package. I once had a Perl update issue bad enough that I removed all perl packages entirely, then remerged them from scratch. Took a while, but it fixed things. >> Kinda defeats the purpose of catalyst in the first place. > > The proper way is to build stage1+2+3 yourself, then this mess > doesn't happen. But like you I too cheat a little, and have to deal > with the mess. Well, I was trying to do it the right way by going stage1 -> stage2 -> stage3. I was using a stage3 that I built over the summer as the seed stage for the new stage1 when I started running into problems with Perl. I finally fixed that, got stage1 built, then got bit by Bug #447126 while trying to build the stage2. So now, I have to start a stage2 run, then after the unpack (but before catalyst drops into the chroot), edit the chroot's make.conf and remove sandbox from FEATURES, which is apparently part of the problem. Just irritating. And I know I'm earning no sympathy when I point out that my build machines (an Octane and an Onyx2) aren't the fastest things on the planet, nor the most power efficient (1 kW between the both of them). But I'd at least like to waste that power on actual compile jobs, not watching emerge's little spinner all the time as I try to fix various dependency bugs or other oddities that seemingly came out of nowhere (because the summertime stage runs were flawless in execution). -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Re: [gentoo-dev] Things one could be upset about
On Sat, 24 Jan 2015 21:54:06 +0400 Alexey Mishustin wrote: > 2015-01-20 14:42 GMT+04:00 Róbert Čerňanský : > > On Tue, 20 Jan 2015 11:08:19 +0300 > > Andrew Savchenko wrote: > > > >> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote: > >> > On Tue, 20 Jan 2015 00:14:29 +0300 > >> > Andrew Savchenko wrote: > >> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > >> > For example, lets have following packages: > >> > > >> > - libbar > >> > - libfoo with IUSE=bar, which depends on: bar? ( libbar ) > >> > - foo, which depends on: libfoo[bar] > > [...] > >> > New behaviour with automatic USE dependencies resolution: > >> > > >> > emerge -av foo > >> > [ebuild N ] libbar > >> > [ebuild N ] libfoo +bar > >> > [ebuild N ] foo > >> > > >> > Now, you can hit Y if you agree what portage wants to do or N and > >> > not to install foo at all. > >> > >> And if I don't agree? What if for some reason I don't want to > >> have libfoo[bar] on my system. What If preferred solution will > >> be not to use libbar at all and to use libclue instread? > > > > In this example, if you do not agree, you have no other option how > > to install foo (with or without automatic USE deps). Because foo > > depends on libfoo[bar] unconditionally. > > Perfect! May be I will prefer to refuse to install that package, after > seeing its dependencies. > > >> Yet again, Gentoo is all about choise. If someone wants that > > > > I agree, but I must say I am quite stunned that there are strong > > voices against it. I somehow thought that edit the overgrowing > > package.use file upon each emerge world annoys anyone the same as > > me. > > But for me this is one of the most useful and convenient options in > Gentoo. Yes, I do edit package.use almost every emerge world. And I > like to do it. And I don't want to delegate this right to any program > - portage, or any other. You would not loose that. First of all, portage would do its changes elsewhere (in /var). Secondly, your settings in package.use would still be respected. But instead of portage telling you to edit package.use it would show you required changes in emerge -av output. If you do not like them then hit N and edit package.use, masks or whatever. In case of conflict, i.e. if portage would want to set a USE flag for a package which you have disabled in package.use then it would fail (similarly as it fails when some package has testing keyword or is masked). So if I have 'net-print/cups -usb' in package.use the automatic USE depencencies would _never_ enable usb for cups. But if I have -python in _make.conf_ and want to emerge app-mobilephone/wammu which depends on app-mobilephone/gammu[python] then portage would be able to enable python for gammu. The intention behind this automatic USE dependencies is to edit package.use only when user wants, not when portage needs it. Robert -- Róbert Čerňanský E-mail: ope...@tightmail.com Jabber: h...@jabber.sk
Re: [gentoo-dev] Things one could be upset about
2015-01-20 14:42 GMT+04:00 Róbert Čerňanský : > On Tue, 20 Jan 2015 11:08:19 +0300 > Andrew Savchenko wrote: > >> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote: >> > On Tue, 20 Jan 2015 00:14:29 +0300 >> > Andrew Savchenko wrote: >> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: >> > For example, lets have following packages: >> > >> > - libbar >> > - libfoo with IUSE=bar, which depends on: bar? ( libbar ) >> > - foo, which depends on: libfoo[bar] > [...] >> > New behaviour with automatic USE dependencies resolution: >> > >> > emerge -av foo >> > [ebuild N ] libbar >> > [ebuild N ] libfoo +bar >> > [ebuild N ] foo >> > >> > Now, you can hit Y if you agree what portage wants to do or N and >> > not to install foo at all. >> >> And if I don't agree? What if for some reason I don't want to >> have libfoo[bar] on my system. What If preferred solution will >> be not to use libbar at all and to use libclue instread? > > In this example, if you do not agree, you have no other option how to > install foo (with or without automatic USE deps). Because foo depends > on libfoo[bar] unconditionally. Perfect! May be I will prefer to refuse to install that package, after seeing its dependencies. >> Yet again, Gentoo is all about choise. If someone wants that > > I agree, but I must say I am quite stunned that there are strong voices > against it. I somehow thought that edit the overgrowing package.use > file upon each emerge world annoys anyone the same as me. But for me this is one of the most useful and convenient options in Gentoo. Yes, I do edit package.use almost every emerge world. And I like to do it. And I don't want to delegate this right to any program - portage, or any other. -- Regards, Alex
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 13:44:21 +0100 Dirkjan Ochtman wrote: > Also, I hate something like > "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']". > What the hell kind of warning is that? I guess maybe these are the > results of USE_EXPAND trickery and what not, but it would sure be nice > to have something more readable. https://bugs.gentoo.org/show_bug.cgi?id=534022 jer
Re: [gentoo-dev] Things one could be upset about
Joshua Kinard wrote: > Using seed stage3 stages I built 6 months ago (but never released due > to getting sidetracked), I run into errors like this: > > !!! Multiple package instances within a single package slot have been pulled > !!! into the dependency graph, resulting in a slot conflict: > > dev-lang/perl:0 > > (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) pulled > in by > =dev-lang/perl-5.20* required by > (virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for > merge) > ^ ^ > (and 16 more with the same problem) > > (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) pulled > in by > dev-lang/perl:0/5.18=[-build(-)] required by > (dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed) > > =dev-lang/perl-5.18* required by > (virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed) > ^ ^ > (and 2 more with the same problems) > > It's hard to read mess like that and trace down the offending package, > fix it, and make catalyst happy. Lots of dev-perl packages have specific minor version dependencies on dev-lang/perl, maybe because sometimes the package is included in perl and sometimes not. It's a f*ing mess. You have to look up all your installed dev-perl packages manually and find which ones are either too old to know about perl-5.20 or not compatible with it, and then you have to unmerge those manually. > Kinda defeats the purpose of catalyst in the first place. The proper way is to build stage1+2+3 yourself, then this mess doesn't happen. But like you I too cheat a little, and have to deal with the mess. //Peter
Re: [gentoo-dev] Things one could be upset about
On 01/17/2015 08:21, Pacho Ramos wrote: > El sáb, 17-01-2015 a las 13:44 +0100, Dirkjan Ochtman escribió: > [...] >> Also, I hate something like >> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']". >> What the hell kind of warning is that? I guess maybe these are the >> results of USE_EXPAND trickery and what not, but it would sure be nice >> to have something more readable. > > Yeah, sometimes the output are really fat, not sure if some heuristic > could be done to, for example, collate the exact same errors that are > coming from every single subprofile. I've been spending the better half of the last two days trying to kickstart catalyst runs on two of my SGI systems, one doing o32 and the other n32. Using seed stage3 stages I built 6 months ago (but never released due to getting sidetracked), I run into errors like this: !!! Multiple package instances within a single package slot have been pulled !!! into the dependency graph, resulting in a slot conflict: dev-lang/perl:0 (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) pulled in by =dev-lang/perl-5.20* required by (virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for merge) ^ ^ (and 16 more with the same problem) (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) pulled in by dev-lang/perl:0/5.18=[-build(-)] required by (dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed) =dev-lang/perl-5.18* required by (virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed) ^ ^ (and 2 more with the same problems) It's hard to read mess like that and trace down the offending package, fix it, and make catalyst happy. Got bit by the splitting of libltdl and libtool as well. Several packages included a block on <=libtool-2.4.2, which was in my stage3 builds from last summer. Not an easy way to work around those in catalyst. Eventually just unpacked the seed stage3 on both systems, updated libtool/libltdl, repacked them, and used those as the as seed stages. Kinda defeats the purpose of catalyst in the first place. Looks like I have to repeat for perl now, which seems to do this every major update. -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org 4096R/D25D95E3 2011-03-28 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Re: [gentoo-dev] Things one could be upset about
On Sun, 18 Jan 2015, Patrick Lauer wrote: On Saturday 17 January 2015 14:00:34 William Hubbs wrote: On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote: On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote: * Stage3 archives are too fat See https://bugs.gentoo.org/show_bug.cgi?id=531632 We're now shipping three python versions and glib for extra fun! Fix: Motivate releng to stop being silly Why the heck do we ship both 3.3 and 3.4? I forget the exact situation with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only is a great option if that remains the default after installation (although it would be fine for just the initial stages). I'm going to be very blunt. I am sick of the finger being pointed only at releng for this. The issue is package dependencies. If even one package in the tree has a dumb dependency on python, e.g. dev-lang/python, it will pull in all stable versionf of python. Only if you absolutely insist that releng can never deviate from tree. What RelEng has insisted in is building what we have in the tree - which, curiously enough, is a good way to test the tree. Anyway, I've started working on a portdir config for the default stages, not because of this, but because of the caps issue (bug 531788). Which is a silly idea, see bindist useflag, which is locally enabled for stage building and then removed. Oh wait, it's not removed because we can't deviate while deviating. So users regularly find ssl 'broken' ... Building stages with USE="bindist" isn't a question of being silly but of making sure we don't give anyone a reason to "sue" Gentoo. We don't want to force USE="bindist" in the released stages, but that means we add one more workaround to catalyst code or we provide a way to specify which use flags we want to use for building the stage and which we want to be kept in make.conf. It took me about 2h of fiddling around to find a few spots where stage3 has useless bloat: - pkgconfig pulling in 30MB of glib - python installing tests (that's 3x 25MB now ...) internal-glib is not something that should be used in the stages. - having more than one python installed (which is not really absolutely necessary, and could easily be reduced to one) If your system has 3 python versions installed because the tree deps make portage install 3 versions, you should expect that to happen in your stages. Since I'll have to work with portdir to address the caps issue above, I'll also take care of this, but if you want to blame someone about this, releng doesn't "maintain the tree". Also, by adding this to the releng repo, that means we're going to have more work as we'll have to track new python versions. Out of 700MB on-disk I could prune about 150MB - about 20%, without affecting functionality It's not about pointing a finger, it's about fixing issues when they are easy to fix and not hiding behind a fake complexity argument. (I would fix things, if I had access and/or certainty that patches provided would be integrated ...) I don't recall ever having seen a patch from you to catalyst, so before you "suggest" we don't want to accept your patches, you may want to send one ;-) Have fun, Patrick Have fun, Jorge
Re: [gentoo-dev] Things one could be upset about
On Tue, 20 Jan 2015 12:01:13 +0100 Luca Barbato wrote: > On 17/01/15 16:03, Ciaran McCreesh wrote: > > On Sat, 17 Jan 2015 22:59:08 +0800 > > Patrick Lauer wrote: > >>> The problem isn't the constants, though. The problem is the > >>> resolution algorithm. There's not much point tweaking performance > >>> until the resolver is fixed to produce a correct answer... > >> > >> Patches welcome :) > > > > If I send you a patch that updates all the documentation to use > > Paludis rather than Portage, will you welcome it? > > > > If you rewrite paludis in C99 or equally language with non-brittle > runtime (libc++ isn't really viable yet =/) I would seriously > consider it. There's always -static-libstdc++. Although Gentoo is still the only distribution which has trouble with having more than one libstdc++ visible at a time... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On 19/01/15 16:47, hasufell wrote: I think you forgot an important point: * lack of practical QA: no review workflow and no appropriate tools for reviewing I could start a long text block about why reviewing is mandatory for QA, but let's just think about it this way: What do you think would happen if the linux kernel switched to CVS and gave the most active 250 collaborators direct push access to the main Linus repository? I hope greg k-h does not read this. He'd probably get a heart attack. Also: people seem to think we don't have enough manpower for a review workflow. No, it's really the other way around. If you make collaboration difficult, then you need a lot more manpower. I already pointed out that there are _not_ good review tools. There are not for a by-email workflow we have in Libav, there aren't really for a tool-mediated workflow we could have in Gentoo. I have no problems in devoting some time on preparing a tool suited for our purpose (once we switch to git), but I'd need more volunteers to help me with it. lu
Re: [gentoo-dev] Things one could be upset about
On 17/01/15 16:03, Ciaran McCreesh wrote: On Sat, 17 Jan 2015 22:59:08 +0800 Patrick Lauer wrote: The problem isn't the constants, though. The problem is the resolution algorithm. There's not much point tweaking performance until the resolver is fixed to produce a correct answer... Patches welcome :) If I send you a patch that updates all the documentation to use Paludis rather than Portage, will you welcome it? If you rewrite paludis in C99 or equally language with non-brittle runtime (libc++ isn't really viable yet =/) I would seriously consider it. lu
Re: [gentoo-dev] Things one could be upset about
On Tue, 20 Jan 2015 11:08:19 +0300 Andrew Savchenko wrote: > On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote: > > On Tue, 20 Jan 2015 00:14:29 +0300 > > Andrew Savchenko wrote: > > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > > > > From my point of view it would do much help if portage resolves > > > > USE dependencies automatically [...] > > > As the user the last thing I want is to have some USE flags > > > changed without my permission [...] > > Is is of course perfectly fine to have that option disabled by > > default. But I am afraid that I do not quite understand yours and > > Daniel's concerns. If I paraphrase your statement with regards to > > current dependency handling, it is as if you were saying: "I do not > > want portage to install any package automatically by its own, I want > > to install all the dependencies explicitly." > > The paraphrasing above is wrong. What I want to say is "I don't want > portage to automagically change _functionality_ of my packages on > its own, even in order to solve dependencies." > > > Why we allow portage to install dependencies and still have things > > under control? Well we have --pretend and --ask options and we can > > see what would/will be done. This would be the same with USE flags. > > You would see in the --pretend output which flags would be changed. > > No this wouldn't be the same. USE flags are _not_ equal to package > dependencies, sometimes they will not trigger dependency change at > all. USE flags are about _functionality_, dependencies are about > the _means_ to implement that functionality (as well as base > functionality of a package). I see your point about functionality. There are USE flags that _changes_ functionality (like threads) and there are others which _adds_ functionality (like for example speex adds possibility to play files in that particular format in mplayer). The former I consider similar to package dependencies because they too can add functionality to the system (for example if ruby is installed as dependency it adds possibility to execute ruby scripts) same as those USE flags adds functionality to an application. However, even if portage wants me to change USE flag that will change functionality, in 99% of time I end up adding to my package.use what it wants, since my goal is to install some application or update my applications. So just reviewing the changes that portage wants to do and hit Y. And in those rare cases when I really do not want some flag enabled or some package installed I hit N and tweak things. > > To match the package dependency handling, there should be also an > > option equivalent to --depclean. It would revert the USE changes > > which were done because of dependencies requirements if no longer > > needed. > > This will dramatically increase complexity of dependencies metadata > as well as of algorithms to handle it (and they are already slow), > with no clear benefits therefore. No, thanks. Are you talking about proposed new USE specific depclean option or emerge in general? If it is the new depclean command that would run slow then I am quite sure it will still be faster than me or anyone else going manually through package.use and for each and every USE flag there trying to remove it, run emerge, see if it passes, if not, add it back. If it is emerge in general that would be slower that it is still better than run it, see it fail and telling you what USE to change, you making the change, and run it again. > > For example, lets have following packages: > > > > - libbar > > - libfoo with IUSE=bar, which depends on: bar? ( libbar ) > > - foo, which depends on: libfoo[bar] [...] > > New behaviour with automatic USE dependencies resolution: > > > > emerge -av foo > > [ebuild N ] libbar > > [ebuild N ] libfoo +bar > > [ebuild N ] foo > > > > Now, you can hit Y if you agree what portage wants to do or N and > > not to install foo at all. > > And if I don't agree? What if for some reason I don't want to > have libfoo[bar] on my system. What If preferred solution will > be not to use libbar at all and to use libclue instread? In this example, if you do not agree, you have no other option how to install foo (with or without automatic USE deps). Because foo depends on libfoo[bar] unconditionally. If however foo would depend either on libfoo[bar] or libclue then the portage would do the same thing as today (I guess it would take the first dependency as mandatory or the one which is already installed if any.) In this case, yes, your preference might be libclue instead what portage chooses. But I see no difference in the way how user choose it comparing to todays '|| ( libfoo libclue )' -like dependencies. > World update on my systems usually involves about 2000 of packages. > It's already hard to read emerge -DNuav output in order to check > for new packages, dependencies, downgraded and removed packages. > Automagick dependency chan
Re: [gentoo-dev] Things one could be upset about
On Tue, 20 Jan 2015 06:51:01 +0100 Róbert Čerňanský wrote: > On Mon, 19 Jan 2015 20:51:31 + > Ciaran McCreesh wrote: > > > On Mon, 19 Jan 2015 21:44:25 +0100 > > Róbert Čerňanský wrote: > > > From my point of view it would do much help if portage resolves USE > > > dependencies automatically instead of telling the user to change USE > > > flags manually (I am talking about bug #258371). > > > > This is only possible in carefully selected circumstances, and to get > > it to work more generally would require a lot of hinting from package > > maintainers. > > But portage already knows that. It tells the user which USE flags > needs to be changed in order to emerge a package. It should just go > one step further - to make the proposed change happen by itself. And ofter it proposes multiple alternative ways to fix this. How do you suppose to select between multiple possible alternative solutions then? Another issues is that sometimes it will be preferred by the user to disable offending functionality at all. Good example here is sci-libs/hdf5. Set USE="cxx threads mpi" in make.conf. Have fun. Especially if in one application (depending on hdf5) you really need cxx support and in another one (also depending on hdf5) mpi support is really needed. In some cases it is preferred to disable hdf5 support at all. Best regards, Andrew Savchenko pgpcWxk85uSeF.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote: > On Tue, 20 Jan 2015 00:14:29 +0300 > Andrew Savchenko wrote: > > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > > > From my point of view it would do much help if portage resolves USE > > > dependencies automatically instead of telling the user to change USE > > > flags manually (I am talking about bug #258371). > > > > As the user the last thing I want is to have some USE flags changed > > without my permission ending up stuff I need to be omitted or stuff > > I don't want to see on my system to be installed. Of course if > > someone prefers USE flags to be randomly changed I don't mind if > > such option will exist (as proposed in bug #258371) as long as it > > is disabled by default. > > Is is of course perfectly fine to have that option disabled by > default. But I am afraid that I do not quite understand yours and > Daniel's concerns. If I paraphrase your statement with regards to > current dependency handling, it is as if you were saying: "I do not > want portage to install any package automatically by its own, I want > to install all the dependencies explicitly." The paraphrasing above is wrong. What I want to say is "I don't want portage to automagically change _functionality_ of my packages on its own, even in order to solve dependencies." > Why we allow portage to install dependencies and still have things > under control? Well we have --pretend and --ask options and we can > see what would/will be done. This would be the same with USE flags. > You would see in the --pretend output which flags would be changed. No this wouldn't be the same. USE flags are _not_ equal to package dependencies, sometimes they will not trigger dependency change at all. USE flags are about _functionality_, dependencies are about the _means_ to implement that functionality (as well as base functionality of a package). > To match the package dependency handling, there should be also an > option equivalent to --depclean. It would revert the USE changes > which were done because of dependencies requirements if no longer > needed. This will dramatically increase complexity of dependencies metadata as well as of algorithms to handle it (and they are already slow), with no clear benefits therefore. No, thanks. > For example, lets have following packages: > > - libbar > - libfoo with IUSE=bar, which depends on: bar? ( libbar ) > - foo, which depends on: libfoo[bar] > > USE flag bar is not enabled. You want to install application foo. > > Current behaviour: > > # emerge -av foo > ... > Required USE changes: > libfoo +bar > ... (sorry for not exact emerge output but you get the idea) > > Now, you either fire up your favorite editor and add "libfoo +bar" to > /etc/portage/package.use, re-run emerge and have libbar installed even > you do not want it. The other option is not to install application > foo at all. > > If you choose to emerge it and after year or so you decide to unmerge > it because you do not need it anymore you still will have a leftover > in package.use file - the line "libfoo +foo" even if you run emerge > --depclean. > > New behaviour with automatic USE dependencies resolution: > > emerge -av foo > [ebuild N ] libbar > [ebuild N ] libfoo +bar > [ebuild N ] foo > > Now, you can hit Y if you agree what portage wants to do or N and not > to install foo at all. And if I don't agree? What if for some reason I don't want to have libfoo[bar] on my system. What If preferred solution will be not to use libbar at all and to use libclue instread? World update on my systems usually involves about 2000 of packages. It's already hard to read emerge -DNuav output in order to check for new packages, dependencies, downgraded and removed packages. Automagick dependency change will make this even harder, much harder because it will involve multiple additional emerge runs in order to deal with issues properly. And each run takes about half an hour. Yet again, Gentoo is all about choise. If someone wants that automagick to be implemented, go ahead and send patches for developers to review. But please keep it disabled by default, in most cases it will provide bizarre results anyway. (Think about conflicting alternatives, no sane way to prefer one above another automatically and a chain of such issues during world update.) Best regards, Andrew Savchenko pgpeQjP_x8K4S.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Tue, 20 Jan 2015 00:14:29 +0300 Andrew Savchenko wrote: > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > > From my point of view it would do much help if portage resolves USE > > dependencies automatically instead of telling the user to change USE > > flags manually (I am talking about bug #258371). > > As the user the last thing I want is to have some USE flags changed > without my permission ending up stuff I need to be omitted or stuff > I don't want to see on my system to be installed. Of course if > someone prefers USE flags to be randomly changed I don't mind if > such option will exist (as proposed in bug #258371) as long as it > is disabled by default. Is is of course perfectly fine to have that option disabled by default. But I am afraid that I do not quite understand yours and Daniel's concerns. If I paraphrase your statement with regards to current dependency handling, it is as if you were saying: "I do not want portage to install any package automatically by its own, I want to install all the dependencies explicitly." Why we allow portage to install dependencies and still have things under control? Well we have --pretend and --ask options and we can see what would/will be done. This would be the same with USE flags. You would see in the --pretend output which flags would be changed. To match the package dependency handling, there should be also an option equivalent to --depclean. It would revert the USE changes which were done because of dependencies requirements if no longer needed. For example, lets have following packages: - libbar - libfoo with IUSE=bar, which depends on: bar? ( libbar ) - foo, which depends on: libfoo[bar] USE flag bar is not enabled. You want to install application foo. Current behaviour: # emerge -av foo ... Required USE changes: libfoo +bar ... (sorry for not exact emerge output but you get the idea) Now, you either fire up your favorite editor and add "libfoo +bar" to /etc/portage/package.use, re-run emerge and have libbar installed even you do not want it. The other option is not to install application foo at all. If you choose to emerge it and after year or so you decide to unmerge it because you do not need it anymore you still will have a leftover in package.use file - the line "libfoo +foo" even if you run emerge --depclean. New behaviour with automatic USE dependencies resolution: emerge -av foo [ebuild N ] libbar [ebuild N ] libfoo +bar [ebuild N ] foo Now, you can hit Y if you agree what portage wants to do or N and not to install foo at all. After unmerging it you run emerge --depclean which removes libfoo with its changed USE flag and libbar, no leftovers. (In this case new --use-depclean command is not required since libfoo was removed.) If libfoo would not be removed because some other package needs it, then emerge --use-depclean would revert the +bar USE flag to its default state and re-emerge libfoo. Robert -- Róbert Čerňanský E-mail: ope...@tightmail.com Jabber: h...@jabber.sk
Re: [gentoo-dev] Things one could be upset about
On Mon, 19 Jan 2015 20:51:31 + Ciaran McCreesh wrote: > On Mon, 19 Jan 2015 21:44:25 +0100 > Róbert Čerňanský wrote: > > From my point of view it would do much help if portage resolves USE > > dependencies automatically instead of telling the user to change USE > > flags manually (I am talking about bug #258371). > > This is only possible in carefully selected circumstances, and to get > it to work more generally would require a lot of hinting from package > maintainers. But portage already knows that. It tells the user which USE flags needs to be changed in order to emerge a package. It should just go one step further - to make the proposed change happen by itself. Robert -- Róbert Čerňanský E-mail: ope...@tightmail.com Jabber: h...@jabber.sk
Re: [gentoo-dev] Things one could be upset about
On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > On Sat, 17 Jan 2015 18:33:45 +0300 > Andrew Savchenko wrote: > > > On Sat, 17 Jan 2015 14:45:51 + Ciaran McCreesh wrote: > > > The problem isn't the constants, though. The problem is the > > > resolution algorithm. There's not much point tweaking performance > > > until the resolver is fixed to produce a correct answer... > > > > Oh, this was discussed so many times already... There is NO single > > correct solution to such problems. And some mathematically correct > > solutions are impractical (e.g. half of the tree rebuild), so other > > ones which are good enough are preferred. As long as imperfect > > solution works fine, I'm ok with it. > > From my point of view it would do much help if portage resolves USE > dependencies automatically instead of telling the user to change USE > flags manually (I am talking about bug #258371). As the user the last thing I want is to have some USE flags changed without my permission ending up stuff I need to be omitted or stuff I don't want to see on my system to be installed. Of course if someone prefers USE flags to be randomly changed I don't mind if such option will exist (as proposed in bug #258371) as long as it is disabled by default. Best regards, Andrew Savchenko pgpqoWBdnzXUy.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/19/2015 12:44 PM, Róbert Čerňanský wrote: > On Sat, 17 Jan 2015 18:33:45 +0300 Andrew Savchenko > wrote: > >> On Sat, 17 Jan 2015 14:45:51 + Ciaran McCreesh wrote: >>> The problem isn't the constants, though. The problem is the >>> resolution algorithm. There's not much point tweaking >>> performance until the resolver is fixed to produce a correct >>> answer... >> >> Oh, this was discussed so many times already... There is NO >> single correct solution to such problems. And some mathematically >> correct solutions are impractical (e.g. half of the tree >> rebuild), so other ones which are good enough are preferred. As >> long as imperfect solution works fine, I'm ok with it. > > From my point of view it would do much help if portage resolves > USE dependencies automatically instead of telling the user to > change USE flags manually (I am talking about bug #258371). > > Robert > > As a user, the last thing I want happening is Portage making USE decisions for me. I'm completely in support of an emerge *flag*, but not doing it unconditionally. -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJUvXBBAAoJEJUrb08JgYgHqsQH/1m/Dgu547RTcooHhZ+B4gt1 FvjPGy1qEKB4W2ErxDj6J6TLlP09ASIiJ/7hrndKonDNd1aP4gAi7tKI5XzetWVt cWYG3UWLhxJRvMc2y7kbOyDSIy68Sz/r1Bruwymqdn+N6ooqnHVK252OJgaMGQHP aDa+ibNAywE7t/CTWS6rQU/ilEHsXIps+c4gmvEGv5iWiCKxlQF5fNKfWjOGEr9c NN23RaSEJj7BCEfFaFgmjd7P0akz/yzg/sr8xuaaEwUv5/KFJp7SI/Q/6GzG48rg H6TiNIYm3Gs0ucEWISCZx+qon5EmkkSREaQ5xeqBBRklNN63evH1pttjFg9rX6o= =RjLq -END PGP SIGNATURE-
Re: [gentoo-dev] Things one could be upset about
On Mon, Jan 19, 2015 at 4:47 AM, Jeroen Roovers wrote: > > repoman doesn't check reverse dependencies for the package you're > working on. > Indeed, it doesn't even check forward dependencies which are blockers. kmod-19 was just stabilized accidentally despite having a blocker on all stable versions of systemd. That resulted in a big scramble to backport a fix to systemd (and incidentally discover that openrc suffered from the same problem, thus resulting in yet another rushed fix). It would have been helpful if repoman could have noticed this. -- Rich
Re: [gentoo-dev] Things one could be upset about
On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote: > From my point of view it would do much help if portage resolves USE > dependencies automatically instead of telling the user to change USE > flags manually (I am talking about bug #258371). This is only possible in carefully selected circumstances, and to get it to work more generally would require a lot of hinting from package maintainers. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 18:33:45 +0300 Andrew Savchenko wrote: > On Sat, 17 Jan 2015 14:45:51 + Ciaran McCreesh wrote: > > The problem isn't the constants, though. The problem is the > > resolution algorithm. There's not much point tweaking performance > > until the resolver is fixed to produce a correct answer... > > Oh, this was discussed so many times already... There is NO single > correct solution to such problems. And some mathematically correct > solutions are impractical (e.g. half of the tree rebuild), so other > ones which are good enough are preferred. As long as imperfect > solution works fine, I'm ok with it. From my point of view it would do much help if portage resolves USE dependencies automatically instead of telling the user to change USE flags manually (I am talking about bug #258371). Robert -- Róbert Čerňanský E-mail: ope...@tightmail.com Jabber: h...@jabber.sk
Re: [gentoo-dev] Things one could be upset about
On Sun, Jan 18, 2015 at 01:21:56PM +0100, Dirkjan Ochtman wrote: > On Sat, Jan 17, 2015 at 9:00 PM, William Hubbs wrote: > >> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation > >> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only > >> is a great option if that remains the default after installation > >> (although it would be fine for just the initial stages). > > > > I'm going to be very blunt. I am sick of the finger being pointed > > only at releng for this. > > To be quite clear, I didn't intend at all to point the finger at > anyone. I mostly reckon it's due to an unfortunate way things hang > together, definitely not any one person or group to blame. I just > really don't understand how it happens. > > Cheers, > > Dirkjan Hey Dirkjan, I didn't mean to imply that *you* personally were pointing the finger at anyone, sorry about the misunderstanding. William signature.asc Description: Digital signature
Re: [gentoo-dev] Things one could be upset about
On 2015-01-19 07:28, Patrick Lauer wrote: > On 01/19/15 17:47, Jeroen Roovers wrote: > > On Sat, 17 Jan 2015 19:35:09 +0800 > > Patrick Lauer wrote: > > > >> * AutoRepoman catches on average maybe 2 user-visible breakages. > >> Mostly removing stable on HPPA ;) > >> Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours) > >> Fix: Remind people that using repoman is not optional > > > > repoman doesn't check reverse dependencies for the package you're > > working on. > > If it were fast enough we could do that. > > pcheck from pkgcore was at one point down to under 3 minutes for a full > tree scan, that's pretty much "fast enough" so that people could run it > on almost every ebuild removal. repoman takes around 30 minutes when > parallelized, on the fastest hardware I currently have access to. Or > about 2 CPU-hours with a single thread ... that's prohibitively slow. I'd be interested to hear if pcheck has regressed significantly at all for you when running it now on a similar set up. Thanks, Tim pgpzx_hu442lz.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Mon, 19 Jan 2015 12:42:45 +0300 Sergey Popov wrote: > 17.01.2015 18:51, Ciaran McCreesh пишет: > > On Sat, 17 Jan 2015 19:49:24 +0400 > > Сергей wrote: > >> Any random user can tell you: -u means UPDATE, -D means DEEP > >> (follow dependencies). > > > > And what do those actually mean? > > Do you need citation from 'man portage'? :-) Well no, because that doesn't answer the question... > -D usually adds to deptree more deps, than usual 'world'(if we are > still talking about 'emerge -uDN world') like deps of deps, etc, > until full tree will be built. > > Maybe i am not totally correct You're not totally correct. > And 'man portage' is telling me pretty much the same things Right, which brings us back to the original point: very few people actually understand what those options *really* do, so claiming that Portage has an intuitive or obvious commandline is rather questionable. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
Patrick Lauer: > Here's a random unsorted list of things that it would make sense to be upset > about. Some issues that people have successfully ignored for a few years ... > > In no way exhaustive list, feel free to remember a dozen things I forgot ;) > (If you suggest other things please try to offer constructive criticism, > i.e. possible strategies to fix issues ... whining by itself is not very > useful) > Thanks, that's an excellent thread. I think you forgot an important point: * lack of practical QA: no review workflow and no appropriate tools for reviewing I could start a long text block about why reviewing is mandatory for QA, but let's just think about it this way: What do you think would happen if the linux kernel switched to CVS and gave the most active 250 collaborators direct push access to the main Linus repository? I hope greg k-h does not read this. He'd probably get a heart attack. Also: people seem to think we don't have enough manpower for a review workflow. No, it's really the other way around. If you make collaboration difficult, then you need a lot more manpower.
Re: [gentoo-dev] Things one could be upset about
On 01/19/15 17:47, Jeroen Roovers wrote: > On Sat, 17 Jan 2015 19:35:09 +0800 > Patrick Lauer wrote: > >> * AutoRepoman catches on average maybe 2 user-visible breakages. >> Mostly removing stable on HPPA ;) >> Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours) >> Fix: Remind people that using repoman is not optional > > repoman doesn't check reverse dependencies for the package you're > working on. If it were fast enough we could do that. pcheck from pkgcore was at one point down to under 3 minutes for a full tree scan, that's pretty much "fast enough" so that people could run it on almost every ebuild removal. repoman takes around 30 minutes when parallelized, on the fastest hardware I currently have access to. Or about 2 CPU-hours with a single thread ... that's prohibitively slow.
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 19:35:09 +0800 Patrick Lauer wrote: > * AutoRepoman catches on average maybe 2 user-visible breakages. > Mostly removing stable on HPPA ;) > Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours) > Fix: Remind people that using repoman is not optional repoman doesn't check reverse dependencies for the package you're working on. eshowkw(1) displays keywording in a pretty nice graph. Avoiding removing the (last) stable ebuild probably involves having that automatically invoked on entering (or working in, at some point) a package directory and actually reading the output before you decide to remove any ebuild. jer
Re: [gentoo-dev] Things one could be upset about
17.01.2015 18:51, Ciaran McCreesh пишет: > On Sat, 17 Jan 2015 19:49:24 +0400 > Сергей wrote: >> Any random user can tell you: -u means UPDATE, -D means DEEP (follow >> dependencies). > > And what do those actually mean? > Do you need citation from 'man portage'? :-) -D usually adds to deptree more deps, than usual 'world'(if we are still talking about 'emerge -uDN world') like deps of deps, etc, until full tree will be built. Maybe i am not totally correct, cause i am not portage developer, but that's my point of view. And 'man portage' is telling me pretty much the same things -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop Effects project lead Gentoo Quality Assurance project lead Gentoo Proxy maintainers project lead signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 13:44:21 +0100 Dirkjan Ochtman wrote: > On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer > wrote: > > > * Some stable bugs are left alone for months > >See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632 > >Fix: Have more people work on stable bugs > >Fix: Motivate people to file more stable bugs (continuous > > updates) > > This is a thorny problem as well. I worry that we lose momentum here > due to size and perfectionism (e.g. we can only stable new gcc once we > fix all the blockers, and we don't have enough active maintainers on > some of those blockers). I think we should maybe stabilize more > optimistically [...] > I also wonder if we could sort of crowd-source archtesting, maybe by > having people contribute their package.keywords through gentoo-stats > or some such to see how well an unstable package is being tested on > stable systems already. This could help in a sense that developers would have more confidence when stabilizing packages. But even without such statistics there is for example the number of open bugs that gives a clue about the quality of a package. It should be used to drive stabilizations in the spirit of good old rule "1 month without bugs => stabilize". At least for less critical packages, mainly end-user applications. I recall that some time ago there were some activities regarding this rule but I am not sure if it is in place. So I would add one more fix for this issue: Fix: Apply "1 month without bugs => stabilize" rule more often. Also I think that end-user applications could be stabilized little more aggresivelly while libraries could keep current conservative approach. For example, having installed ~1700 packages of which ~500 are in world file, recent update world after two months gave me: 292 packages (183 upgrades, 60 new, 4 in new slots, 45 reinstalls). However counting number of end-user applications that were updated I end up with 9 of them of which only 6 was a somewhat major update that could bring new features. (I do not consider system utilites -- like for example lsof -- as end-user applications even that number of them are in my world file.) So if you look to it from this perspective such update does not look very efficient since out of ~300 builded packages only 6 brings potential to increase productivity/bring new features. Such experiences brings me to conclusion that end-user applications may be stabilized more often. Regards, Robert -- Róbert Čerňanský E-mail: ope...@tightmail.com Jabber: h...@jabber.sk
Re: [gentoo-dev] Things one could be upset about
On Sat, Jan 17, 2015 at 9:00 PM, William Hubbs wrote: >> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation >> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only >> is a great option if that remains the default after installation >> (although it would be fine for just the initial stages). > > I'm going to be very blunt. I am sick of the finger being pointed > only at releng for this. To be quite clear, I didn't intend at all to point the finger at anyone. I mostly reckon it's due to an unfortunate way things hang together, definitely not any one person or group to blame. I just really don't understand how it happens. Cheers, Dirkjan
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 17:43:17 -0800 Zac Medico wrote: > On 01/17/2015 04:46 PM, Patrick Lauer wrote: > > On Saturday 17 January 2015 13:12:56 Zac Medico wrote: > >> On 01/17/2015 03:35 AM, Patrick Lauer wrote: > >>> * Portage is too slow > >>> > >>> On 'small' hardware emerge -upNDv @world can take enough time > >>> to make updates prohibitive - on an 800Mhz machine it took me > >>> about 3 days to figure out a solution to some silly blockers due to > >>> the > >>> very slow change test cycle > >>> Fix: Do some profiling and un-refactoring to remove useless code > >>> Fix: Set up a reference system (CI) to catch future regressions > >> > >> I'm certainly in favor of improving portage performance. However, for > >> "small" hardware (which includes many ARM and MIPS systems), we really > >> need to focus on binary package support. Note that dependency resolution > >> for installing binary packages tends to be much simpler than for > >> building ebuilds from source, and this translates into much shorter > >> dependency resolution time. > >> > > That's an orthogonal problem. And that's coming from someone who > > extensively > > uses binpkgs already ... > > Sure, but building from source on "small" hardware is not necessarily > the best practice. If our binary package support was better, then people > might be more likely to use "big" hardware to build packages for > distribution to "small" hardware. Maybe I'm going offtopic, but emerge performance is not limiting factor here, at least from my experience; of course performance improvements in emerge are always welcomed. The largest problem with "small" hardware is cross-compilation from "large" hardware. Less powerful systems often have completely different architecture, e.g. I'd like to install Gentoo on Paspberry Pi B (why? just for fun and to compare its performance to Raspbian), but I don't have any other ARM hosts, so I have to build packages on ~amd64. Setting distcc is not enough here, because it can't handle configure, autotools, linking, non-C/C+ +/ObjC code and so no. And cross-compilation was always a black magick. I tried to setup ~amd64 host to build arbitrary arm packages, but I failed. The main problem is that too many packages bootstrap during compilation phase, e.g. compile some binary/library which is used later during compilation. Of course, code compiled for arm will not run on amd64, so package fails to build. Probably I should try to use QEMU as described in embedded handbook and offload compilation to distcc, even distcc on the same host in order to move most compilation process out of QEMU VM. Best regards, Andrew Savchenko pgpR_I7ZF6aEr.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On 01/17/2015 04:46 PM, Patrick Lauer wrote: > On Saturday 17 January 2015 13:12:56 Zac Medico wrote: >> On 01/17/2015 03:35 AM, Patrick Lauer wrote: >>> * Portage is too slow >>> >>> On 'small' hardware emerge -upNDv @world can take enough time >>> to make updates prohibitive - on an 800Mhz machine it took me >>> about 3 days to figure out a solution to some silly blockers due to >>> the >>> very slow change test cycle >>> Fix: Do some profiling and un-refactoring to remove useless code >>> Fix: Set up a reference system (CI) to catch future regressions >> >> I'm certainly in favor of improving portage performance. However, for >> "small" hardware (which includes many ARM and MIPS systems), we really >> need to focus on binary package support. Note that dependency resolution >> for installing binary packages tends to be much simpler than for >> building ebuilds from source, and this translates into much shorter >> dependency resolution time. >> > That's an orthogonal problem. And that's coming from someone who extensively > uses binpkgs already ... Sure, but building from source on "small" hardware is not necessarily the best practice. If our binary package support was better, then people might be more likely to use "big" hardware to build packages for distribution to "small" hardware. > The problem with (dependency resolution) performance is that in some > scenarios, even on fast machines, it takes "too long" - especially if there's > some silly useflag mismatch and two packages that have ~arch keywords, and > now > you need 12 attempts to figure out a solution that works for you ... > At 30 seconds for each resolution cycle that's 6 minutes waiting for portage. > > If it were faster it'd almost be interactive ... Yeah, that would be great. On a related note, I think that the default emerge --backtrack=10 setting is too high, causing emerge to waste lots of cpu time before it ultimately fails to find a valid solution. I've filed a bug to make it default to --backtrack=3 [1]. > Also - just for comparison: > On my currently fastest box "emerge -ep @world" takes about 3 seconds since > there's almost no packages installed. > On my currently slowest box same takes about 15 minutes, because (1) lots of > packages installed and (2) slow CPU and (3) brutally slow IO > > Binpkgs only help so much - if any dependencies change I need to sync the > configuration between client and server, and to see if it resolves I need to > do > a dryrun on the client (or be very certain that the configuration really > matches) - or risk that it's going to fail because of mismatches. For some, maybe a nice way to sync configuration would be to create a customized profile. Then, emerge --sync would pull in your configuration updates from the server. With "profile-formats = portage-2" in metadata/layout.conf of your profiles repository, you can put things like "gentoo:default/linux/amd64/13.0/desktop" in the parent file of your profile, so it inherits from a profile in the "gentoo" repository. > I haven't quite figured out why portage needs such humongous amounts of > memory > (>300MB ?!) Yes, I would like to work on reducing the memory consumption. > and why it needs to spend a minute to figure out how to install a > simple almost-no-dependencies tool like htop or iotop. Note that the emerge --complete-graph-if-new-ver and --complete-graph-if-new-use options are enabled by default. These options will force emerge to create a complete dependency graph, in order to ensure that all reverse-dependencies are respected. > There's some obvious > badness, and just saying "well let's use binpkgs" won't fix it (and, well, on > the binpkg server if you have 3k packages installed you get the same > performance issues, so ignoring the issue is kinda not a good idea) Of course not. > There's been some good progress, I remember you reducing memory usage already > (>500MB -> 350MB or so for merging kernel sources) and there's some speedups > from the last round of performance work. Still I think we can do a lot better > :) Sure we can. > Thanks for your efforts, > > Patrick [1] https://bugs.gentoo.org/show_bug.cgi?id=536926 -- Thanks, Zac
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 13:12:56 Zac Medico wrote: > On 01/17/2015 03:35 AM, Patrick Lauer wrote: > > * Portage is too slow > > > > On 'small' hardware emerge -upNDv @world can take enough time > > to make updates prohibitive - on an 800Mhz machine it took me > > about 3 days to figure out a solution to some silly blockers due to > > the > > very slow change test cycle > > Fix: Do some profiling and un-refactoring to remove useless code > > Fix: Set up a reference system (CI) to catch future regressions > > I'm certainly in favor of improving portage performance. However, for > "small" hardware (which includes many ARM and MIPS systems), we really > need to focus on binary package support. Note that dependency resolution > for installing binary packages tends to be much simpler than for > building ebuilds from source, and this translates into much shorter > dependency resolution time. > That's an orthogonal problem. And that's coming from someone who extensively uses binpkgs already ... The problem with (dependency resolution) performance is that in some scenarios, even on fast machines, it takes "too long" - especially if there's some silly useflag mismatch and two packages that have ~arch keywords, and now you need 12 attempts to figure out a solution that works for you ... At 30 seconds for each resolution cycle that's 6 minutes waiting for portage. If it were faster it'd almost be interactive ... Also - just for comparison: On my currently fastest box "emerge -ep @world" takes about 3 seconds since there's almost no packages installed. On my currently slowest box same takes about 15 minutes, because (1) lots of packages installed and (2) slow CPU and (3) brutally slow IO Binpkgs only help so much - if any dependencies change I need to sync the configuration between client and server, and to see if it resolves I need to do a dryrun on the client (or be very certain that the configuration really matches) - or risk that it's going to fail because of mismatches. I haven't quite figured out why portage needs such humongous amounts of memory (>300MB ?!) and why it needs to spend a minute to figure out how to install a simple almost-no-dependencies tool like htop or iotop. There's some obvious badness, and just saying "well let's use binpkgs" won't fix it (and, well, on the binpkg server if you have 3k packages installed you get the same performance issues, so ignoring the issue is kinda not a good idea) There's been some good progress, I remember you reducing memory usage already (>500MB -> 350MB or so for merging kernel sources) and there's some speedups from the last round of performance work. Still I think we can do a lot better :) Thanks for your efforts, Patrick
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 14:00:34 William Hubbs wrote: > On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote: > > On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote: > > > * Stage3 archives are too fat > > > > > > See https://bugs.gentoo.org/show_bug.cgi?id=531632 > > > We're now shipping three python versions and glib for extra fun! > > > Fix: Motivate releng to stop being silly > > > > Why the heck do we ship both 3.3 and 3.4? I forget the exact situation > > with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only > > is a great option if that remains the default after installation > > (although it would be fine for just the initial stages). > > I'm going to be very blunt. I am sick of the finger being pointed > only at releng for this. > > The issue is package dependencies. > > If even one package in the tree has a dumb dependency on python, e.g. > dev-lang/python, it will pull in all stable versionf of python. Only if you absolutely insist that releng can never deviate from tree. Which is a silly idea, see bindist useflag, which is locally enabled for stage building and then removed. Oh wait, it's not removed because we can't deviate while deviating. So users regularly find ssl 'broken' ... It took me about 2h of fiddling around to find a few spots where stage3 has useless bloat: - pkgconfig pulling in 30MB of glib - python installing tests (that's 3x 25MB now ...) - having more than one python installed (which is not really absolutely necessary, and could easily be reduced to one) Out of 700MB on-disk I could prune about 150MB - about 20%, without affecting functionality It's not about pointing a finger, it's about fixing issues when they are easy to fix and not hiding behind a fake complexity argument. (I would fix things, if I had access and/or certainty that patches provided would be integrated ...) Have fun, Patrick
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 15:03:05 Ciaran McCreesh wrote: > On Sat, 17 Jan 2015 22:58:33 +0800 > > Patrick Lauer wrote: > > On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote: > > > On Sat, 17 Jan 2015 21:03:30 +0800 > > > > > > Patrick Lauer wrote: > > > > Last time I tested paludis it was slower > > > > > > You've yet to do a like-for-like comparison... > > > > Hello hostile upstream. > > > > It was as "like for like" as possible > > Well that's the point... A rare case of violent agreement? Excellent. And/or you're terminally bored - either way you're not adding anything relevant to this conversation.
Re: [gentoo-dev] Things one could be upset about
On Sat, Jan 17, 2015 at 12:00 PM, William Hubbs wrote: > On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote: >> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote: >> > * Stage3 archives are too fat >> > See https://bugs.gentoo.org/show_bug.cgi?id=531632 >> > We're now shipping three python versions and glib for extra fun! >> > Fix: Motivate releng to stop being silly >> >> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation >> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only >> is a great option if that remains the default after installation >> (although it would be fine for just the initial stages). > > I'm going to be very blunt. I am sick of the finger being pointed > only at releng for this. Well, there are trivial things that could be done on the stage building end... and complete intransigence toward totally reasonable ideas like not building with default options for a particularly specialized target (stages, livecds, etc.). Blame whomever you like, but ultimately the stages are a product of the release engineering team who has the ability to fix the issue themselves but choose not to.
Re: [gentoo-dev] Things one could be upset about
On 01/17/2015 03:35 AM, Patrick Lauer wrote: > * Portage is too slow > On 'small' hardware emerge -upNDv @world can take enough time > to make updates prohibitive - on an 800Mhz machine it took me > about 3 days to figure out a solution to some silly blockers due to the > very slow change test cycle > Fix: Do some profiling and un-refactoring to remove useless code > Fix: Set up a reference system (CI) to catch future regressions I'm certainly in favor of improving portage performance. However, for "small" hardware (which includes many ARM and MIPS systems), we really need to focus on binary package support. Note that dependency resolution for installing binary packages tends to be much simpler than for building ebuilds from source, and this translates into much shorter dependency resolution time. As I have expressed in other emails [1], I am currently working on making Gentoo's binary package support more competitive with binary distros. I have created a sys-apps/portage- ebuild [2] for people who would like to test features not released in mainline portage yet. [1] http://thread.gmane.org/gmane.linux.gentoo.project/4205/focus=4217 [2] https://github.com/zmedico/portage-binpkg-support-overlay -- Thanks, Zac
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 19:35:09 +0800 Patrick Lauer wrote: > * AutoRepoman catches issues, but no one but me seems to care > Fix: Remind people of > http://packages.gentooexperimental.org/repoman-current-issues.txt I've tweaked two random things in this list :) Thank you! -- Sergei signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote: > On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote: > > * Stage3 archives are too fat > > See https://bugs.gentoo.org/show_bug.cgi?id=531632 > > We're now shipping three python versions and glib for extra fun! > > Fix: Motivate releng to stop being silly > > Why the heck do we ship both 3.3 and 3.4? I forget the exact situation > with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only > is a great option if that remains the default after installation > (although it would be fine for just the initial stages). I'm going to be very blunt. I am sick of the finger being pointed only at releng for this. The issue is package dependencies. If even one package in the tree has a dumb dependency on python, e.g. dev-lang/python, it will pull in all stable versionf of python. If all the dependencies are correct, on the other hand, it is a matter of setting python_targets to include the versions of python we want in the stages and making sure that everything we include in the stages works with those versions of python. William signature.asc Description: Digital signature
Re: [gentoo-dev] Things one could be upset about
Patrick Lauer wrote: > they can all be fixed. > > Let's not tolerate mediocrity. All you can do is to try to set an example, but you'll likely find that most of the time, nobody is willing to live with the tradeoffs for excellence - the obvious one being perceived slower development. Countless others are perfectly happy with mediocrity, and wouldn't care even if they were to understand just how mediocre. //Peter
Re: [gentoo-dev] Things one could be upset about
Dnia 2015-01-17, o godz. 19:35:09 Patrick Lauer napisał(a): > Here's a random unsorted list of things that it would make sense to be upset > about. Some issues that people have successfully ignored for a few years ... > > In no way exhaustive list, feel free to remember a dozen things I forgot ;) > (If you suggest other things please try to offer constructive criticism, > i.e. possible strategies to fix issues ... whining by itself is not very > useful) * people writing to the mailing lists and starting pointless discussions which never bring anything good and only waste people's time. -- Best regards, Michał Górny pgp_ADBD4x_m1.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 19:49:24 +0400 Сергей wrote: > Any random user can tell you: -u means UPDATE, -D means DEEP (follow > dependencies). And what do those actually mean? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 18:33:45 +0300 Andrew Savchenko wrote: > Oh, this was discussed so many times already... There is NO single > correct solution to such problems. And some mathematically correct > solutions are impractical (e.g. half of the tree rebuild), so other > ones which are good enough are preferred. As long as imperfect > solution works fine, I'm ok with it. And this is how errors accumulate until your system is a broken mess... > - slower then portage Not to do the same thing it isn't. > (I don't care here if it is more correct or not, I just want to do my > daily job); Well, either you spend a tiny bit more time every now and again avoiding problems, or you spend a huge amount of time when something breaks at the least convenient possible moment. > - not fully compatible with portage: it triggers a lot of problems > portage doesn't. While this may be good for QA and tree > improvement, I don't want to hang my workflow due to these issues; Most of the problems you'll encounter are a one-off thing when switching, particularly if you've allowed your system to be full of broken dependencies etc by long-term use of Portage (see above). Once you've cleaned up your system and fixed all the breakages you've introduced by general sloppiness over the years, the problems you'll see are genuine ones that need to be dealt with properly. > - importare instead of local overlay is a complete nightmare: > usually I don't want to install package exactly as make install > does: often it lacks some required files (e.g. init scripts) or > installs something unneeded on Gentoo system. > Besides I use local overlay to test packages before committing to > public overlays or Gentoo main tree. Lack of local overlay support > is sufficient to send paludis straight to waste bin on my systems. Uh, Paludis supports overlays, and always has. > - completely insane command line options, arguments required to do > what I want to do are quite cumbersome, portage is much saner here. "emerge -uUDpvN --with-bdeps=y" vs "cave resolve --complete"? Here's a quiz for you: exactly what do -u and -D even do with Portage? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
> * AutoRepoman catches on average maybe 2 user-visible breakages. > Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours) > Fix: Remind people that using repoman is not optional Fix: Make more repoman warnings fatal. Please. Fix: Make the QA team deliver a friendly but stern warning to people who don't use repoman and thus commit crap. If repeated or blatantly wrong, revert [1]. Dream: If the git migration ever happens, make later git refuse pushes that break the tree. (if only by pushing to a staging area first, which is then automatically forwarded to the main tree if ok.) > * AutoRepoman still runs on my hardware > Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064 > * mail archives have been broken since 2012 > Fix: get infra to either fix it, or provide enough information that it > can be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27 > * git.overlays.gentoo.org only partially functional (web interface / cgit > down) > * euscan doesn't run on infra hardware > Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886 > * libreoffice-bin isn't built on infra hardware > Fix: Remind infra to set up a build environment for that Fix: Maybe accept more volunteers into infra? It would be interesting to hear if/how recruiting of new infra members progresses, in relation to workload. [I don't buy the argument that the mail archives are unneeded, ever since I've tried assembling a council agenda and half the relevant e-mails were missing at gmane. AutoRepoman and irker are extremely useful and a direct help for all developers.] * Our main website is still stuck in the 90ies. Fix: migrate all remaining projects to the wiki, then set up something new as main page. Willing to help out with the first step (generating the wiki pages) if people are willing to accept help. * We need a bit more Public Relations. Feeling a bit annoyed right now that we still don't have a FOSDEM booth (maybe we should just occupy the ChromeOS one). Fix: Next year, start getting annoyed in time and annoy some fellow Europeans. :] [1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-libs/libusbhp/ChangeLog?hideattic=0&view=log -- Andreas K. Huettel Gentoo Linux developer (council, kde) dilfri...@gentoo.org http://www.akhuettel.de/
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 14:45:51 + Ciaran McCreesh wrote: > On Sat, 17 Jan 2015 17:39:29 +0300 > Andrew Savchenko wrote: > > There is some progress here. In portage-2.2.15 profile based > > optimizations are included (see bugs 529660, 530010). On my > > hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they > > speed up dependency resolution by ~ factor 2. > > The problem isn't the constants, though. The problem is the resolution > algorithm. There's not much point tweaking performance until the > resolver is fixed to produce a correct answer... Oh, this was discussed so many times already... There is NO single correct solution to such problems. And some mathematically correct solutions are impractical (e.g. half of the tree rebuild), so other ones which are good enough are preferred. As long as imperfect solution works fine, I'm ok with it. As for paludis, I tried it. Everything mentioned below is my _personal_ experience and not supposed to pretend to be objective. So, from my experience paludis turned out to be: - slower then portage (I don't care here if it is more correct or not, I just want to do my daily job); - not fully compatible with portage: it triggers a lot of problems portage doesn't. While this may be good for QA and tree improvement, I don't want to hang my workflow due to these issues; - importare instead of local overlay is a complete nightmare: usually I don't want to install package exactly as make install does: often it lacks some required files (e.g. init scripts) or installs something unneeded on Gentoo system. Besides I use local overlay to test packages before committing to public overlays or Gentoo main tree. Lack of local overlay support is sufficient to send paludis straight to waste bin on my systems. - completely insane command line options, arguments required to do what I want to do are quite cumbersome, portage is much saner here. Everything above is just my personal experience, so your mileage may vary. But after all that issues I don't even want to try paludis in the foreseeable future. Best regards, Andrew Savchenko pgpCxUFZ8zSgo.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 22:59:08 +0800 Patrick Lauer wrote: > > The problem isn't the constants, though. The problem is the > > resolution algorithm. There's not much point tweaking performance > > until the resolver is fixed to produce a correct answer... > > Patches welcome :) If I send you a patch that updates all the documentation to use Paludis rather than Portage, will you welcome it? -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 22:58:33 +0800 Patrick Lauer wrote: > On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote: > > On Sat, 17 Jan 2015 21:03:30 +0800 > > Patrick Lauer wrote: > > > Last time I tested paludis it was slower > > > > You've yet to do a like-for-like comparison... > > Hello hostile upstream. > > It was as "like for like" as possible Well that's the point... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 14:45:51 Ciaran McCreesh wrote: > On Sat, 17 Jan 2015 17:39:29 +0300 > > Andrew Savchenko wrote: > > There is some progress here. In portage-2.2.15 profile based > > optimizations are included (see bugs 529660, 530010). On my > > hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they > > speed up dependency resolution by ~ factor 2. > > The problem isn't the constants, though. The problem is the resolution > algorithm. There's not much point tweaking performance until the > resolver is fixed to produce a correct answer... Patches welcome :)
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote: > On Sat, 17 Jan 2015 21:03:30 +0800 > > Patrick Lauer wrote: > > Last time I tested paludis it was slower > > You've yet to do a like-for-like comparison... Hello hostile upstream. It was as "like for like" as possible, even when there had to be workaround to make paludis build (and wait multiple hours as it is kinda horribly C++) as it tried to OOM very nicely. You can continue whining about it as much as you want, my results were duplicated by multiple people and you never offered a tweaked comparison that is to your liking. So stop whining (which is amusingly the trigger for my initial email - people whining about irrelevant things instead of doing something useful).
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 17:39:29 +0300 Andrew Savchenko wrote: > There is some progress here. In portage-2.2.15 profile based > optimizations are included (see bugs 529660, 530010). On my > hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they > speed up dependency resolution by ~ factor 2. The problem isn't the constants, though. The problem is the resolution algorithm. There's not much point tweaking performance until the resolver is fixed to produce a correct answer... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 19:35:09 +0800 Patrick Lauer wrote: > * Portage is too slow > On 'small' hardware emerge -upNDv @world can take enough time > to make updates prohibitive - on an 800Mhz machine it took me > about 3 days to figure out a solution to some silly blockers due to the > very slow change test cycle > Fix: Do some profiling and un-refactoring to remove useless code > Fix: Set up a reference system (CI) to catch future regressions There is some progress here. In portage-2.2.15 profile based optimizations are included (see bugs 529660, 530010). On my hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they speed up dependency resolution by ~ factor 2. Of course it will be great to see further optimizations, though as far as I remember this will require more complicated changes. Best regards, Andrew Savchenko pgpOKujBNeLwv.pgp Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
On Sat, 17 Jan 2015 21:03:30 +0800 Patrick Lauer wrote: > Last time I tested paludis it was slower You've yet to do a like-for-like comparison... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Things one could be upset about
El sáb, 17-01-2015 a las 13:44 +0100, Dirkjan Ochtman escribió: [...] > Also, I hate something like > "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']". > What the hell kind of warning is that? I guess maybe these are the > results of USE_EXPAND trickery and what not, but it would sure be nice > to have something more readable. Yeah, sometimes the output are really fat, not sure if some heuristic could be done to, for example, collate the exact same errors that are coming from every single subprofile.
Re: [gentoo-dev] Things one could be upset about
El sáb, 17-01-2015 a las 19:35 +0800, Patrick Lauer escribió:
[...]
> * stable genkernel still doesn't enable all useful kernel features
>e.g. accounting statistics are absent, so iotop doesn't work ootb
>See for example #442936 (from 2012 ?!)
>Fix: Stable newer versions
>
I see there isn't even any stable bug report for that, is anyone major
blocking newer versions to be stabilized?
> * AutoRepoman catches issues, but no one but me seems to care
> Fix: Remind people of
> http://packages.gentooexperimental.org/repoman-current-issues.txt
> Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}
>
Maybe some warnings could be raised to fatal to prevent to commit them
(that shouldn't be a problem as that are usually easy to fix problems).
For example, the dodoc "LICENSE" stuff and WANT_AUTOCONF=latest
> * mail archives have been broken since 2012
> Fix: get infra to either fix it, or provide enough information that it
> can
> be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
>
Is there any advantage of setting them up again instead of relying on
other existing external resources? :/
Re: [gentoo-dev] Things one could be upset about
On Sat, Jan 17, 2015 at 2:03 PM, Patrick Lauer wrote: > Most issues are transient, often fixed with either a keywording bug, or > careful > masking of useflags / pruning of old versions. Per-maintainer doesn't really > make sense as most issues are indirect - things like "removing package.mask > entry for new version causes dependency breakage on ia64 to become visible", > or "dependency of dependency changed". In both cases it's often hard enough to > figure out what caused the issue. I think louder in the sense of a weekly -dev email could still makes sense. #-bugs and #-qa are way too noisy for me. Cheers, Dirkjan
Re: [gentoo-dev] Things one could be upset about
On Saturday 17 January 2015 13:44:21 Dirkjan Ochtman wrote:
> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote:
> > * AutoRepoman catches issues, but no one but me seems to care
> >
> > Fix: Remind people of
> > http://packages.gentooexperimental.org/repoman-current-issues.txt
> > Fix: Make it yell louder? It currently reports on IRC to
> > #gentoo-{bugs,qa}
> I'll happily fix my repoman issues when I notice them. I try to always
> run repoman full on a package, but like you say, a tree-wide scan
> isn't really viable on a per-commit basis. Can we have AutoRepoman
> report open issues to gentoo-dev on a weekly basis? That seems like a
> fine idea to me. Per-maintainer and per-team/project/herd RSS/Atom
> feeds would also be pretty nice to have.
Most issues are transient, often fixed with either a keywording bug, or careful
masking of useflags / pruning of old versions. Per-maintainer doesn't really
make sense as most issues are indirect - things like "removing package.mask
entry for new version causes dependency breakage on ia64 to become visible",
or "dependency of dependency changed". In both cases it's often hard enough to
figure out what caused the issue.
> Also, I hate something like
> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_pyt
> hon2_7(-)]']". What the hell kind of warning is that? I guess maybe these
> are the results of USE_EXPAND trickery and what not, but it would sure be
> nice to have something more readable.
Indeed, portage output is quite complex and often hard to read. I'm not sure
what's the best way to improve it - I've filed a few small bugs for output
prettyfication, but I don't even know what I'd want to be displayed for these
useflag / blocker issues.
>
> > * Portage is too slow
> >
> > On 'small' hardware emerge -upNDv @world can take enough time
> > to make updates prohibitive - on an 800Mhz machine it took me
> > about 3 days to figure out a solution to some silly blockers due to
> > the
> > very slow change test cycle
> > Fix: Do some profiling and un-refactoring to remove useless code
> > Fix: Set up a reference system (CI) to catch future regressions
>
> Why not use paludis, or another package manager?
Last time I tested paludis it was slower (and building it OOMed on that
machine with default settings, so that's quite amusing)
Also it suffers from a hostile upstream, makes bugreports very tricky to handle
etc.etc.
Pkgcore is in hibernation, it needs a bit more work to become really useful
again. Possibly some ideas from pkgcore can be migrated to portage again, but
that'd need someone with lots of free time.
Thanks for your feedback,
Patrick
Re: [gentoo-dev] Things one could be upset about
On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer wrote:
> In no way exhaustive list, feel free to remember a dozen things I forgot ;)
> (If you suggest other things please try to offer constructive criticism,
> i.e. possible strategies to fix issues ... whining by itself is not very
> useful)
I think this is a good list, thanks for coming up with it. I very much
agree that whining by itself is not very useful, and we should find
ways to move things forward (even if sometimes it's not the most
straightforward way?).
> * AutoRepoman catches on average maybe 2 user-visible breakages.
> Mostly removing stable on HPPA ;)
> Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
> Fix: Remind people that using repoman is not optional
>
> * AutoRepoman catches issues, but no one but me seems to care
> Fix: Remind people of
> http://packages.gentooexperimental.org/repoman-current-issues.txt
> Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}
I'll happily fix my repoman issues when I notice them. I try to always
run repoman full on a package, but like you say, a tree-wide scan
isn't really viable on a per-commit basis. Can we have AutoRepoman
report open issues to gentoo-dev on a weekly basis? That seems like a
fine idea to me. Per-maintainer and per-team/project/herd RSS/Atom
feeds would also be pretty nice to have.
Also, I hate something like
"['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']".
What the hell kind of warning is that? I guess maybe these are the
results of USE_EXPAND trickery and what not, but it would sure be nice
to have something more readable.
> * Portage is too slow
> On 'small' hardware emerge -upNDv @world can take enough time
> to make updates prohibitive - on an 800Mhz machine it took me
> about 3 days to figure out a solution to some silly blockers due to the
> very slow change test cycle
> Fix: Do some profiling and un-refactoring to remove useless code
> Fix: Set up a reference system (CI) to catch future regressions
Why not use paludis, or another package manager?
> * Stage3 archives are too fat
> See https://bugs.gentoo.org/show_bug.cgi?id=531632
> We're now shipping three python versions and glib for extra fun!
> Fix: Motivate releng to stop being silly
Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
is a great option if that remains the default after installation
(although it would be fine for just the initial stages).
> * AutoRepoman still runs on my hardware
> Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064
>
> * euscan doesn't run on infra hardware
> Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886
>
> * mail archives have been broken since 2012
> Fix: get infra to either fix it, or provide enough information that it can
> be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
>
> * libreoffice-bin isn't built on infra hardware
> Fix: Remind infra to set up a build environment for that
These ones, the euscan one in particular, also have been stuff I cared
about, but after trying a bunch of times to start helping infra out,
apparently they don't have any ways to absorb new contributors. This
does seem like a big problem given the amount of technical debt on the
infra-side you're laying out here.
I understand that it's a big job and that there are security aspects
to it, but if there are no ways to empower new people to help them
out, that is worrying to me.
> * Some stable bugs are left alone for months
>See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632
>Fix: Have more people work on stable bugs
>Fix: Motivate people to file more stable bugs (continuous updates)
This is a thorny problem as well. I worry that we lose momentum here
due to size and perfectionism (e.g. we can only stable new gcc once we
fix all the blockers, and we don't have enough active maintainers on
some of those blockers). I think we should maybe stabilize more
optimistically at least on common architectures, e.g. by having a
lighter-weight upfront testing process and relying more on maintainers
keeping up to speed on subsequent bugs.
I also wonder if we could sort of crowd-source archtesting, maybe by
having people contribute their package.keywords through gentoo-stats
or some such to see how well an unstable package is being tested on
stable systems already.
Or if we should have a different process for e.g. Python/Ruby/Perl/PHP
packages compared to C/C++ packages, since the former are much less
sensitive to architecture differences.
Also, one thing you didn't mention is the git migration; I think our
usage of CVS definitely counts as a big gob of technical debt. What's
that currently blocked on, anyway?
Cheers,
Dirkjan
[gentoo-dev] Things one could be upset about
Here's a random unsorted list of things that it would make sense to be upset
about. Some issues that people have successfully ignored for a few years ...
In no way exhaustive list, feel free to remember a dozen things I forgot ;)
(If you suggest other things please try to offer constructive criticism,
i.e. possible strategies to fix issues ... whining by itself is not very
useful)
* stable genkernel still doesn't enable all useful kernel features
e.g. accounting statistics are absent, so iotop doesn't work ootb
See for example #442936 (from 2012 ?!)
Fix: Stable newer versions
* stage3 still enables bindist in make.conf
See https://bugs.gentoo.org/show_bug.cgi?id=473332
Causes random breakage of tools like wget/curl, etc.
Fix: poke releng until they stop being silly
* AutoRepoman catches on average maybe 2 user-visible breakages.
Mostly removing stable on HPPA ;)
Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
Fix: Remind people that using repoman is not optional
* Portage is too slow
On 'small' hardware emerge -upNDv @world can take enough time
to make updates prohibitive - on an 800Mhz machine it took me
about 3 days to figure out a solution to some silly blockers due to the
very slow change test cycle
Fix: Do some profiling and un-refactoring to remove useless code
Fix: Set up a reference system (CI) to catch future regressions
* Stage3 archives are too fat
See https://bugs.gentoo.org/show_bug.cgi?id=531632
We're now shipping three python versions and glib for extra fun!
Fix: Motivate releng to stop being silly
* AutoRepoman catches issues, but no one but me seems to care
Fix: Remind people of
http://packages.gentooexperimental.org/repoman-current-issues.txt
Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}
* AutoRepoman still runs on my hardware
Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064
* mail archives have been broken since 2012
Fix: get infra to either fix it, or provide enough information that it can
be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
* git.overlays.gentoo.org only partially functional (web interface / cgit
down)
* euscan doesn't run on infra hardware
Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886
* libreoffice-bin isn't built on infra hardware
Fix: Remind infra to set up a build environment for that
* Some stable bugs are left alone for months
See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632
Fix: Have more people work on stable bugs
Fix: Motivate people to file more stable bugs (continuous updates)
* Updates from very old installs are still difficult
Fix: Collect stage3 archives and tree snapshots maybe every 3 months apart
then update from one snapshot to the next. Possibly fix upgrade bugs
retroactively in the snapshots and automate the process
It'd be nice to have such things fixed, but alas, many rely on someone else to
respond. And for some there's no 'easy' solution.
But they can all be fixed.
Let's not tolerate mediocrity.
Take care,
Patrick
