[gentoo-portage-dev] Re: Dynamic USE dependencies
Rich Freeman ri...@gentoo.org wrote: Keep in mind that keeping track of past decisions made by portage does not require user-editable config files in /etc. Yes, but you might not always agree with portage's decisions, and the resolution might be non-unique. Although the user might always influence it with /etc/portage/package.use it might be better to keep this transparent to the user - therefore, I suggested a directory in /etc and not in /var (Moreover, a cache would not make sense, since the previous decision could always be trivially calculated from /var/db/pkg, and the user-selected state of the USE-flag for the corresponding package. You could not check the validity of such a cache, since you cannot check whether USE or /etc/portage/package.use has changed) That said, portage still has to spend time basically re-validating the consistency of the entire system *Validating* consistency is not time-consuming. Looking for possibilities to unset use-flags is a much more time-consuming story. If everything is consistent or can easily be resolved, you hardly need any backtracking...
[gentoo-portage-dev] Re: Dynamic USE dependencies
Rich Freeman ri...@gentoo.org wrote: 1. They can STILL populate /etc/portage/package.use 2. They could manually install a package with one-time specified USE Why do we need another mechanism to control what flags a package gets installed We do not *need* it. By why reject it if you can get it for free by just making the it transparent to the user? The user should be able to decide e.g. when to clean it up and select perhaps e.g. only a certain classes which should be cleaned up (e.g. by removing all/certain cpu_flags_x86 and letting them regenerate).
Re: [gentoo-portage-dev] Re: Dynamic USE dependencies
On Mon, Apr 6, 2015 at 8:02 AM, Martin Vaeth mar...@mvath.de wrote: My suggestion is something in between - less invasive (and, in particular, less time consuming) than your suggestion to recalculate the USE-settings with every emerge, but more automatic than the current state. Keep in mind that keeping track of past decisions made by portage does not require user-editable config files in /etc. It just requires a cache of some kind, much as we do with installed packages/etc. That said, portage still has to spend time basically re-validating the consistency of the entire system because we allow the use of overlays and other situations that don't guarantee that portage will have some kind of consistent pre-calculated depgraph handed to it. If we required all repositories to have some kind of pre-generated cache in them and ensured that this was always up-to-date and better controlled the kinds of dependency changes we made, then maybe there might be an opportunity to offload some of the work to the repository level instead of doing it on every Gentoo system. Still, unless we banned overlays I'm not sure how much even this would buy you, since you'd have many of these that need to be merged somehow. From Zac's email and other discussions in the past it seems like we're basically committed to doing all these calculations all the time anyway, so we shouldn't be too shy about taking advantage of them. -- Rich
[gentoo-portage-dev] Re: Dynamic USE dependencies
Rich Freeman ri...@gentoo.org wrote: On Sun, Apr 5, 2015 at 11:47 AM, Martin Vaeth mar...@mvath.de wrote: One suggestion around this problem would be to use different directories for these two types of use-flags, say package.use and package.use.needed. I still think we need a better long-term solution, but the workaround is simpler than that. I try to keep files in directories for package.keywords and package.use. One starts with a z and is a dumping ground for auto-whatever. So you use package.use/z instead of package.use.needed as in my suggestion. (In fact, I do currently a similar thing, only that I use a file named zzz_autounmask, because this is even later in the alphabet). I think it would be more transparent to the user if really a dedicated directory/file would be used and not some which by accident is the last in the alphabet. Anyway, this is a minor issue. The main problem is how to cleanup this directory/file (no matter how it is called). Currently, the only way to do this, is to remove this file and to hope that portage can recreate it. Unfortunately, quite often portage is not able to do this. The problem is at least NP hard, so probably any backtracking value just is too low. If portage would use the old values as a hint, how conflicts might possibly be resolved, perhaps this algorithm might be improved. However, I think a cleaner solution would be better... ++ My suggestion is something in between - less invasive (and, in particular, less time consuming) than your suggestion to recalculate the USE-settings with every emerge, but more automatic than the current state. With my suggestion the USE-settings have to be recalculated only if a special command is used (or - as currently - if resolving is not possible without changing USE-flags). Probably other approaches are possible, too.
Re: [gentoo-portage-dev] Re: Dynamic USE dependencies
On Sun, Apr 5, 2015 at 11:47 AM, Martin Vaeth mar...@mvath.de wrote: One suggestion around this problem would be to use different directories for these two types of use-flags, say package.use and package.use.needed. I still think we need a better long-term solution, but the workaround is simpler than that. I try to keep files in directories for package.keywords and package.use. One starts with a z and is a dumping ground for auto-whatever. The other files are where I try to remember to store personal preferences. So, in theory at any time I can wipe the z-file and run emerge and watch it wrestle with re-creating it. However, I think a cleaner solution would be better... -- Rich
[gentoo-portage-dev] Re: Dynamic USE dependencies
Zac Medico zmed...@gentoo.org wrote: On 04/02/2015 09:32 AM, Rich Freeman wrote: Out of curiosity, what is keeping us from having USE flag dependencies handled dynamically, in the same way that package dependencies are? It's already able to adjust USE automatically via autounmask support. The problem is that once it is in your package.use (no matter whether added manually or with autounmask), it will stay there forever if the user does not analyze the dependencies manually eventually. And this is different from the dependency handling. The most obvious example (which presumably also Rich had in mind) is the case of abi_x86_32: Assume that you have package foo in your world file which (in version foo-1) depends on package bar. In version foo-2, which appears some years later, it no longer depends on package bar (but e.g. on package bazooka, instead). In this case, emerge --depclean will eventually unmerge bar. This works nicely. However, if foo-1 depends on bar[abi_x86_32(-)] and something else depends on bar[abi_x86_64] then, after upgrading to foo-2, most users wil keep compiling bar with ABI_X86=32 64 forever, because emerge --depclean (or any other tool) does not tell them that now they might remove abi_x86_32 from package bar. In fact, such a tool *cannot* exist, currently: There is no way to distinguish from package.use what is there because the user *wanted* this feature and what is there only because of a (possibly temporary) USE-dependency for another package. In other words: The difference between wanted and needed (reflected usually in the difference between the packages in world and those in /var/db/pkg) is not visible in package.use, in principle. So far, this was never a problem, because the number of flags which were there because they were needed (and not, because they were wanted) was rather low. But with the arrival of ABI_X86=32 the situation has changed; 100-200 needed flags are now normal on many systems. One suggestion around this problem would be to use different directories for these two types of use-flags, say package.use and package.use.needed. Perhaps it would make sense that portage --autounmask writes only to package.use.needed, und that there is a cleanup command which removes all data from package.use.needed except for that strictly necessary to resolve the dependencies of the running system (analogously like emerge --depclean currently removes unneeded packages).
[gentoo-portage-dev] Re: Dynamic USE dependencies
Brian Dolbec posted on Fri, 03 Apr 2015 06:31:27 -0700 as excerpted: On Fri, 3 Apr 2015 11:52:39 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Brian Dolbec posted on Thu, 02 Apr 2015 23:59:06 -0700 as excerpted: enalyze is little known to users. I'd never heard of enalyze before Please consider writing up a tips-n-tricks section feature for enalyze in an upcoming gentoo news Been there, done that... ;) https://blogs.gentoo.org/news/2014/03/ brown paper bag I read that issue too. But I think I was running on about four hours sleep in three days (OK, maybe six hours, eight if you count dozing on the bus and lunch break... obviously not enough!) when I did... Given the evidence, I guess read isn't the appropriate word... I /slept/ /thru/ that issue??? I /sleep-read/ that issue??? Anyway, thanks. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-portage-dev] Re: Dynamic USE dependencies
On Fri, 3 Apr 2015 06:38:48 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Rich Freeman posted on Thu, 02 Apr 2015 22:26:03 -0400 as excerpted: If you stuck -* in your make.conf then this change would have no affect at all, since you've explicitly set the configuration of every use flag. That (and package.use still sticking) eases my mind considerably. The current configuration forces you to use config files to capture settings you care about, and also ones you don't actually care about, and unless you're careful you'll have trouble telling these settings apart later. It is like sticking every installed package in your world file. That comparison is quite persuasive, indeed. =:^) Thanks! I understand your idea much better, now, and (cautiously) agree. =:^) (Tho FWIW, I guess I'm a careful one. I use -* and put non-global USE flags in make.conf too if possible, and review USE flags for all new packages and changes, so everything there is cared about for one reason or another. Package.use thus contains only individual package exceptions, and I comment those with both a date and why they /are/ exceptions to the otherwise global policy, so if the only justification is because package X requires it, that's in the comment. Make.conf's USE= setting does still accumulate unannotated stale flags over time, but I just went thru and verified all USE flags were still used recently, deleting the ones that equery hasuse didn't raise a hit on, and justifying either by-name or by equery uses every remaining entry, so everything there is verified there for a reason now, too.) This is a reminder to you all... since enalyze is little known to users. enalyze application in gentoolkit I made does installed db analysis. The analyze submodule creates detailed reports about use flags (and other stuff) usage. The rebuild module can create new pkg.* files for you to maintain the installed state. Allowing you to easily make make.conf USE= changes or profile changes and have your pkg.* files fixed to match. Of course detailed comments it does not do ;) plus it saves the new file to your user space, you can do what you like with them from there. Editing your detail commented files to match, etc.. stright swap,... -- Brian Dolbec dolsen
[gentoo-portage-dev] Re: Dynamic USE dependencies
Rich Freeman posted on Thu, 02 Apr 2015 22:26:03 -0400 as excerpted: If you stuck -* in your make.conf then this change would have no affect at all, since you've explicitly set the configuration of every use flag. That (and package.use still sticking) eases my mind considerably. The current configuration forces you to use config files to capture settings you care about, and also ones you don't actually care about, and unless you're careful you'll have trouble telling these settings apart later. It is like sticking every installed package in your world file. That comparison is quite persuasive, indeed. =:^) Thanks! I understand your idea much better, now, and (cautiously) agree. =:^) (Tho FWIW, I guess I'm a careful one. I use -* and put non-global USE flags in make.conf too if possible, and review USE flags for all new packages and changes, so everything there is cared about for one reason or another. Package.use thus contains only individual package exceptions, and I comment those with both a date and why they /are/ exceptions to the otherwise global policy, so if the only justification is because package X requires it, that's in the comment. Make.conf's USE= setting does still accumulate unannotated stale flags over time, but I just went thru and verified all USE flags were still used recently, deleting the ones that equery hasuse didn't raise a hit on, and justifying either by-name or by equery uses every remaining entry, so everything there is verified there for a reason now, too.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-portage-dev] Re: Dynamic USE dependencies
On Fri, 3 Apr 2015 11:52:39 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Brian Dolbec posted on Thu, 02 Apr 2015 23:59:06 -0700 as excerpted: This is a reminder to you all... since enalyze is little known to users. Thanks for the hint. I'd never heard of enalyze before but it sounds useful. I'll investigate. =:^) Please consider writing up a tips-n-tricks section feature for enalyze in an upcoming gentoo news, too. That should bring it rather more exposure among the power users at least, who will propagate the knowledge thru the lists and forums, etc. I know that's the way I've come across several tools I now keep in my virtual toolbox, over the years. =:^) Been there, done that... ;) https://blogs.gentoo.org/news/2014/03/ And there are several main #gentoo contributors that know about it. I keep coming across people that don't know about it still. -- Brian Dolbec dolsen
Re: [gentoo-portage-dev] Re: Dynamic USE dependencies
On Thu, Apr 2, 2015 at 10:10 PM, Duncan 1i5t5.dun...@cox.net wrote: Rich Freeman posted on Thu, 02 Apr 2015 12:32:41 -0400 as excerpted: Out of curiosity, what is keeping us from having USE flag dependencies handled dynamically, in the same way that package dependencies are? If portage can figure out that I need libxml2 installed even if I don't put it in /var/lib/portage/world, why can't it figure out that I need it built with USE=icu even if I don't put that in /etc/portage/package.use? Because USE flags are binary, with not-yes implying no, while world set listings are binary, with not-yes implying do it anyway if needed? That doesn't need to be true. Not listing a flag in your config can just mean do it anyway if needed. Package USE dependencies already work this way - you can depend on a flag being set, on it being unset, or not mention the flag at all which means the package doesn't care. Changing the implied meaning of not-yes to match in both cases could certainly be done, but while I'm not opposed if the justification really is there, I think there's the potential here for a rather massive change in base assumptions, and if that is indeed what's involved, I believe it'd call for equally massive justifications. There is no question that it would be a change in behavior. However, I wouldn't anticipate a lot of changes. If you stuck -* in your make.conf then this change would have no affect at all, since you've explicitly set the configuration of every use flag. Anything already in package.use would still stick, since that is also configuration. If you wiped your package.use clean then there would be a possible change, but that is something the user has control over, and presumably for them it is a change for the better if they're making it. Things would still default to their profile/package defaults as they already do if you don't stick anything in your config files. However, future package installs that require a USE change would just make the change if you didn't explicitly bar that configuration. It wouldn't modify your config files. And I'm worried that the suggestion here is going further down that emerge writing its own config path, without (IMO) appropriate safeguards. I think this is exactly the opposite. It would get rid of the need to have portage go modifying package.use at all in many cases. You'd still need to modify configuration (perhaps with automatic help) if you had explicitly set something which conflicts with what you want to install (such as USE=-* globally). However, for users who just go with the profile defaults they wouldn't have to stick nearly as much stuff in package.use just to change a flag setting they never expressed a personal preference for one way or the other. The current configuration forces you to use config files to capture settings you care about, and also ones you don't actually care about, and unless you're careful you'll have trouble telling these settings apart later. It is like sticking every installed package in your world file. -- Rich
[gentoo-portage-dev] Re: Dynamic USE dependencies
Rich Freeman posted on Thu, 02 Apr 2015 12:32:41 -0400 as excerpted: Out of curiosity, what is keeping us from having USE flag dependencies handled dynamically, in the same way that package dependencies are? If portage can figure out that I need libxml2 installed even if I don't put it in /var/lib/portage/world, why can't it figure out that I need it built with USE=icu even if I don't put that in /etc/portage/package.use? Because USE flags are binary, with not-yes implying no, while world set listings are binary, with not-yes implying do it anyway if needed? OK, so the USE flag not-yes implied no /is/ a bit weak, packages omit the USE flag if they (or their maintainer) actually require the feature and simply hard-require what would otherwise be toggled by the USE flag, but by that same token, the very fact that the USE flag exists makes it an option for the admin that would toggle the flag, strengthening the USE flag's implied-no of the not-yes, and in any case, it's still *far* stronger than the do-it-anyway-if-needed of simply not listing a package in the world set. Meanwhile, there is of course a way to have no for a world listing, put it in package.mask. Similarly, there'a a way to enforce an explicit no for that implied by a USE not-yes, by setting USE=-* at the beginning, and users who eventually get tired of having to worry about profile changes meaning implied USE flag changes, etc, may well eventually set it, as I did. But that doesn't change the basic difference in what not- yes is implied to mean in the two cases. Changing the implied meaning of not-yes to match in both cases could certainly be done, but while I'm not opposed if the justification really is there, I think there's the potential here for a rather massive change in base assumptions, and if that is indeed what's involved, I believe it'd call for equally massive justifications. OTOH, maybe you're thinking something a bit more incremental, which would accordingly require lower justification. I'm just a bit worried... ... And I /still/ don't like that --ask, implies that I think it's OK for portage to write to my package.* files (even with config-protection), if I accidentally hit enter. When the danger was simply that a package merge that would take some time and thus could easily be aborted, that was IMO reasonable. The new situation is IMO far more borderline. I'd be a whole lot more comfortable if some EMERGE_DEFAULTs parameter could be set to turn off portage's writing to package.* entirely, while keeping the old --ask package-merging /and/ use-flag/mask/keywording SUGGESTION behavior. And I'm worried that the suggestion here is going further down that emerge writing its own config path, without (IMO) appropriate safeguards. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
