Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 30 September 2006 15:34, Brian Harring wrote:
If that's what folks want, sure, but what you're proposing is just
sliding NEEDED in as the defacto solution without labeling it as such.
no idea what this means
Re-read your emails, and mine please. The scenario I pointed out was
ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2.
then let me turn it around ... how exactly does your solution account for all
these little details ? i dont recall you outlining anything ...
How exactly is this forcing wasted rebuilds? You're stating that
maintainers are going to bump it willy nilly. As I said, it is a key
that would be bumped *as needed*, and would only affected pkgs that
specified that node as a binding dep (specially marked atom).
no, i mean for example scenarios where a package provides more than one
library (say libfoo and libbar) and only one of those changes ABI values
(say libfoo.0 - libfoo.1). if another package links against just
libbar, you've got a pointless rebuild.
If one changes it's version, it's a fair bet that any consumer of that
pkg is linked to more then just that lib; as such they would be
rebuilt *anyways*.
it isnt guaranteed ... you asked for a pointless rebuild and i gave you
one ... one that is not falsely flagged when reviewing the NEEDED list
the other example is where the ABI changes for one arch but not for any
other ... what do you do, force ABI rebuild for everyone ? ok, maybe that
arch was x86 so we say screw you to the smaller arch teams ... but what if
that arch was a smaller arch team ?
Sorry, but if a developer is bumping a pkg and doesn't even look to
see if the damn thing is potentially going to break others via soname
changes, that maintainer is being an idiot.
and yet it does happen and again, we're looking at this from different
angles ... you'd rather assume the developer is 100% competent and never gets
things wrong; i'd rather assume nothing and let the details be discovered
automatically.
in the end, the people using Gentoo are the ones that suffer and i'm tired of
seeing systems broken beyond usuability because a developer unleashed a
package without realizing the consequences of it
Realistically, just the same as the NEEDED solution has holes, the
maintaining dev can screw up and miss a BINCOMPAT bump. Difference is
that they can do something for BINCOMPAT; hell, QA checks can be
automated to catch missing BINCOMPAT bumps.
i bet a post-emerge would make this painless and automate the QA step ...
after src_install(), you run something like `scanelf -qsRS ${D}` and save the
results in vdb/SONAME ... and then on subsequent installs, you run it again
and if the SONAME changes but BINCOMPAT did not, you bail with a die message
telling the user to go file a bug and preventing the merge which would have
broken his system
KEYWORDed arches bit, bit unlikely that the underlying arch is
actually going to screw with the soname, thus I'd want actual examples
backing it up.
how about libc.so from glibc and libgcc.so from gcc ? or would you like
other packages ?
Considering such a change would be internal ABI, NEEDED doesn't
actually fix anything; if it were a soname change, level playing
ground again.
it is a SONAME change, why else would i mention this
Reiterating; devs should know the high level affects of their changes.
reiterating: you're hoping for the best; i'm looking at our history and
assuming that devs will make mistakes as everyone does
If it's going to change the soname version, they should know from the
get go- unless it's an arch specific breakage (which I still posit is
the rare/corner case), they will *see* it for their arch and bump it
already.
maybe, but it's still a possibility which analyzing of SONAME would catch
Stating that things are broken doesn't make NEEDED automagically
better either; *both* enable a way to fix it, so you need to justify
the accounting for the worst; not hoping for the best.
justify how ? would you like me to dig up the bugs where devs bumped packages
into stable and the SONAME change broke a ton of people's systems ? has it
never happened to you ?
or dig
through the source code line by line and hope to catch all such cases
by hand/eye ?
Bit of FUD here, although I spose I'll just point out that if so's
change as massively as you're implying above, the affects on -p are
that much worse.
awesome, mark something you disagree with as FUD, problem solved. my
point is that you can never know completely for sure the behavior of a
package without digging through the entire source code.
It's FUD due to the fact NEEDED suffers the same fucking issue. The
irony is that BINCOMPAT would at least give a knob to mark it as a
breakage, NEEDED cannot.
and then you get angry ? try to have a discussion without getting all anxious
as that only wastes time
there's a few
Re: [gentoo-dev] RFC about another *DEPEND variable
On Wednesday 27 September 2006 03:54, Brian Harring wrote: On Wed, Sep 27, 2006 at 02:24:41AM -0400, Mike Frysinger wrote: as i said, if you have changed ABI without an ABI bump, then the upstream package maintainer is screwing everyone who uses the package, not just Gentoo ... so perhaps we should be talking to them to get the situation resolved for all consumers, not just Gentoo Happens however; afaik, we also weren't monkey patching ssl in the past to preserve the abi either so it still is valid (although rare if upstream is behaving) scenario. we've never monkey patched ssl so i dont really know what you're referring to Bleh, this is getting back to exactly my point that it's unbounded resolution. To support this, every step of execution would require scanning for dangling nodes to punt; aside from invalidating -p's output it *still* is a collision-protect hit. when the package upgrade detects an ABI change you recalculate the packages that need to be rebuilt ... Reiterating, this screws over any form of up front calculation; dialup users/per minute connections can't rely on emerge -f to actually fetch all involved sources, -p results ain't guranteed valid, parallelization must now lock at each threads merge on the off chance a recalc is forced. it does indeed prevent full up front calculation. we can always make the behavior configurable; revdep on the fly or allow people to break it up. the key being that their system will still continue to function as the ABI lib has been preserved automagically dangling nodes get recorded in the new package and considering these old files are not detrimental to the health of the system, you can do such a cleanup once at the end of the emerge It's not orphaning files, but your scheme doesn't work for any form of interruption; failed builds, user intervention, etc, they all can leave the lib stuck in the new contents. huh ? i outlined in a previous e-mail how by simply ordering the operations sanely, there is no race condition Dealing with that possibility means the manager has to maintain on disk a list of the refcount of each dangling libs to decrement, unmerge has to modify said list, etc. which is already being done in the NEEDED file ... funny how unpainless it is to generate that file It also involves changing vdb nodes from installed and usable to installed/usable or lingering no ... the old versions get merged into the new one as their existence is purely hidden How exactly are they going to be hidden? A new file? If so, backwards compatibility for vdb for transitioning in such a support has to be addressed. purely hidden from the standpoint of any new package trying to use it ... since you're only installing the runtime ABI library, you cannot link against it or utilize it any way other than existing files Tracking BINCOMPAT should *not* be that hard. it's one more thing to keep track of and considering all of the possibilities i outlined, a single maintainer can easily lose his sanity ... or you force wasted rebuilds on people when it's only needed in some circumstances How exactly is this forcing wasted rebuilds? You're stating that maintainers are going to bump it willy nilly. As I said, it is a key that would be bumped *as needed*, and would only affected pkgs that specified that node as a binding dep (specially marked atom). no, i mean for example scenarios where a package provides more than one library (say libfoo and libbar) and only one of those changes ABI values (say libfoo.0 - libfoo.1). if another package links against just libbar, you've got a pointless rebuild. Seriously, maintainers ought to know *now* when they're forcing a revdep on folks systems, I'm not seeing the massive overhead from pushing that info into a var, nor am I seeing mass forced useless rebuilds. there's a ton of things maintainers ought to know ... pretty soon our package maintainers are going to have to be gods if they want to write an ebuild as they're going to have to know every detail about the package inside and out Realistically, just the same as the NEEDED solution has holes, the maintaining dev can screw up and miss a BINCOMPAT bump. Difference is that they can do something for BINCOMPAT; hell, QA checks can be automated to catch missing BINCOMPAT bumps. what's the difference ? in my scenario they dont have to do anything because the bump would have been caught automatically ? KEYWORDed arches bit, bit unlikely that the underlying arch is actually going to screw with the soname, thus I'd want actual examples backing it up. how about libc.so from glibc and libgcc.so from gcc ? or would you like other packages ? Besides, again, for keywording, the dev *should* be compiling it, so there is a way to catch it :P. A revbump isn't going to break things unless the dev is introducing something intrusive, minor version bumps (1.1.0 to
Re: [gentoo-dev] RFC about another *DEPEND variable
On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote: On Wednesday 27 September 2006 03:54, Brian Harring wrote: Bleh, this is getting back to exactly my point that it's unbounded resolution. To support this, every step of execution would require scanning for dangling nodes to punt; aside from invalidating -p's output it *still* is a collision-protect hit. when the package upgrade detects an ABI change you recalculate the packages that need to be rebuilt ... Reiterating, this screws over any form of up front calculation; dialup users/per minute connections can't rely on emerge -f to actually fetch all involved sources, -p results ain't guranteed valid, parallelization must now lock at each threads merge on the off chance a recalc is forced. it does indeed prevent full up front calculation. we can always make the behavior configurable; revdep on the fly or allow people to break it up. the key being that their system will still continue to function as the ABI lib has been preserved automagically Folks aren't that daft; you make it an optional component, it means a *proper* solution will never be pushed because the duct tape is in place already. If that's what folks want, sure, but what you're proposing is just sliding NEEDED in as the defacto solution without labeling it as such. dangling nodes get recorded in the new package and considering these old files are not detrimental to the health of the system, you can do such a cleanup once at the end of the emerge It's not orphaning files, but your scheme doesn't work for any form of interruption; failed builds, user intervention, etc, they all can leave the lib stuck in the new contents. huh ? i outlined in a previous e-mail how by simply ordering the operations sanely, there is no race condition Re-read your emails, and mine please. The scenario I pointed out was ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2. Now how does portage know that it needs to complete that upgrade for the next emerge action? How does it know to even scan for lib1, let alone punting? It's *not* simple, and I keep pointing out this issue (and you're missing it every damn time). Please address it, or point to where you have (gone over the thread and still not seeing anything even remotely touching this flaw). Dealing with that possibility means the manager has to maintain on disk a list of the refcount of each dangling libs to decrement, unmerge has to modify said list, etc. which is already being done in the NEEDED file ... funny how unpainless it is to generate that file First of all, unpainless is painful. Which is apt, actually. Familiar with old style virtuals? That whole, you have to walk the whole vdb to collect all old style virtuals ? This is the same damn thing. There is no refcount maintained via NEEDED, just a list of libs a binary uses; you _still_ have to walk the damn vdb to build the refcount list. Now either you're forcing a fairly huge mapping in memory, or you're forcing a scan of the vdb for every pkg operation that has NEEDED entries it will install. So no, NEEDED doesn't cover this, and my point still stands. It also involves changing vdb nodes from installed and usable to installed/usable or lingering no ... the old versions get merged into the new one as their existence is purely hidden How exactly are they going to be hidden? A new file? If so, backwards compatibility for vdb for transitioning in such a support has to be addressed. purely hidden from the standpoint of any new package trying to use it ... since you're only installing the runtime ABI library, you cannot link against it or utilize it any way other than existing files. Except for dlopen, but thats again besides my original point; how do you note to portage that the lib is one to watch so it can be punted? You're suggesting it get shoved into contents, and for portage to identify it, it has to do a revdep mapping on the fly to find it. This *sucks* massively, both from a speed and complexity standpoint; further, the lib isn't hidden from pkg ops (say quickpkging, or binpkging) in any way so the cruft spreads. That's surprisingly minor in comparison to implications of relying on refcounting to identify the lib to punt. If the lib to punt isn't tracked in some fashion, the only algo you're left with is attempting to find all libs that have a refcount of zero, and punt those- obviously, this is going to screw up just about every single dlopen based linkage, literally, suck an algo won't spot that python dlopen's it's modules, and would think the refcount was zero (thus the so could get booted). The response there is to add blacklists, directories to *not* inspect, which is a further hack to try and make this tank fly. *OR*, you're stuck maintaining a seperated list of libs to punt, rather then just
Re: [gentoo-dev] RFC about another *DEPEND variable
On Monday 25 September 2006 14:16, Brian Harring wrote: Bad soname handling is just *part* of what BINCOMPAT could do; it's not the sole reason for it's existance, as such it's not quite right dismissing it just because it addresses a rarity the NEEDED approach doesn't. :) i dismiss it as being a real advantage and/or argument for taking BINCOMPAT over an automated NEEDED scan as i said, if you have changed ABI without an ABI bump, then the upstream package maintainer is screwing everyone who uses the package, not just Gentoo ... so perhaps we should be talking to them to get the situation resolved for all consumers, not just Gentoo *IF* we actually had that in place it would enable detecting and rebuilding c++ code whenever gcc pulls its next c++ abi change with appropriate deps in place (iow, kill off the implicit system deps). scanelf already does this ... the only time you need to rebuild is when the ABI breaks ... aka libstdc++.so.5 - libstdc++.so.6 If that were the case, why do we have libraries listed in DEPEND then? because i have a short memory Bleh, this is getting back to exactly my point that it's unbounded resolution. To support this, every step of execution would require scanning for dangling nodes to punt; aside from invalidating -p's output it *still* is a collision-protect hit. when the package upgrade detects an ABI change you recalculate the packages that need to be rebuilt ... dangling nodes get recorded in the new package and considering these old files are not detrimental to the health of the system, you can do such a cleanup once at the end of the emerge It also involves changing vdb nodes from installed and usable to installed/usable or lingering no ... the old versions get merged into the new one as their existence is purely hidden Tracking BINCOMPAT should *not* be that hard. it's one more thing to keep track of and considering all of the possibilities i outlined, a single maintainer can easily lose his sanity ... or you force wasted rebuilds on people when it's only needed in some circumstances Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED data (folks should be compiling the pkg anyways), point is it's mainly common sense for maintainenance of it. and when are you going to run that tool ? when you bump the package ? so now the maintainer has to test it on all the KEYWORDed arches with all the USE combos, or de-KEYWORD it and make the arch maintainers test it ? or dig through the source code line by line and hope to catch all such cases by hand/eye ? Yes, if the solution can be automated without flinging poo into the code, I'm for it; that said I know what the implementation is going to have to look like, and it's *nasty*. i dont think either of our solutions are satisfactory for all presented cases; a hybrid model would be needed -mike pgpq4H5zbMG0Q.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Wed, Sep 27, 2006 at 02:24:41AM -0400, Mike Frysinger wrote: On Monday 25 September 2006 14:16, Brian Harring wrote: Bad soname handling is just *part* of what BINCOMPAT could do; it's not the sole reason for it's existance, as such it's not quite right dismissing it just because it addresses a rarity the NEEDED approach doesn't. :) i dismiss it as being a real advantage and/or argument for taking BINCOMPAT over an automated NEEDED scan as i said, if you have changed ABI without an ABI bump, then the upstream package maintainer is screwing everyone who uses the package, not just Gentoo ... so perhaps we should be talking to them to get the situation resolved for all consumers, not just Gentoo Happens however; afaik, we also weren't monkey patching ssl in the past to preserve the abi either so it still is valid (although rare if upstream is behaving) scenario. *IF* we actually had that in place it would enable detecting and rebuilding c++ code whenever gcc pulls its next c++ abi change with appropriate deps in place (iow, kill off the implicit system deps). scanelf already does this ... the only time you need to rebuild is when the ABI breaks ... aka libstdc++.so.5 - libstdc++.so.6 If that were the case, why do we have libraries listed in DEPEND then? because i have a short memory Bleh, this is getting back to exactly my point that it's unbounded resolution. To support this, every step of execution would require scanning for dangling nodes to punt; aside from invalidating -p's output it *still* is a collision-protect hit. when the package upgrade detects an ABI change you recalculate the packages that need to be rebuilt ... Reiterating, this screws over any form of up front calculation; dialup users/per minute connections can't rely on emerge -f to actually fetch all involved sources, -p results ain't guranteed valid, parallelization must now lock at each threads merge on the off chance a recalc is forced. This is a rather huge step *backwards*; it actually puts us at a similar level to autopackages 'resolver'. The approach would have to solve it perfectly imo for it to even be realistically considered. For your proposal to fly, it must address this somehow imo which isn't possible from where I'm sitting. dangling nodes get recorded in the new package and considering these old files are not detrimental to the health of the system, you can do such a cleanup once at the end of the emerge It's not orphaning files, but your scheme doesn't work for any form of interruption; failed builds, user intervention, etc, they all can leave the lib stuck in the new contents. Dealing with that possibility means the manager has to maintain on disk a list of the refcount of each dangling libs to decrement, unmerge has to modify said list, etc. Further nastyness in short. It also involves changing vdb nodes from installed and usable to installed/usable or lingering no ... the old versions get merged into the new one as their existence is purely hidden How exactly are they going to be hidden? A new file? If so, backwards compatibility for vdb for transitioning in such a support has to be addressed. Tracking BINCOMPAT should *not* be that hard. it's one more thing to keep track of and considering all of the possibilities i outlined, a single maintainer can easily lose his sanity ... or you force wasted rebuilds on people when it's only needed in some circumstances How exactly is this forcing wasted rebuilds? You're stating that maintainers are going to bump it willy nilly. As I said, it is a key that would be bumped *as needed*, and would only affected pkgs that specified that node as a binding dep (specially marked atom). Seriously, maintainers ought to know *now* when they're forcing a revdep on folks systems, I'm not seeing the massive overhead from pushing that info into a var, nor am I seeing mass forced useless rebuilds. What I *am* seeing is one cluster fuck done to resolution and some nasty side affects you're glossing over. Invalidating -p's output opens up some nasty issues. To be fair, one claimed con could be that it is one more knob maintainers have to fool with, although counter is that they should already know about the issue (see dev quiz about what should be done when bumping a lib for an example). Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED data (folks should be compiling the pkg anyways), point is it's mainly common sense for maintainenance of it. and when are you going to run that tool ? when you bump the package ? One would hope that when bumping a package, the dev actually attempts to *compile* the damn thing. Theres your opportunity to run a check, hell integrate it as a feature into portage that does the check for soname change. Gurantee that mod is far less intrusive, and far less error prone then what you're proposing.
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 10:30, Brian Harring wrote: dlopen? we already said that this will need a new depend variable How does this fix openssl horkage? (bad soname handling) it wont, but such things are broken regardless outside of Gentoo ... and trying to accommodate something that happens every three blue moons at the cost of developer time is not worth it Also... what do we do for python/perl (*-updater scripts in general) where a change in a pkg state means we have to rebuild the revdeps? my solution does not address this, but what you're proposing would over address this ... how do you know when a package needs to be rebuilt (a perl module) or rebuilding is a waste of time (a package installs perl scripts that execute `perl` and nothing else) What you're suggesting works for strictly linkage; still think this shouuld work for the general problem rather then just one subset. yes, i am addressing what i see to be the most critical issue and the easiest to break a user's system Clarifying my statement; we don't break our DEPEND down into this is what is executed in building the package (toolchain), DEPEND vs this is the crap the binaries we build against need avail to be linked against, literally what winds up as -l args. RDEPEND If punting the old lib (as I assumed), means we would potentially be making certain DEPEND atoms unusable if they're required in an execution context (rather then just winding up as a -l arg). no ... lemme give a perfect example. user has openssl-0.9.7j installed and they upgrade to openssl-0.9.8c ... the old version provided SONAME files libcrypto.so.0.9.7 and libssl.so.0.9.7 while the new one provides SONAME files libcrypto.so.0.9.8 and libssl.so.0.9.8 everything from openssl-0.9.7j is unmerged except for the two files libcrypto.so.0.9.7 and libssl.so.0.9.7. openssl-0.9.8c, being a different ABI, does not provide these files thus there is no clash. portage keeps track of libcrypto.so.0.9.7 and libssl.so.0.9.7 and once no more packages have NEEDED entries for these, will silently clean them out openssl is odd in that it encodes .x.y.z version into the ABI ... if we use the more common example with say expat, older versions install libexpat.so.0.1, libexpat.so.0.2, libexpat.so.0.3, etc... but the ABI SONAME is still just libexpat.so.0. when the next major version of expat comes out, the SONAME is bumped to libexpat.so.1 and portage needs to keep around the last libexpat.so.0 In that case, wouldn't mind a response to the what about ctrl+c during the run? The potential for orphaning there sucks; recording the old library in the new version sucks also since it complicates the merge process, that lib *must* be removed else it's a potential collision-protect minefield. - portage merges new version 2 to $ROOT - system now has version 1 and version 2 in $ROOT - portage starts to clean out version 1 from $ROOT - do not fully trim version 1's vdb until version 2 has been updated with the new information so ctrl+c at any point and so what ? you dont remove old files until new files are fully placed and you can resume at any point Finally, even if the lib is temporarily left behind, this solution doesn't gurantee the library actually would *work* still- it only can work if the lib is standalone from the rest of the pkg and doesn't rely on any external data from the pkg. we're talking about preserving the system long enough to rebuild things; we're not talking about keeping the system forever in a sane state. i would guess that this corner case is not the norm and thus we can ignore it as the situation is still a lot better than what we have now: $ foo foo: error while loading shared libraries: libbar.so.1: cannot open shared object file: No such file or directory awesome Basically trying to point out that what you're proposing only works in a subset of the cases revdep must deal with, and that revdep itself doesn't deal with *all* situations as is; hence BINCOMPAT as a way to try and shift it under maintainers control. revdep-rebuild doesnt take into consideration any of the issues you raised and forcing maintainers to always track BINCOMPAT is unwieldy ... a single package provides multiple SONAMEs ? bitrot ? SONAME is dynamic based upon architecture or USE flags ? -mike pgpmiqh87n1FA.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 22:36, Ciaran McCreesh wrote: How would it know what other files are required? For example, if libexpat.so.0 were to rely upon /usr/share/expat-0/config , how would the package manager know not to clobber that file? Or are you suggesting leaving (or reparenting, if you prefer) all a package's files, not just the .so files? i'm talking about just libexpat.so.0 ... keep the file around long enough to update the system Or a related question: what proportion of breakages will be fixed merely by keeping .so files and nothing else around? Will implementing this prevent enough breakages to make it worthwhile? of course it will when you look at core things like openssl, acl, openldap, etc... if you remove anyone one of these packages completely, you risk breaking everything (coreutils and many system packages use libacl) or all your network capabilities (sshd/ssh no longer usuable; now you require on-site service to fix ... hopefully you have packages cached cause wget no longer works either) -mike pgp9kTJ8dIDgk.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Sat, Sep 23, 2006 at 11:31:12PM -0400, Mike Frysinger wrote: On Saturday 23 September 2006 10:30, Brian Harring wrote: dlopen? we already said that this will need a new depend variable How does this fix openssl horkage? (bad soname handling) it wont, but such things are broken regardless outside of Gentoo ... and trying to accommodate something that happens every three blue moons at the cost of developer time is not worth it Bad soname handling is just *part* of what BINCOMPAT could do; it's not the sole reason for it's existance, as such it's not quite right dismissing it just because it addresses a rarity the NEEDED approach doesn't. :) Also... what do we do for python/perl (*-updater scripts in general) where a change in a pkg state means we have to rebuild the revdeps? my solution does not address this, but what you're proposing would over address this ... how do you know when a package needs to be rebuilt (a perl module) or rebuilding is a waste of time (a package installs perl scripts that execute `perl` and nothing else) Original email stated that 'binding deps' would be required for that, marking deps in some way such that it indicates they're sensitive to changes in BINCOMPAT of the match. *IF* we actually had that in place it would enable detecting and rebuilding c++ code whenever gcc pulls its next c++ abi change with appropriate deps in place (iow, kill off the implicit system deps). Back to your example, if it's just a caller of it, it's not binding; now if it were a perl module that sticks its modules into the perl installation, yes, binding (it needs to rebuild to merge to the new location). What you're suggesting works for strictly linkage; still think this shouuld work for the general problem rather then just one subset. yes, i am addressing what i see to be the most critical issue and the easiest to break a user's system Clarifying my statement; we don't break our DEPEND down into this is what is executed in building the package (toolchain), DEPEND vs this is the crap the binaries we build against need avail to be linked against, literally what winds up as -l args. RDEPEND If that were the case, why do we have libraries listed in DEPEND then? DEPEND is (and always has been) this is the crap I need merged to be able to build an install image of myself, RDEPEND is (and always has been) this is the crap I need on the fs to actually run my binaries/libs and PDEPEND is around to cover up portages resolver, but moreso the trees mostly whacked deps. This is why eradicator, solar, and you were discussing splitting link depends out of DEPEND for saner CHOST/CTARGET support around a year back also. Semantics at this point, but RDEPEND does *not* need to be merged to build a pkg, only DEPEND; it's never been any other way. If punting the old lib (as I assumed), means we would potentially be making certain DEPEND atoms unusable if they're required in an execution context (rather then just winding up as a -l arg). no ... lemme give a perfect example. user has openssl-0.9.7j installed and they upgrade to openssl-0.9.8c ... the old version provided SONAME files libcrypto.so.0.9.7 and libssl.so.0.9.7 while the new one provides SONAME files libcrypto.so.0.9.8 and libssl.so.0.9.8 everything from openssl-0.9.7j is unmerged except for the two files libcrypto.so.0.9.7 and libssl.so.0.9.7. openssl-0.9.8c, being a different ABI, does not provide these files thus there is no clash. portage keeps track of libcrypto.so.0.9.7 and libssl.so.0.9.7 and once no more packages have NEEDED entries for these, will silently clean them out openssl is odd in that it encodes .x.y.z version into the ABI ... if we use the more common example with say expat, older versions install libexpat.so.0.1, libexpat.so.0.2, libexpat.so.0.3, etc... but the ABI SONAME is still just libexpat.so.0. when the next major version of expat comes out, the SONAME is bumped to libexpat.so.1 and portage needs to keep around the last libexpat.so.0 In that case, wouldn't mind a response to the what about ctrl+c during the run? The potential for orphaning there sucks; recording the old library in the new version sucks also since it complicates the merge process, that lib *must* be removed else it's a potential collision-protect minefield. - portage merges new version 2 to $ROOT - system now has version 1 and version 2 in $ROOT - portage starts to clean out version 1 from $ROOT - do not fully trim version 1's vdb until version 2 has been updated with the new information so ctrl+c at any point and so what ? you dont remove old files until new files are fully placed and you can resume at any point Bleh, this is getting back to exactly my point that it's unbounded resolution. To support this, every step of execution would require scanning for dangling nodes to punt; aside from invalidating
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 13:15, Alin Nastac wrote: Unless you save the specific compatibility version of the net-dialup/ppp used by net-dialup/pptpd for building the package, I don't see how can it help me. Judging after /var/db/pkg content, I have no such information. it is all there right now actually :) check out the NEEDED file ... combine that with CONTENTS of other packages, and you have all the binding info you want ... -mike pgp8bgoWaDBYG.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 11:41, Duncan Coutts wrote: On Thu, 2006-09-21 at 11:11 -0400, Mike Frysinger wrote: On Thursday 21 September 2006 10:56, Duncan Coutts wrote: If we do go in this direction it'd be great to be able to slot on the ABI and still have dependencies resolved correctly. For example imagine having parallel python-2.3 and 2.4 installations with some libs installed for both. Crucially, deps need to be resolved to the version of a lib with the right ABI. ugh, no ... we are not a binary distribution so we should not have to worry about ABI baggage So we can't ever install two versions of python or ghc at once? That seems a shame. that's an issue for the python or ghc maintainers to address -mike pgpXbGNFGDdo9.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 06:35, Duncan Coutts wrote: I was worried from your ABI/API comments that you meant that we should never be allowed to do it. i was commenting on the more general case; SLOTing something that wasnt meant to be SLOTed -mike pgpQseGjV9xuk.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Sat, Sep 23, 2006 at 06:20:44AM -0400, Mike Frysinger wrote: On Thursday 21 September 2006 11:08, Brian Harring wrote: On Thu, Sep 21, 2006 at 10:43:11AM -0400, Mike Frysinger wrote: i'm referring to the specific file of course, not anything else in the package ... so integrating the hack eutils.eclass:preserve_old_lib() into portage so it isnt a hack (not a knock against the current implementation here; it's always going to be a hack until portage manages proper unmerging of the ABI library) The reason folks aren't talking about using NEEDED is that NEEDED data is generated _after_ building; as well it should be ... trying to enumerate the linking ABI possibilities before hand is not doable and not worth wasting the time of maintainers when it can be automated getting the info into the resolver up front allows for a helluva lot more options, and makes stuff like ensuring you have all sources required downloaded *prior* actually simple to do, rather then inserting recalculating hacks into the resolver. rather than integrating it all into the resolver in a one-pass system, a two pass system would work: - build all the packages requested - after each package, if an ABI library is being removed, check to see if it is needed by any other package. if it is needed, record it and preserve it and move on You're assuming that after the merge of the pkg that breaks compatibility, building is actually _still_ possible for the depends. We don't classify our deps as actual build depends vs link depends; as such trying to (essentially) patch things up after allow for the scenario where merging breaks the toolchain, thus building isn't possible. Because we don't classify the type of build time dep, that means each DEPEND match must have it's run time depends (RDEPEND) satisfied prior to being usable as a DEPEND; that right there punches a whole in the delay till the end approach, and is a good example of what I mean when I say this is unbounded resolution; the only way to solve it is to redo the resolution between finished building and merging the pkg to determine if it will break any required DEPENDS for later steps. - once all the packages requested have been merged, you start the second phase and calculate everything that needs to be rebuilt. as ABI libs are no longer needed on a system, portage can scrub them out as ABI libs are no longer needed on a system, phrasing of that implies you're suggesting that portage should leave the older package in place till it's updated all revdeps, then wipe it. Which is fairly nasty, especially if you factor in the user potential of ctrl+c'ing it. Finally, if I'm interpretting your statement there correctly, still isn't guranteed to work- just having the lib around doesn't mean that the older package is left in a working state with the new partially merged over it, only way that would work is if the two were slotted (indicating they could coexist on disk). ~harring pgpW99BPE62sk.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 09:14, Brian Harring wrote: You're assuming that after the merge of the pkg that breaks compatibility, building is actually _still_ possible for the depends. of course i am; i just said that portage would make sure to not unmerge any ABI lib still in use We don't classify our deps as actual build depends vs link depends; as such trying to (essentially) patch things up after allow for the scenario where merging breaks the toolchain, thus building isn't possible. huh ? RDEPEND is linktime ... see my statement above - once all the packages requested have been merged, you start the second phase and calculate everything that needs to be rebuilt. as ABI libs are no longer needed on a system, portage can scrub them out as ABI libs are no longer needed on a system, phrasing of that implies you're suggesting that portage should leave the older package in place till it's updated all revdeps, then wipe it. no i am not; read my previous e-mails where i said it would leave behind the 1 ABI lib required ... aka whatever is encoded in SONAME -mike pgpyAGCZ2QnUO.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 09:50, Mike Frysinger wrote: On Saturday 23 September 2006 09:14, Brian Harring wrote: We don't classify our deps as actual build depends vs link depends; as such trying to (essentially) patch things up after allow for the scenario where merging breaks the toolchain, thus building isn't possible. huh ? RDEPEND is linktime ... see my statement above lemme rephrase ... you keep the ABI SONAME lib around until all packages are done with it; this does not affect any other package (including upgrades to the same package) as the ABI filename is unique in the filesystem it is not used at linktime because it does not provide a linkable file ... it merely provides the library used at runtime so you can upgrade on the fly as the old ABI lib does not conflict with anything -mike pgpWyxlIfwykH.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Mike Frysinger wrote: On Thursday 21 September 2006 13:15, Alin Nastac wrote: Unless you save the specific compatibility version of the net-dialup/ppp used by net-dialup/pptpd for building the package, I don't see how can it help me. Judging after /var/db/pkg content, I have no such information. it is all there right now actually :) check out the NEEDED file ... combine that with CONTENTS of other packages, and you have all the binding info you want ... I see only libraries in NEEDED and it is probably generated automatically. There is no way for the automatic tools to discover the dependency between pptpd and ppp version. Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file of the pptpd, the amount of computation needed to discover which package offers such thing would be prohibitive. The reciprocal operation (find which packages use the old path before upgrade) would also be prohibitive. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Sat, Sep 23, 2006 at 09:50:12AM -0400, Mike Frysinger wrote: On Saturday 23 September 2006 09:14, Brian Harring wrote: You're assuming that after the merge of the pkg that breaks compatibility, building is actually _still_ possible for the depends. of course i am; i just said that portage would make sure to not unmerge any ABI lib still in use dlopen? How does this fix openssl horkage? (bad soname handling) Also... what do we do for python/perl (*-updater scripts in general) where a change in a pkg state means we have to rebuild the revdeps? What you're suggesting works for strictly linkage; still think this shouuld work for the general problem rather then just one subset. We don't classify our deps as actual build depends vs link depends; as such trying to (essentially) patch things up after allow for the scenario where merging breaks the toolchain, thus building isn't possible. huh ? RDEPEND is linktime ... see my statement above RDEPEND is execution requirements; to use the binary, this is what needs to be in the graph. Clarifying my statement; we don't break our DEPEND down into this is what is executed in building the package (toolchain), vs this is the crap the binaries we build against need avail to be linked against, literally what winds up as -l args. If punting the old lib (as I assumed), means we would potentially be making certain DEPEND atoms unusable if they're required in an execution context (rather then just winding up as a -l arg). So... ignore that bit since you're talking about lingering files. - once all the packages requested have been merged, you start the second phase and calculate everything that needs to be rebuilt. as ABI libs are no longer needed on a system, portage can scrub them out as ABI libs are no longer needed on a system, phrasing of that implies you're suggesting that portage should leave the older package in place till it's updated all revdeps, then wipe it. no i am not; read my previous e-mails where i said it would leave behind the 1 ABI lib required ... aka whatever is encoded in SONAME Yeah, missed the presvered (woot for 5am wakeup). In that case, wouldn't mind a response to the what about ctrl+c during the run? The potential for orphaning there sucks; recording the old library in the new version sucks also since it complicates the merge process, that lib *must* be removed else it's a potential collision-protect minefield. Finally, even if the lib is temporarily left behind, this solution doesn't gurantee the library actually would *work* still- it only can work if the lib is standalone from the rest of the pkg and doesn't rely on any external data from the pkg. Example would be pkg foobar that internally has libconvience, used by it's libs but not externally linked, contains (oddly enough) convience bits shared across foobars libraries. libconvience is *not* to be externally linked against, consumers must access the other libs (say libfoo); any soname bumps to libfoo, the old version gets broke in the process despite due to it linking internally against an unversioned so. Granted, semi retarded, but gnomes libegg comes to mind as a potential case of this. Basically trying to point out that what you're proposing only works in a subset of the cases revdep must deal with, and that revdep itself doesn't deal with *all* situations as is; hence BINCOMPAT as a way to try and shift it under maintainers control. Maintainence of it *should* be pretty simple also; for sane upstream soname handling, you just bump it with the majors; for the rest, its a knob that can be fiddled to at least give up front warning of the issue. ~harring pgphglIgSikSW.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Saturday 23 September 2006 10:24, Alin Nastac wrote:
I see only libraries in NEEDED and it is probably generated
automatically. There is no way for the automatic tools to discover the
dependency between pptpd and ppp version.
that gets back to ABI versus dynamic plugins ... we already know we'll need a
new DEPEND to track dlopen-ed plugins
Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file
of the pptpd, the amount of computation needed to discover which package
offers such thing would be prohibitive. The reciprocal operation (find
which packages use the old path before upgrade) would also be prohibitive.
no it wouldnt ... when you merge a package, you record all the SONAME's it
provides:
scanelf -qRS ${D} SONAME
in fact, running `scanelf -qlpRS` doesnt take that long on my machine
-mike
pgpvTUPVCKbO6.pgp
Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Sat, Sep 23, 2006 at 10:34:03AM -0400, Mike Frysinger wrote:
On Saturday 23 September 2006 10:24, Alin Nastac wrote:
I see only libraries in NEEDED and it is probably generated
automatically. There is no way for the automatic tools to discover the
dependency between pptpd and ppp version.
that gets back to ABI versus dynamic plugins ... we already know we'll need a
new DEPEND to track dlopen-ed plugins
Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file
of the pptpd, the amount of computation needed to discover which package
offers such thing would be prohibitive. The reciprocal operation (find
which packages use the old path before upgrade) would also be prohibitive.
no it wouldnt ... when you merge a package, you record all the SONAME's it
provides:
scanelf -qRS ${D} SONAME
in fact, running `scanelf -qlpRS` doesnt take that long on my machine
Flush the cache... Makes a world of difference. Additionally, he is
talking about what is *done* with that data after the fact, iow other
words walking the entire vdb to find all affected pkgs. Can pretty
much gurantee after a build that data isn't likely to be available
(part of the reason portageq calls during building are slow).
Could collapse that into a simple mapping, but that introduces
backwards compatibility issues...
~harring
pgpa8FBw1B83Y.pgp
Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, 21 Sep 2006 11:01:40 -0400 Mike Frysinger [EMAIL PROTECTED] wrote: | On Thursday 21 September 2006 10:54, Donnie Berkholz wrote: | Yes, I agree with you. For example, take expat. The maintainers have | refused to allow both versions to exist simultaneously on a system | because it apparently causes more breakage than just breaking every | app on your system by removing .so.0. | | that is the exact case portage should be handling for you | | it would go oh hey, check out libexpat.so.0 ... some things seem to | want it ... HEY USER, you need to rebuild: ... once all the | packages still consuming libexpat.so.0 are rebuilt, portage could | silently trim it from the system | | complicated ? not really, scanelf can produce all this information | in an easily digestable format How would it know what other files are required? For example, if libexpat.so.0 were to rely upon /usr/share/expat-0/config , how would the package manager know not to clobber that file? Or are you suggesting leaving (or reparenting, if you prefer) all a package's files, not just the .so files? Or a related question: what proportion of breakages will be fixed merely by keeping .so files and nothing else around? Will implementing this prevent enough breakages to make it worthwhile? -- Ciaran McCreesh Mail: ciaranm at ciaranm.org Web : http://ciaranm.org/ as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13 signature.asc Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Alin Nastac wrote: I reckon this could be solved by yet another *DEPEND variable. The only atoms accepted by this variable would be CATEGORY/PN. Every time when a package gets updated from PV1 to PV2 (distinct versions, revisions will not count), portage will automatically re-emerge those packages that depend on it. Thoughts? It would require revdep resolution on emerge... how painful would be? lu -- Luca Barbato Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote: Alin Nastac wrote: I reckon this could be solved by yet another *DEPEND variable. The only atoms accepted by this variable would be CATEGORY/PN. Every time when a package gets updated from PV1 to PV2 (distinct versions, revisions will not count), portage will automatically re-emerge those packages that depend on it. Thoughts? It would require revdep resolution on emerge... how painful would be? Off the top of my head, adding revdeps (period) for unmerge functionality (fex) is the harder part; slipping something of this sort in is just a logic tweak. The problem with this proposal however is that it's attempting automatic revdep based off of version; _any_ non-rev version bump is way too rebuild happy. Proposal I was pushing a while back was addition of a metadata key; it's not exactly .so version, but pretty close- a _manually_ maintained counter var in the ebuild that represents the abi compatibility for that packages versions. Example would be openssl-0.9.7, you stick BINCOMPAT=0.9.7 in it, in openssl-0.9.8, you stick BINCOMPAT=0.9.8 in it, for a replace op the resolver sees that it's breaking compatibility and knows to rebuild any revdeps. Why have the explicit var? Because 0.9.7a through 0.9.7c may all be compatible, but 0.9.7d isn't. If you force an automatic algo that tries to (effectively) guess, you get a lot of rebuilds through a,b,c, end result being folks likely update less because it becomes a bigger pain in the ass. There is one flaw with this though; packages can provide both libraries _and_ binaries. Our dependencies don't represent whether the dep is actual linkage, or just commandline consuming, so (using the openssl example) any package that invokes openssl via the commandline to do a few simple chksum ops gets rebuilt, despite the fact it wasn't affected by linkage change ups. So... short version, at least what jstubbs/marius/myself hashed out in irc a long while back, need binding dependencies and actual tracking of the lib breakage in the metadata. Alternative to shoving an extra key in would be extending slot rules, but that can be somewhat ugly. My 2 cents, for what its worth. ~harring pgpQ0KLtqw55W.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Luca Barbato wrote: Alin Nastac wrote: I reckon this could be solved by yet another *DEPEND variable. The only atoms accepted by this variable would be CATEGORY/PN. Every time when a package gets updated from PV1 to PV2 (distinct versions, revisions will not count), portage will automatically re-emerge those packages that depend on it. Thoughts? It would require revdep resolution on emerge... how painful would be? I don't know anything about portage intricacies, but I guess it would be fairly easy to have a map of CATEGORY1/PN1 - CATEGORY2/PN2 key-values, where package 2 depends on package 1 (package 2 is the one that defines the xxxDEPEND variable). In order to add such (key, value) in the map the following assumptions must be satisfied: - package 2 must be installed (mandatory) - package 1 must be a direct or indirect RDEPEND dependency of the package 2 (optional). signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 06:35, Alin Nastac wrote: For instance, the recent openssl-0.9.8* update broke dev-libs/neon (and consequently subversion) because neon library isn't happy just by linking with libssl.so.0.9.7 but also check the libssl version when loads the ssl library. Another example is the subtle dependency between the pppd version and pppd plugins (net-dialup/pptpd needs to be rebuild every time you change PV of the net-dialup/ppp because pptpd builds a plugin for the ppp daemon). i guess you're referring to the plugins eh ? this was proposed some time ago by eradicator but i dont know where that development track left off (my guess is nowhere significant) -mike pgpbSBwac2PGR.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 07:59, Brian Harring wrote: Why have the explicit var? Because 0.9.7a through 0.9.7c may all be compatible, but 0.9.7d isn't. If you force an automatic algo that tries to (effectively) guess, you get a lot of rebuilds through a,b,c, end result being folks likely update less because it becomes a bigger pain in the ass. if it isnt compatible then it shouldnt have the same SONAME, simple as that ... that is after all the point of encoding the ABI version number into the SONAME forcing devs to maintain a manual var which is basically duplicating the SONAME smells like maintenance nightmare -mike pgpotWD8NW3lp.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, Sep 21, 2006 at 09:52:27AM -0400, Mike Frysinger wrote: On Thursday 21 September 2006 07:59, Brian Harring wrote: Why have the explicit var? Because 0.9.7a through 0.9.7c may all be compatible, but 0.9.7d isn't. If you force an automatic algo that tries to (effectively) guess, you get a lot of rebuilds through a,b,c, end result being folks likely update less because it becomes a bigger pain in the ass. if it isnt compatible then it shouldnt have the same SONAME, simple as that ... that is after all the point of encoding the ABI version number into the SONAME forcing devs to maintain a manual var which is basically duplicating the SONAME smells like maintenance nightmare I agree; while I'm labeling it ABI, includes both bad soname handling and seperate sonames. Re: forcing devs... the request was to fold revdep-rebuild into resolution; in other words, 3 options 1) situation gets ignored, stays as is 2) all packages are somehow fixed (ultra restrictive deps) to never require revdep-rebuild 3) revdep-rebulid capabilities get inline into resolution. Feel free to point out a 4th option if I'm missing it, but for the request, that's what exists afaict; meanwhile, stating that pkgs are being stupid, while true, doesn't actually solve the issue :) ~harring pgpj0tlMIO7jl.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Mike Frysinger wrote: On Thursday 21 September 2006 07:59, Brian Harring wrote: Why have the explicit var? Because 0.9.7a through 0.9.7c may all be compatible, but 0.9.7d isn't. If you force an automatic algo that tries to (effectively) guess, you get a lot of rebuilds through a,b,c, end result being folks likely update less because it becomes a bigger pain in the ass. if it isnt compatible then it shouldnt have the same SONAME, simple as that ... that is after all the point of encoding the ABI version number into the SONAME forcing devs to maintain a manual var which is basically duplicating the SONAME smells like maintenance nightmare Not adding it into the ebuild means that it's impossible to show in advance what packages will actually be installed, because you don't know whether the sover will bump. Thanks, Donnie -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
Brian Harring wrote:
On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote:
There is one flaw with this though; packages can provide both
libraries _and_ binaries. Our dependencies don't represent whether
the dep is actual linkage, or just commandline consuming, so (using
the openssl example) any package that invokes openssl via the
commandline to do a few simple chksum ops gets rebuilt, despite the
fact it wasn't affected by linkage change ups.
I like BINCOMPAT proposal but it solves only half of the problem. You
assume that all dependent packages cares about binary compatibility.
Why not using a BDEPEND var in those dependent packages affected by the
BINCOMPAT values of their dependencies?
For instance, I would set the following:
- in net-dialup/ppp ebuild: BINCOMPAT=${PV}
- in net-dialup/pptpd ebuild: BDEPEND=net-dialup/ppp
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 10:14, Donnie Berkholz wrote: Not adding it into the ebuild means that it's impossible to show in advance what packages will actually be installed, because you don't know whether the sover will bump. sometimes you dont know as the ABI bump may be arch or feature specific ... pathological case i agree ;) you would certainly know between the time you merge the new one and unmerge the old one though ... but now we tread into the territory of portage should not unmerge ABI libraries until all consumers have been destroyed which is better than the current craptastic situation of you gotta revdep-rebuild after the fact with a broken system -mike pgpAf6pcdhThy.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 10:04, Brian Harring wrote: I agree; while I'm labeling it ABI, includes both bad soname handling and seperate sonames. those people should be smacked (for the interest of disclosure, i have violated the bad soname rule for the sake of following upstream) Feel free to point out a 4th option if I'm missing it, but for the request, that's what exists afaict; meanwhile, stating that pkgs are being stupid, while true, doesn't actually solve the issue :) 4) portage maintains a list of ABI SONAMEs in use and does not unmerge the library until all consumers are gone i'm referring to the specific file of course, not anything else in the package ... so integrating the hack eutils.eclass:preserve_old_lib() into portage so it isnt a hack (not a knock against the current implementation here; it's always going to be a hack until portage manages proper unmerging of the ABI library) -mike pgpkGngJWrBQP.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 10:38, Alin Nastac wrote:
Brian Harring wrote:
There is one flaw with this though; packages can provide both
libraries _and_ binaries. Our dependencies don't represent whether
the dep is actual linkage, or just commandline consuming, so (using
the openssl example) any package that invokes openssl via the
commandline to do a few simple chksum ops gets rebuilt, despite the
fact it wasn't affected by linkage change ups.
I like BINCOMPAT proposal but it solves only half of the problem. You
assume that all dependent packages cares about binary compatibility.
Why not using a BDEPEND var in those dependent packages affected by the
BINCOMPAT values of their dependencies?
For instance, I would set the following:
- in net-dialup/ppp ebuild: BINCOMPAT=${PV}
- in net-dialup/pptpd ebuild: BDEPEND=net-dialup/ppp
i think you're merging the two issues you brought up ... there is binary ABI
issues (which should not require a new DEPEND variable as portage can extract
that information out at runtime) and there is runtime plugin issues with
packages using dlopen() (which portage cannot extract as the dependency
cannot ever be extracted)
or did i not read your original e-mail incorrectly ?
-mike
pgpzT9ZX7vRoe.pgp
Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Mike Frysinger wrote: On Thursday 21 September 2006 10:14, Donnie Berkholz wrote: Not adding it into the ebuild means that it's impossible to show in advance what packages will actually be installed, because you don't know whether the sover will bump. sometimes you dont know as the ABI bump may be arch or feature specific ... pathological case i agree ;) you would certainly know between the time you merge the new one and unmerge the old one though ... but now we tread into the territory of portage should not unmerge ABI libraries until all consumers have been destroyed which is better than the current craptastic situation of you gotta revdep-rebuild after the fact with a broken system Yes, I agree with you. For example, take expat. The maintainers have refused to allow both versions to exist simultaneously on a system because it apparently causes more breakage than just breaking every app on your system by removing .so.0. Thanks, Donnie -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, 2006-09-21 at 09:52 -0400, Mike Frysinger wrote:
On Thursday 21 September 2006 07:59, Brian Harring wrote:
Why have the explicit var? Because 0.9.7a through 0.9.7c may all be
compatible, but 0.9.7d isn't. If you force an automatic algo that
tries to (effectively) guess, you get a lot of rebuilds through a,b,c,
end result being folks likely update less because it becomes a bigger
pain in the ass.
if it isnt compatible then it shouldnt have the same SONAME, simple as
that ... that is after all the point of encoding the ABI version number into
the SONAME
forcing devs to maintain a manual var which is basically duplicating the
SONAME smells like maintenance nightmare
There are various kinds of ABI. Certainly for C libs the SONAME is
probably the most common. If we go for some kind of ABI reverse deps
feature I would beg for something a little more general - though
certainly with SONAME as a common case.
Other languages have similar problems. For example Python has
incompatible ABIs for each major release 2.2, 2.3 etc. They have a
similar solution to the revdep-rebuild: python-updater.
As lead of the Haskell team my interest in this is that our primary
Haskell compiler GHC has ABI incompatibility between versions too. We've
made a ghc-updater similar in style to the python one but this is
clearly an unsatisfactory situation. It's more acute for us as even
minor revisions are ABI-incompatible.
So for it's something like:
# for C:
ABI=${SONAME}
# for python
ABI=${PY_PV}
# for haskell:
ABI=${GHC_PV}
paludis has something going in this direction but I don't think it'd
quite solve the python/ghc abi issue. It was aimed more at cases like
mips with it's multiple abis.
If we do go in this direction it'd be great to be able to slot on the
ABI and still have dependencies resolved correctly. For example imagine
having parallel python-2.3 and 2.4 installations with some libs
installed for both. Crucially, deps need to be resolved to the version
of a lib with the right ABI.
debian solves this problem in an ad-hoc way by tacking extra components
into the package name:
pyfoo-py22.deb which deps on pybar-py22.deb
pyfoo-py23.deb which deps on pybar-py23.deb
so all can be installed at once and deps are resolved within the right
ABI. Now this is obviously not in the Gentoo tradition. We much prefer
cleaner solutions like SLOTing. I would love to see this extended to
allow us to SLOT on the abi and then correctly resolve based on that
abi.
If we just SLOTed at the moment we'd get the sitation where
dev-python/bar built for py 2.2 would be considered ok to satisfy a
dependency of dev-python/foo that is being built for py 2.3.
We want some kind of extra component to be able to resolve on:
dev-python/foo-1.0.ebuild:
SLOT=${PV}-${PY_PV}
ABI=${PY_PV}
DEPEND=dev-python/bar @ ${PY_PV}
Actually for Haskell the situation is even more fun; we have multiple
haskell implementations, so we would like to install a lib and SLOT upon
and correctly resolve deps for multiple haskell compilers.
Fun stuff. :-)
If portage people are interested in moving in this direction we have an
experimental emerge-compatible mini dep-resolver which might be useful
to prototype an extended ABI/SLOTing system.
Duncan Coutts
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 10:54, Donnie Berkholz wrote: Yes, I agree with you. For example, take expat. The maintainers have refused to allow both versions to exist simultaneously on a system because it apparently causes more breakage than just breaking every app on your system by removing .so.0. that is the exact case portage should be handling for you it would go oh hey, check out libexpat.so.0 ... some things seem to want it ... HEY USER, you need to rebuild: ... once all the packages still consuming libexpat.so.0 are rebuilt, portage could silently trim it from the system complicated ? not really, scanelf can produce all this information in an easily digestable format -mike pgpHM7hO3h7Iv.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, Sep 21, 2006 at 05:38:08PM +0300, Alin Nastac wrote:
Brian Harring wrote:
On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote:
There is one flaw with this though; packages can provide both
libraries _and_ binaries. Our dependencies don't represent whether
the dep is actual linkage, or just commandline consuming, so (using
the openssl example) any package that invokes openssl via the
commandline to do a few simple chksum ops gets rebuilt, despite the
fact it wasn't affected by linkage change ups.
I like BINCOMPAT proposal but it solves only half of the problem. You
assume that all dependent packages cares about binary compatibility.
Why not using a BDEPEND var in those dependent packages affected by the
BINCOMPAT values of their dependencies?
For instance, I would set the following:
- in net-dialup/ppp ebuild: BINCOMPAT=${PV}
- in net-dialup/pptpd ebuild: BDEPEND=net-dialup/ppp
BDEPEND was actually a seperate proposal/idea, intention there was to
have that be the deps that *must* be CHOST (gcc would be an example);
bits that are used to actually build the pkg, not data it consumes in
building (headers would be data).
Meanwhile, for this I don't see the point in using a seperate metadata
key. Overload DEPEND and add a marker char that is used to indicate
that a particular dependency is 'binding', ie, it is linkage.
~harring
pgpTdTMlvzIuz.pgp
Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, Sep 21, 2006 at 10:43:11AM -0400, Mike Frysinger wrote: On Thursday 21 September 2006 10:04, Brian Harring wrote: I agree; while I'm labeling it ABI, includes both bad soname handling and seperate sonames. those people should be smacked (for the interest of disclosure, i have violated the bad soname rule for the sake of following upstream) Feel free to point out a 4th option if I'm missing it, but for the request, that's what exists afaict; meanwhile, stating that pkgs are being stupid, while true, doesn't actually solve the issue :) 4) portage maintains a list of ABI SONAMEs in use and does not unmerge the library until all consumers are gone i'm referring to the specific file of course, not anything else in the package ... so integrating the hack eutils.eclass:preserve_old_lib() into portage so it isnt a hack (not a knock against the current implementation here; it's always going to be a hack until portage manages proper unmerging of the ABI library) The reason folks aren't talking about using NEEDED is that NEEDED data is generated _after_ building; getting the info into the resolver up front allows for a helluva lot more options, and makes stuff like ensuring you have all sources required downloaded *prior* actually simple to do, rather then inserting recalculating hacks into the resolver. Clarifying the 'recalculating', what you're suggesting is effectivelly unbounded resolution, re-calculating at each step. That route is *very* nasty since you can't gurantee up front the resolution will work, let alone ensuring the bugger doesn't go cyclic. ~harring pgpoVTJpTCkcq.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Duncan Coutts wrote:
So for it's something like:
# for C:
ABI=${SONAME}
# for python
ABI=${PY_PV}
# for haskell:
ABI=${GHC_PV}
paludis has something going in this direction but I don't think it'd
quite solve the python/ghc abi issue. It was aimed more at cases like
mips with it's multiple abis.
It's all about multilib and has (except for the unfortunate name)
nothing to do with this issue.
--
Kind Regards,
Simon Stelling
Gentoo/AMD64 developer
--
gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thursday 21 September 2006 10:56, Duncan Coutts wrote: If we do go in this direction it'd be great to be able to slot on the ABI and still have dependencies resolved correctly. For example imagine having parallel python-2.3 and 2.4 installations with some libs installed for both. Crucially, deps need to be resolved to the version of a lib with the right ABI. ugh, no ... we are not a binary distribution so we should not have to worry about ABI baggage we SLOT based upon API, not ABI -mike pgpPmPiAwyWo0.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, 2006-09-21 at 11:11 -0400, Mike Frysinger wrote: On Thursday 21 September 2006 10:56, Duncan Coutts wrote: If we do go in this direction it'd be great to be able to slot on the ABI and still have dependencies resolved correctly. For example imagine having parallel python-2.3 and 2.4 installations with some libs installed for both. Crucially, deps need to be resolved to the version of a lib with the right ABI. ugh, no ... we are not a binary distribution so we should not have to worry about ABI baggage So we can't ever install two versions of python or ghc at once? That seems a shame. we SLOT based upon API, not ABI Here's another example; I'm not sure if it passes the ABI/API test: We would like to support 3 Haskell implementations: * GHC which compiles to native code (ELF binaries static .a libs) * Hugs which is an interpreter so installation is .hs source files * YHC which compiles to portable bytecode A single Haskell library is likely to work with all three implementations. So that's API. Once installed however each implementation is very different. So that's incompatible ABI. This could be 'solved' by having dev-haskell/foo-ghc, dev-haskell/foo-hugs, dev-haskell/foo-yhc, but that's obviously not the Gentoo way (though it's pretty much what debian does). These multiple impls is pretty similar to multiple versions of the same compiler. So my point is, I don't think it can be simply dismissed as ABI nonsense that we don't have to deal with. Being able to SLOT on the compiler flavour (and possibly version) would allow us to do useful things that we cannot currently do. Duncan -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
Brian Harring wrote: BDEPEND was actually a seperate proposal/idea, intention there was to have that be the deps that *must* be CHOST (gcc would be an example); bits that are used to actually build the pkg, not data it consumes in building (headers would be data). Well, until now I didn't thought at the build compatibility. My concern was only the runtime compatibility. Meanwhile, for this I don't see the point in using a seperate metadata key. Overload DEPEND and add a marker char that is used to indicate that a particular dependency is 'binding', ie, it is linkage. Lets suppose we use as 'binding' dependency marker. What sense would DEPEND=net-dialup/ppp have in a context of an ebuild. It certainly don't specify the necessity of package rebuild whenever net-dialup/ppp version is changed. Unless you save the specific compatibility version of the net-dialup/ppp used by net-dialup/pptpd for building the package, I don't see how can it help me. Judging after /var/db/pkg content, I have no such information. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Mike Frysinger wrote: i think you're merging the two issues you brought up ... there is binary ABI issues (which should not require a new DEPEND variable as portage can extract that information out at runtime) and there is runtime plugin issues with packages using dlopen() (which portage cannot extract as the dependency cannot ever be extracted) Okay, maybe I hijacked BINCOMPAT purpose. As I said in a previous reply, my original message was about runtime compatibility, not compilation compatibility. I want to be able to save in a package metadata that it was build using some version (as in compatibility version, not necessarily PV) of another package and I want emerge to automatically rebuild my package whenever this version is changed. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] RFC about another *DEPEND variable
Duncan Coutts wrote: So my point is, I don't think it can be simply dismissed as ABI nonsense that we don't have to deal with. Being able to SLOT on the compiler flavour (and possibly version) would allow us to do useful things that we cannot currently do. what about making them build what you want depending on useflags? lu -- Luca Barbato Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, Sep 21, 2006 at 08:15:48PM +0300, Alin Nastac wrote: Brian Harring wrote: BDEPEND was actually a seperate proposal/idea, intention there was to have that be the deps that *must* be CHOST (gcc would be an example); bits that are used to actually build the pkg, not data it consumes in building (headers would be data). Well, until now I didn't thought at the build compatibility. My concern was only the runtime compatibility. Meanwhile, for this I don't see the point in using a seperate metadata key. Overload DEPEND and add a marker char that is used to indicate that a particular dependency is 'binding', ie, it is linkage. Lets suppose we use as 'binding' dependency marker. What sense would DEPEND=net-dialup/ppp have in a context of an ebuild. It certainly don't specify the necessity of package rebuild whenever net-dialup/ppp version is changed. Unless you save the specific compatibility version of the net-dialup/ppp used by net-dialup/pptpd for building the package, I don't see how can it help me. Judging after /var/db/pkg content, I have no such information. Any such implementation would require storing some extra data in the vdb For this, would just walk the *DEPEND collecting 'binding' dependencies, and storing their BINCOMPAT in a simple mapping. ~harring pgpOEVz6DdaZV.pgp Description: PGP signature
Re: [gentoo-dev] RFC about another *DEPEND variable
On Thu, 2006-09-21 at 20:27 +0200, Luca Barbato wrote: Duncan Coutts wrote: So my point is, I don't think it can be simply dismissed as ABI nonsense that we don't have to deal with. Being able to SLOT on the compiler flavour (and possibly version) would allow us to do useful things that we cannot currently do. what about making them build what you want depending on useflags? Aye, for the implementation flavours that's probably the way to go once we have use-deps. We'll have to hold off on multiple versions of the same compiler though. It might get a bit hairy though :-) DEPEND=ghc? ( dev-haskell/foo @ ghc ) hugs? ( dev-haskell/foo @ hugs ) yhc? ( dev-haskell/foo @ yhc ) jhc? ( dev-haskell/foo @ jhc ) (I've not looked up what the use-dep syntax is, I'm just guessing) Duncan -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC about another *DEPEND variable
Duncan Coutts wrote: On Thu, 2006-09-21 at 20:27 +0200, Luca Barbato wrote: Duncan Coutts wrote: So my point is, I don't think it can be simply dismissed as ABI nonsense that we don't have to deal with. Being able to SLOT on the compiler flavour (and possibly version) would allow us to do useful things that we cannot currently do. what about making them build what you want depending on useflags? Aye, for the implementation flavours that's probably the way to go once we have use-deps. We'll have to hold off on multiple versions of the same compiler though. It might get a bit hairy though :-) DEPEND=ghc? ( dev-haskell/foo @ ghc ) hugs? ( dev-haskell/foo @ hugs ) yhc? ( dev-haskell/foo @ yhc ) jhc? ( dev-haskell/foo @ jhc ) (I've not looked up what the use-dep syntax is, I'm just guessing) as now, you can handle that with an eclass that takes deps, checks those deps with certain use if the use has been enabled and dies if those criterions aren't meet (asking you to falling back to hc-updater to keep those applications in shape) pretty hacky and itchy but that's probably the best you can do w/out use-deps. (PS can you slot on use now?) lu -- Luca Barbato Gentoo/linux Gentoo/PPC http://dev.gentoo.org/~lu_zero -- gentoo-dev@gentoo.org mailing list
