Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Sunday 04 of April 2010 17:33:17 Tiziano Müller wrote: Besides I can already imagine PMS-related discussion regarding make the PMs check for rdeps per default before unmerging things - thx but no thx. This is not related to PMS. Paludis for example does it already with the current EAPIs. So how does Paludis handle those issues? (Read my comments about emerge atomicy below) It is a problem which can _only_ be solved at the PM level. You have to tell the PM when API breakages happen. Either by slotting the lib or by something else. ^^^ This is important - as slotting is not a practical solution. What is it then? Using guesswork to determine whether or not a library should be removed may be a temporary solution. Remember: you're partially ignoring a users request since he wanted the package to be removed - and we all know how things like protect the user from himself end up. Unconditionally removing libraries (instead of preserving them) and making their reverse runtime dependencies reinstalled is unacceptable because emerge process involving multiple packages is not atomic. Simple as that. Is this what you suggest? Correct me if I'm wrong: 1. Users wants to uninstall or reinstall package, we let him do it provided reverse runtime dependencies are satisfied afterwards. Let's say he wants to upgrade expat. 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime reverse deps will still be satisfied when we upgrade. 3. Expat has been upgraded sucessfully, 4a. emerge discovers reverse runtime dependencies are broken and it starts to rebuild them, then it bails out due to error ld: libexpat.so.sth not found. Because step 3 cannot be rolled back (no atomicy) - game over. or 4b. emerge does not discover those and does nothing. python is broken so emerge cannot be used anymore. Game over -- regards MM
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote: Unconditionally removing libraries (instead of preserving them) and making their reverse runtime dependencies reinstalled is unacceptable because emerge process involving multiple packages is not atomic. Simple as that. Is this what you suggest? Correct me if I'm wrong: 1. Users wants to uninstall or reinstall package, we let him do it provided reverse runtime dependencies are satisfied afterwards. Let's say he wants to upgrade expat. 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime reverse deps will still be satisfied when we upgrade. 3. Expat has been upgraded sucessfully, 4a. emerge discovers reverse runtime dependencies are broken and it starts to rebuild them, then it bails out due to error ld: libexpat.so.sth not found. Because step 3 cannot be rolled back (no atomicy) - game over. or 4b. emerge does not discover those and does nothing. python is broken so emerge cannot be used anymore. Game over This is called 'nondeterministic resolution'- known issue also w/ proposals of that sort. Pretty much everytime someone proposes it as a solution, it gets smacked down by most folk since an emerge -p invocation that is a single pkg upgade shouldn't be able to go rebuild your entire world. The alternative is a slotted ABI var- basically a counter (although it doesn't have to be) w/in ebuilds themselves to indicate if they're carrying a new ABI from upstream for that slotting. For example, you've got EXPAT merged w/ ABI=2, version 2.0. version 2.0.1, for whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or 3, as said it's an arbitrary value). Via that, the resolver can see that a rebuild is necessary and plan a rebuild of all consumers (whether NEEDED based or revdep). Note preserve-lib would be rather useful here- specifically holding onto the intermediate lib while doing rebuilding. This however breaks down a bit when the ABI change is in reverse of normal versioning. Usually whenever I've floated this idea, it's gotten smacked down by devs due to the extra work it may entail- someone doing an experiment on this would be rather useful (someone with a tinderbox specifically). ~harring pgpzALSfinToO.pgp Description: PGP signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
Am Sonntag, den 04.04.2010, 23:44 -0700 schrieb Brian Harring: On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote: Unconditionally removing libraries (instead of preserving them) and making their reverse runtime dependencies reinstalled is unacceptable because emerge process involving multiple packages is not atomic. Simple as that. Is this what you suggest? Correct me if I'm wrong: 1. Users wants to uninstall or reinstall package, we let him do it provided reverse runtime dependencies are satisfied afterwards. Let's say he wants to upgrade expat. 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime reverse deps will still be satisfied when we upgrade. 3. Expat has been upgraded sucessfully, 4a. emerge discovers reverse runtime dependencies are broken and it starts to rebuild them, then it bails out due to error ld: libexpat.so.sth not found. Because step 3 cannot be rolled back (no atomicy) - game over. or 4b. emerge does not discover those and does nothing. python is broken so emerge cannot be used anymore. Game over This is called 'nondeterministic resolution'- known issue also w/ proposals of that sort. Pretty much everytime someone proposes it as a solution, it gets smacked down by most folk since an emerge -p invocation that is a single pkg upgade shouldn't be able to go rebuild your entire world. The alternative is a slotted ABI var- basically a counter (although it doesn't have to be) w/in ebuilds themselves to indicate if they're carrying a new ABI from upstream for that slotting. For example, you've got EXPAT merged w/ ABI=2, version 2.0. version 2.0.1, for whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or 3, as said it's an arbitrary value). Via that, the resolver can see that a rebuild is necessary and plan a rebuild of all consumers (whether NEEDED based or revdep). Note preserve-lib would be rather useful here- specifically holding onto the intermediate lib while doing rebuilding. No, it doesn't help since you may have the same problems some people try to solve in this thread. This however breaks down a bit when the ABI change is in reverse of normal versioning. How so? Such a var should just specify the ABI and the PM only has to check whether it changed from one PVR to the other. The how is completely irrelevant. -- Tiziano Müller Gentoo Linux Developer Areas of responsibility: Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor E-Mail : dev-z...@gentoo.org GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30 smime.p7s Description: S/MIME cryptographic signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
Am Montag, den 05.04.2010, 08:16 +0200 schrieb Maciej Mrozowski: On Sunday 04 of April 2010 17:33:17 Tiziano Müller wrote: Besides I can already imagine PMS-related discussion regarding make the PMs check for rdeps per default before unmerging things - thx but no thx. This is not related to PMS. Paludis for example does it already with the current EAPIs. So how does Paludis handle those issues? (Read my comments about emerge atomicy below) It is a problem which can _only_ be solved at the PM level. You have to tell the PM when API breakages happen. Either by slotting the lib or by something else. ^^^ This is important - as slotting is not a practical solution. What is it then? Using guesswork to determine whether or not a library should be removed may be a temporary solution. Remember: you're partially ignoring a users request since he wanted the package to be removed - and we all know how things like protect the user from himself end up. Unconditionally removing libraries (instead of preserving them) and making their reverse runtime dependencies reinstalled is unacceptable because emerge process involving multiple packages is not atomic. Simple as that. A side mark: preserving libs may work in a number of cases, but there are a lot of (possible) examples where the rdeps (and/or the preserved libs) are nevertheless broken or become useless (think of: ui-files, plugins, dlopen'ed libs, etc.). Please forget your atomicity. It's a really nice idea (and I'd like to see it too) but not possible now or in the near future. Is this what you suggest? Correct me if I'm wrong: 1. Users wants to uninstall or reinstall package, we let him do it provided reverse runtime dependencies are satisfied afterwards. Let's say he wants to upgrade expat. 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime reverse deps will still be satisfied when we upgrade. 3. Expat has been upgraded sucessfully, 4a. emerge discovers reverse runtime dependencies are broken and it starts to rebuild them, then it bails out due to error ld: libexpat.so.sth not found. Because step 3 cannot be rolled back (no atomicy) - game over. or In that case you most probably have a problem in your dep-tree (either unspecified deps or a dep-resolver bug) -- Tiziano Müller Gentoo Linux Developer Areas of responsibility: Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor E-Mail : dev-z...@gentoo.org GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30 smime.p7s Description: S/MIME cryptographic signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Mon, Apr 05, 2010 at 03:27:34PM +0200, Tiziano MMMller wrote: Via that, the resolver can see that a rebuild is necessary and plan a rebuild of all consumers (whether NEEDED based or revdep). Note preserve-lib would be rather useful here- specifically holding onto the intermediate lib while doing rebuilding. No, it doesn't help since you may have the same problems some people try to solve in this thread. Might I suggest in the future purning down what you're responding to, to just that blurb? At first read of your response I thought you were arguing against the ABI var itself (which didn't make a helluva lot of sense). Meanwhile, yes, using preserve lib there still has some issues- the reason I mentioned it possibly being held onto is that if we're talking about something like openssl, having your system be horked for the intervening period isn't a great thing thus it may be useful as an option at the least. This however breaks down a bit when the ABI change is in reverse of normal versioning. How so? Such a var should just specify the ABI and the PM only has to check whether it changed from one PVR to the other. The how is completely irrelevant. That comment is in reference to if preserve-lib is still enabled for the rebuild. ~harring pgpYBJABgPHnz.pgp Description: PGP signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
Am Samstag, den 03.04.2010, 23:05 +0200 schrieb Maciej Mrozowski: On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote: Shouldn't we fix that buildsystem then? Do you have an example of a package/buildsystem that does that? We already do, the thing is that maybe we don't have to. https://bugs.gentoo.org/show_bug.cgi?id=240323 From top of my head: python having issues with sys-libs/db as well as some packages with readline. It would indeed. Now when I think about it, moving stuff to preserved library dir could be just done - provided it's possible - along with fixing/setting DT_RPATH's in reverse runtime dependencies. This way no system-wide LIBRARY_PATH's would be necessary. Is it possible? Mike? No, unless you somehow make sure you reserve space for this, by e.g. setting a bogus rpath entry at buildtime. If you want to go that route, you probably want to look at the Prefix' binutils-config wrapper that already calls the linker with added rpath arguments. Afterwards you can use chrpath to set it to the correct location. Will get messy with the vdb though, but if Portage's doing it, it can probably be dealt with. Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they allow such DT_RPATH operations? It should be probably also restricted for binary-only packages. On Saturday 03 of April 2010 20:51:43 Tiziano Müller wrote: Don't fix the hack. Remove the preserve libs feature, make the PMs check for rdeps per default before unmerging things. This will only prevent creating orphans of uninstalled libraries, what about upgraded ones when SOVERSION has been bumped (the most common case)? I addressed this in the next phrase. Besides I can already imagine PMS-related discussion regarding make the PMs check for rdeps per default before unmerging things - thx but no thx. This is not related to PMS. Paludis for example does it already with the current EAPIs. Slot libraries where needed, slot dep operators (EAPI 4) will help. Again, you suggest to SLOT every library that happened to bump SOVERSION. It's unrealistic. Besides library should be slotted when it's API changes, for source based distributions it's not needed for ABI changes - let's not confuse those two. Also excessive slotting increases probability of breaking library discovery mechanisms in various build systems (not everyone uses pkg-config). I know that slotting can cause problems. But forcing build systems to use specific versions of libraries is much easier than shadowing libraries present in library search dirs, especially when those libs are not even tracked by the PM. And if that doesn't work out we need a separate var to give the PM a hint when API/ABI breakages happen (such that the PM knows when to re-install the rev deps). It needs PMS amended and thus GLEP. Wrong, we don't have GLEPs for 90% of the PMS changes going into EAPI-3 or 4. Please submit a GLEP item for this if you see it fit. Anyway, as explained in OT, it's not a problem of package manager dependencies system but issue with broken/not smart build systems - no dependency tree magic will solve this issue. It is a problem which can _only_ be solved at the PM level. You have to tell the PM when API breakages happen. Either by slotting the lib or by something else. Using guesswork to determine whether or not a library should be removed may be a temporary solution. Remember: you're partially ignoring a users request since he wanted the package to be removed - and we all know how things like protect the user from himself end up. -- Tiziano Müller Gentoo Linux Developer Areas of responsibility: Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor E-Mail : dev-z...@gentoo.org GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30 smime.p7s Description: S/MIME cryptographic signature
[gentoo-dev] [RFC] More reliable hiding preserved libraries
Problem
..is known, let me summarize briefly.
Uninstalling packages providing libraries, without checking reverse runtime
dependencies of those packages leaves their dependencies unsatisfied (packages
with broken executables and/or shared libs).
Some package managers try their best not to remove said libraries, yet
allowing packages to be removed.
Those orphaned libraries cause problems[1] as build systems of some other
packages being (re)installed afterwards pick them up and abuse those orphaned
libraries. (we don't like orphans abused, we prefer them... gone).
Solution
Now, I suppose there are some ideas how to make orphaned libraries not go in a
way. Basically then need to be available for system, but hidden for emerge.
There is opt-out suggestion[2], unfortunately it does not provide any info how
exactly it's supposed to be achieved. As far as portage/pkgcore is concerned,
maybe - as Brian Harring suggested - sandbox could be used to somehow hide
preserved libraries or preserved library directory from ebuild environment
(preserved library directory a'ka purgatory - libs could be moved there when
considered orphaned).
Opt-in suggestion is as follows:
1. Use some library path (read by ld loader from environment) and export it
globally to environment pointing it to preserved library directory.
2. During emerge, unset environment variable corresponding to said preserved
library directory - orphans are no longer located.
Attached patch for glibc (2.11, but should apply to any other glibc around)
and uClibc (this one is not tested but should work as well) that makes ld
loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable.
(LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be
safely mangled.. on the second though it can - one could filter out preserved
library paths from it during emerge).
Thoughts?
1. https://bugs.gentoo.org/show_bug.cgi?id=240323
2. https://bugs.gentoo.org/show_bug.cgi?id=307391
--
regards
MM
diff -ru ../glibc-2.11/elf/rtld.c ./elf/rtld.c
--- ../glibc-2.11/elf/rtld.c 2009-10-30 18:17:08.0 +0100
+++ ./elf/rtld.c 2010-04-03 10:51:46.468906521 +0200
@@ -874,6 +874,8 @@
/* The library search path. */
static const char *library_path attribute_relro;
+/* Gentoo preserved library path. */
+static const char *preserved_library_path attribute_relro;
/* The list preloaded objects. */
static const char *preloadlist attribute_relro;
/* Nonzero if information about versions has to be printed. */
@@ -1385,7 +1387,18 @@
/* Initialize the data structures for the search paths for shared
objects. */
- _dl_init_paths (library_path);
+ char *llp = alloca ((library_path != NULL ? strlen (library_path) : 0) +
+(preserved_library_path != NULL ? strlen (preserved_library_path) : 0) + 2);
+ *llp = '\0';
+ if (library_path != NULL)
+{
+ strcat (llp, library_path);
+ if (preserved_library_path != NULL)
+strcat (llp, :);
+}
+ if (preserved_library_path != NULL)
+strcat (llp, preserved_library_path);
+ _dl_init_paths (llp);
/* Initialize _r_debug. */
struct r_debug *r = _dl_debug_initialize (GL(dl_rtld_map).l_addr,
@@ -2648,6 +2661,16 @@
mode = trace;
break;
+ case 29:
+ /* Read Gentoo preserved library path from env here. Watch out for
+ possible future 'case' additions in this switch as well as in
+ EXTRA_LD_ENVVARS defined in sysdeps/arch/dl-librecon.h.
+ LD_GENTOO_PRESERVED_LIBRARY_PATH is blacklisted for SUID
+ programs in sysdeps/generic/unsecvars.h. */
+ if (memcmp (envline, GENTOO_PRESERVED_LIBRARY_PATH, 29) == 0)
+ preserved_library_path = envline[30];
+ break;
+
/* We might have some extra environment variable to handle. This
is tricky due to the pre-processing of the length of the name
in the switch statement here. The code here assumes that added
diff -ru ../glibc-2.11/sysdeps/generic/unsecvars.h ./sysdeps/generic/unsecvars.h
--- ../glibc-2.11/sysdeps/generic/unsecvars.h 2009-10-30 18:17:08.0 +0100
+++ ./sysdeps/generic/unsecvars.h 2010-04-03 10:48:07.192280437 +0200
@@ -10,6 +10,7 @@
LD_DEBUG_OUTPUT\0 \
LD_DYNAMIC_WEAK\0 \
LD_LIBRARY_PATH\0 \
+ LD_GENTOO_PRESERVED_LIBRARY_PATH\0 \
LD_ORIGIN_PATH\0 \
LD_PRELOAD\0 \
LD_PROFILE\0 \
diff -ru ../uClibc-0.9.30.1/ldso/include/dl-hash.h ./ldso/include/dl-hash.h
--- ../uClibc-0.9.30.1/ldso/include/dl-hash.h 2008-11-03 16:41:17.0 +0100
+++ ./ldso/include/dl-hash.h 2010-04-03 11:23:44.026291507 +0200
@@ -132,6 +132,7 @@
extern int _dl_linux_dynamic_link(void);
extern char * _dl_library_path;
+extern char * preserved_library_path;
extern char * _dl_not_lazy;
static __inline__ int _dl_symbol(char * name)
diff -ru ../uClibc-0.9.30.1/ldso/include/ldso.h ./ldso/include/ldso.h
--- ../uClibc-0.9.30.1/ldso/include/ldso.h 2008-05-30 16:35:31.0 +0200
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Sat, Apr 03, 2010 at 12:38:17PM +0200, Maciej Mrozowski wrote: exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, maybe - as Brian Harring suggested - sandbox could be used to somehow hide preserved libraries or preserved library directory from ebuild environment (preserved library directory a'ka purgatory - libs could be moved there when considered orphaned). When I've tried experimenting w/ this I've had zero luck in intercepting lib loadup on that one. ~harring pgpoGunNZ7rNI.pgp Description: PGP signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On 03-04-2010 12:38:17 +0200, Maciej Mrozowski wrote: Problem ..is known, let me summarize briefly. Uninstalling packages providing libraries, without checking reverse runtime dependencies of those packages leaves their dependencies unsatisfied (packages with broken executables and/or shared libs). Some package managers try their best not to remove said libraries, yet allowing packages to be removed. Those orphaned libraries cause problems[1] as build systems of some other packages being (re)installed afterwards pick them up and abuse those orphaned libraries. (we don't like orphans abused, we prefer them... gone). Is it known why this does happen exactly? When a lib is kept because it is still used, only its soname + what the soname points to should be kept. That would mean the lib can no longer be found during linking, unless you add some trickery (or does GNU ld do something handy out of the box right here?). So for example: % ls usr/lib/libfoo.so - libfoo.so.2.1 usr/lib/libfoo.so.2 - libfoo.so.2.1 usr/lib/libfoo.so.2.1 % scanelf --soname usr/lib/libfoo.so libfoo.so.2 usr/lib/libfoo.so what should kept preserved is: usr/lib/libfoo.so.2 - libfoo.so.2.1 usr/lib/libfoo.so.2.1 because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should (in theory and on some platforms at least) fail. Solution Now, I suppose there are some ideas how to make orphaned libraries not go in a way. Basically then need to be available for system, but hidden for emerge. There is opt-out suggestion[2], unfortunately it does not provide any info how exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, maybe - as Brian Harring suggested - sandbox could be used to somehow hide preserved libraries or preserved library directory from ebuild environment (preserved library directory a'ka purgatory - libs could be moved there when considered orphaned). Opt-in suggestion is as follows: 1. Use some library path (read by ld loader from environment) and export it globally to environment pointing it to preserved library directory. you mean: move preserved libs to some special place and have that place being found at runtime somehow? 2. During emerge, unset environment variable corresponding to said preserved library directory - orphans are no longer located. Attached patch for glibc (2.11, but should apply to any other glibc around) and uClibc (this one is not tested but should work as well) that makes ld loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable. (LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be safely mangled.. on the second though it can - one could filter out preserved library paths from it during emerge). In general, LD_LIBRARY_PATH is considered harmful, and I wouldn't like to see it being used for normal operation. Instead I'd like to know first why applications can find retained libs, because from the Portage side, in theory they shouldn't. Maybe patching GNU ld if it turns out being too smart may solve problems in a nicer way. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Sat, 3 Apr 2010 12:38:17 +0200 Maciej Mrozowski reave...@gmail.com wrote: 2. During emerge, unset environment variable corresponding to said preserved library directory - orphans are no longer located. Wouldn't that cause failure when the toolkit relies on a 'hidden' preserved library? -- Best regards, Michał Górny http://mgorny.alt.pl xmpp:mgo...@jabber.ru signature.asc Description: PGP signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
Le samedi 03 avril 2010 à 12:38 +0200, Maciej Mrozowski a écrit : There is opt-out suggestion[2], unfortunately it does not provide any info how exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, maybe - as Brian Harring suggested - sandbox could be used to somehow hide preserved libraries or preserved library directory from ebuild environment (preserved library directory a'ka purgatory - libs could be moved there when considered orphaned). that sounds nice, it would allow us to more easily spot packages/upstreams doing it wrong (maybe that would work for packages linking to themselves too btw) -- Gilles Dartiguelongue e...@gentoo.org Gentoo signature.asc Description: Ceci est une partie de message numériquement signée
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Saturday 03 of April 2010 12:56:04 Fabian Groffen wrote: Is it known why this does happen exactly? When a lib is kept because it is still used, only its soname + what the soname points to should be kept. That would mean the lib can no longer be found during linking, unless you add some trickery (or does GNU ld do something handy out of the box right here?). So for example: % ls usr/lib/libfoo.so - libfoo.so.2.1 usr/lib/libfoo.so.2 - libfoo.so.2.1 usr/lib/libfoo.so.2.1 % scanelf --soname usr/lib/libfoo.so libfoo.so.2 usr/lib/libfoo.so what should kept preserved is: usr/lib/libfoo.so.2 - libfoo.so.2.1 usr/lib/libfoo.so.2.1 because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should (in theory and on some platforms at least) fail. It doesn't matter, as 'broken' build system may alphabetically find library by file name, and link to this library by full path. On Saturday 03 of April 2010 13:13:00 Michał Górny wrote: 2. During emerge, unset environment variable corresponding to said preserved library directory - orphans are no longer located. Wouldn't that cause failure when the toolkit relies on a 'hidden' preserved library? It would indeed. Now when I think about it, moving stuff to preserved library dir could be just done - provided it's possible - along with fixing/setting DT_RPATH's in reverse runtime dependencies. This way no system-wide LIBRARY_PATH's would be necessary. Is it possible? Mike? On Saturday 03 of April 2010 13:33:16 Gilles Dartiguelongue wrote: There is opt-out suggestion[2], unfortunately it does not provide any info how exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, maybe - as Brian Harring suggested - sandbox could be used to somehow hide preserved libraries or preserved library directory from ebuild environment (preserved library directory a'ka purgatory - libs could be moved there when considered orphaned). that sounds nice, it would allow us to more easily spot packages/upstreams doing it wrong (maybe that would work for packages linking to themselves too btw) Keeping preserved libraries in separate location (in purgatory or dumping place) is just a method for making implementation possibly easier (or possible at all), with nice side effects though. -- regards MM
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On 03-04-2010 14:09:42 +0200, Maciej Mrozowski wrote: because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should (in theory and on some platforms at least) fail. It doesn't matter, as 'broken' build system may alphabetically find library by file name, and link to this library by full path. Shouldn't we fix that buildsystem then? Do you have an example of a package/buildsystem that does that? On Saturday 03 of April 2010 13:13:00 Michał Górny wrote: 2. During emerge, unset environment variable corresponding to said preserved library directory - orphans are no longer located. Wouldn't that cause failure when the toolkit relies on a 'hidden' preserved library? It would indeed. Now when I think about it, moving stuff to preserved library dir could be just done - provided it's possible - along with fixing/setting DT_RPATH's in reverse runtime dependencies. This way no system-wide LIBRARY_PATH's would be necessary. Is it possible? Mike? No, unless you somehow make sure you reserve space for this, by e.g. setting a bogus rpath entry at buildtime. If you want to go that route, you probably want to look at the Prefix' binutils-config wrapper that already calls the linker with added rpath arguments. Afterwards you can use chrpath to set it to the correct location. Will get messy with the vdb though, but if Portage's doing it, it can probably be dealt with. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
Am Samstag, den 03.04.2010, 12:38 +0200 schrieb Maciej Mrozowski: Problem ..is known, let me summarize briefly. Uninstalling packages providing libraries, without checking reverse runtime dependencies of those packages leaves their dependencies unsatisfied (packages with broken executables and/or shared libs). Some package managers try their best not to remove said libraries, yet allowing packages to be removed. Those orphaned libraries cause problems[1] as build systems of some other packages being (re)installed afterwards pick them up and abuse those orphaned libraries. (we don't like orphans abused, we prefer them... gone). Solution Now, I suppose there are some ideas how to make orphaned libraries not go in a way. Basically then need to be available for system, but hidden for emerge. There is opt-out suggestion[2], unfortunately it does not provide any info how exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, maybe - as Brian Harring suggested - sandbox could be used to somehow hide preserved libraries or preserved library directory from ebuild environment (preserved library directory a'ka purgatory - libs could be moved there when considered orphaned). Opt-in suggestion is as follows: 1. Use some library path (read by ld loader from environment) and export it globally to environment pointing it to preserved library directory. 2. During emerge, unset environment variable corresponding to said preserved library directory - orphans are no longer located. Attached patch for glibc (2.11, but should apply to any other glibc around) and uClibc (this one is not tested but should work as well) that makes ld loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable. (LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be safely mangled.. on the second though it can - one could filter out preserved library paths from it during emerge). Thoughts? Don't fix the hack. Remove the preserve libs feature, make the PMs check for rdeps per default before unmerging things. Slot libraries where needed, slot dep operators (EAPI 4) will help. And if that doesn't work out we need a separate var to give the PM a hint when API/ABI breakages happen (such that the PM knows when to re-install the rev deps). -- Tiziano Müller Gentoo Linux Developer Areas of responsibility: Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor E-Mail : dev-z...@gentoo.org GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30 smime.p7s Description: S/MIME cryptographic signature
Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries
On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote: Shouldn't we fix that buildsystem then? Do you have an example of a package/buildsystem that does that? We already do, the thing is that maybe we don't have to. https://bugs.gentoo.org/show_bug.cgi?id=240323 From top of my head: python having issues with sys-libs/db as well as some packages with readline. It would indeed. Now when I think about it, moving stuff to preserved library dir could be just done - provided it's possible - along with fixing/setting DT_RPATH's in reverse runtime dependencies. This way no system-wide LIBRARY_PATH's would be necessary. Is it possible? Mike? No, unless you somehow make sure you reserve space for this, by e.g. setting a bogus rpath entry at buildtime. If you want to go that route, you probably want to look at the Prefix' binutils-config wrapper that already calls the linker with added rpath arguments. Afterwards you can use chrpath to set it to the correct location. Will get messy with the vdb though, but if Portage's doing it, it can probably be dealt with. Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they allow such DT_RPATH operations? It should be probably also restricted for binary-only packages. On Saturday 03 of April 2010 20:51:43 Tiziano Müller wrote: Don't fix the hack. Remove the preserve libs feature, make the PMs check for rdeps per default before unmerging things. This will only prevent creating orphans of uninstalled libraries, what about upgraded ones when SOVERSION has been bumped (the most common case)? Besides I can already imagine PMS-related discussion regarding make the PMs check for rdeps per default before unmerging things - thx but no thx. Slot libraries where needed, slot dep operators (EAPI 4) will help. Again, you suggest to SLOT every library that happened to bump SOVERSION. It's unrealistic. Besides library should be slotted when it's API changes, for source based distributions it's not needed for ABI changes - let's not confuse those two. Also excessive slotting increases probability of breaking library discovery mechanisms in various build systems (not everyone uses pkg-config). And if that doesn't work out we need a separate var to give the PM a hint when API/ABI breakages happen (such that the PM knows when to re-install the rev deps). It needs PMS amended and thus GLEP. Please submit a GLEP item for this if you see it fit. Anyway, as explained in OT, it's not a problem of package manager dependencies system but issue with broken/not smart build systems - no dependency tree magic will solve this issue. -- regards MM
