Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 08:51:42AM +0100, Kevin F. Quinn wrote: On Wed, 14 Dec 2005 07:59:23 +0100 Harald van Dijk [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 03:50:16AM +, Mike Frysinger wrote: my gnu stack docs are actually complete: http://hardened.gentoo.org/gnu-stack.xml A question about that: you discourage fixing this with --noexecstack because it's better to be able to submit a patch upstream. What's your take on patches that modify configure scripts or similar files to check for this flag, keeping it out of the ebuild? Is that good, acceptable, or bad, and why? Using '--noexecstack' overrides anything the compiler works out for itself, so applying it indiscriminately is a bad idea. For example, if an application contains asm code with no markings, but also contains code that creates trampolines, it should be marked for executable stack even if the asm code is fixed. Applying '--noexecstack' via LDFLAGS would break such an application. Regarding patches, it's usually much simpler to patch asm source code compared to patching an application's make process. Patching asm source code just means appending a few lines depending on the type of assembler used. As far as ebuilds are concerned, if you add it to LDFLAGS you will need to re-check the application every time you bump the ebuild, and it's difficult to find new occurrences of nested functions for example if you've applied '--noexecstack'. LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, and would need rechecking of the assembly code on updates just as much as patches which add .note.GNU-stack would, right? pgpNUhmpkS0CL.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 08:44:33AM +0100, Kevin F. Quinn wrote: The seriousness of the textrel issue is different for Hardened Gentoo and normal Gentoo. For Hardened Gentoo they cause real problems and must to be fixed to avoid compromising the overall strategy. For non-hardened Gentoo it's not so serious. I'll focus on non-hardened Gentoo here. [snip] Thank you for this splendid explaination of the issue. Regards, Brix -- Henrik Brix Andersen [EMAIL PROTECTED] Gentoo Metadistribution | Mobile computing herd pgpe9tTQZFu8d.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 09:19:56AM +0100, Harald van D??k wrote: LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, correct would need rechecking of the assembly code on updates just as much as patches which add .note.GNU-stack would, right? no you were supposed to send that patch upstream, i guess you didnt huh bad dev -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 01:43:28PM +, Mike Frysinger wrote: On Wed, Dec 14, 2005 at 09:19:56AM +0100, Harald van D??k wrote: LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, correct would need rechecking of the assembly code on updates just as much as patches which add .note.GNU-stack would, right? no you were supposed to send that patch upstream, i guess you didnt huh In my first message I explained the reason for asking about these kinds of patches: because your reason against them -- that they can't be sent upstream -- didn't apply. So I guess this situation would be that I did send a patch upstream, but it simply wasn't accepted yet. This can happen with any patch, even those of your preferred kind. pgpY8ZFmI5Uod.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 02:38:00PM +, Mike Frysinger wrote: there's a few problems with trying to get configure to detect whether the host assembler supports the --noexecstack option: - it's very easy to get the detection wrong and i'd bet money that anyone doing it for the first time would screw it up - it'll only fix the package if the host binutils are new enough ... this is becoming less of a problem every day, but it's still not uncommon - patching the code to use .note.GNU-stack would work regardless of the binutils version Okay, thanks for the info. pgpsoylpbNWLP.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, 2005-12-14 at 00:25 +, Mike Frysinger wrote: On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote: Mark Loeser wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm looking for. Issues like textrels, executable stacks, etc is what I'm looking for to be defined and explained why we are to always avoid them. This should be added to existing documentation policy so it is somewhere for new devs to know about, and existing devs to have for a reference. Only problem I see with this is binary packages. We can not control upstream binaries as everyone is aware of. So when does it become safe to override stable packages that have texrel's and executable stacks? no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. Actually, it is not fixed yet. Ryan is aware of the issue, but it was not fixed in the latest version. There was an error on my part that made me think it was fixed. -- Chris Gianelloni Release Engineering - Strategic Lead x86 Architecture Team Games - Developer Gentoo Linux signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Textrels in packages policy
On Wed, 2005-12-14 at 00:25 +, Mike Frysinger wrote: On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote: Mark Loeser wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm looking for. Issues like textrels, executable stacks, etc is what I'm looking for to be defined and explained why we are to always avoid them. This should be added to existing documentation policy so it is somewhere for new devs to know about, and existing devs to have for a reference. Only problem I see with this is binary packages. We can not control upstream binaries as everyone is aware of. So when does it become safe to override stable packages that have texrel's and executable stacks? no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. Let me take that back again. This is resolved on amd64, but not on x86. There's 2 binaries. This also explains why I didn't see the error and closed the bug, I'm on amd64, whereas the bug reporter is on x86. -- Chris Gianelloni Release Engineering - Strategic Lead x86 Architecture Team Games - Developer Gentoo Linux signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 10:27:19AM -0500, Chris Gianelloni wrote: On Wed, 2005-12-14 at 00:25 +, Mike Frysinger wrote: On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote: Only problem I see with this is binary packages. We can not control upstream binaries as everyone is aware of. So when does it become safe to override stable packages that have texrel's and executable stacks? no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. Let me take that back again. This is resolved on amd64, but not on x86. There's 2 binaries. This also explains why I didn't see the error and closed the bug, I'm on amd64, whereas the bug reporter is on x86. yeah, Ryan's one cool cat -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 14 Dec 2005 09:19:56 +0100 Harald van Dijk [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 08:51:42AM +0100, Kevin F. Quinn wrote: On Wed, 14 Dec 2005 07:59:23 +0100 Harald van Dijk [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 03:50:16AM +, Mike Frysinger wrote: my gnu stack docs are actually complete: http://hardened.gentoo.org/gnu-stack.xml A question about that: you discourage fixing this with --noexecstack because it's better to be able to submit a patch upstream. What's your take on patches that modify configure scripts or similar files to check for this flag, keeping it out of the ebuild? Is that good, acceptable, or bad, and why? Using '--noexecstack' overrides anything the compiler works out for itself, so applying it indiscriminately is a bad idea. For example, if an application contains asm code with no markings, but also contains code that creates trampolines, it should be marked for executable stack even if the asm code is fixed. Applying '--noexecstack' via LDFLAGS would break such an application. Regarding patches, it's usually much simpler to patch asm source code compared to patching an application's make process. Patching asm source code just means appending a few lines depending on the type of assembler used. As far as ebuilds are concerned, if you add it to LDFLAGS you will need to re-check the application every time you bump the ebuild, and it's difficult to find new occurrences of nested functions for example if you've applied '--noexecstack'. LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, and would need rechecking of the assembly code on updates just as much as patches which add .note.GNU-stack would, right? You're right there. I was thinking of '-Wl,-z,[no]execstack' which can be used on LDFLAGS, but overrides the setting for the whole application. - -- Kevin F. Quinn -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDoRfM9G2S8dekcG0RAoiRAKDcjEaXjLU4AmC+1NLM8zzOZ7DoDQCeJILV oncYVeaOrMf77XZyRwWCBUA= =ua9o -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
On Tue, 13 Dec 2005, Mark Loeser wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. Given my limited knowledge on this, this is my understanding. TEXTRELS are basically text relocations. What this is, is relocation within the text segment of the process image. This brings up the question of what a relocation is. A relocation is simply the replacement of some text with a memory location. The big issue with this is that the text segment is usually suppose to be read only for security reasons. But because the text segment needs a relocation, it needs to be read-write since the relocation happens at runtime dynamically. The constant need to look up the address is what causes the performance degredation. The performance degredation however is of no worry to us. The issue is that since the text segment is now read-write, the image of the process is no longer guaranteed to remain the same as it can be overwritten (allowing code modifications at runtime which can happen other ways as well). Because of this, the application is far more vurnerable to arbitrary code execution as if an exploit manages to overwrite the text segment properly, it can execute code that it wants. I am not sure how correct this explanation is or it is even what you were looking for. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm looking for. Issues like textrels, executable stacks, etc is what I'm looking for to be defined and explained why we are to always avoid them. This should be added to existing documentation policy so it is somewhere for new devs to know about, and existing devs to have for a reference. I agree, this would be very nice to have. It would make stabilization of packages a little bit easier. Thanks. Saleem Abdulrasool compnerd (at) gentoo (dot) org pgp64bkVZkTlP.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark Loeser wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm looking for. Issues like textrels, executable stacks, etc is what I'm looking for to be defined and explained why we are to always avoid them. This should be added to existing documentation policy so it is somewhere for new devs to know about, and existing devs to have for a reference. Thanks Only problem I see with this is binary packages. We can not control upstream binaries as everyone is aware of. So when does it become safe to override stable packages that have texrel's and executable stacks? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFDn2BVGDfjNg8unQIRAupGAKCiTPJseSVrklDjWXqwEdeHFDxnRQCcD0xQ mzjn2yXHiNSdBcnFkCTD+u0= =RYEw -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote: Mark Loeser wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm looking for. Issues like textrels, executable stacks, etc is what I'm looking for to be defined and explained why we are to always avoid them. This should be added to existing documentation policy so it is somewhere for new devs to know about, and existing devs to have for a reference. Only problem I see with this is binary packages. We can not control upstream binaries as everyone is aware of. So when does it become safe to override stable packages that have texrel's and executable stacks? no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
Mike Frysinger [EMAIL PROTECTED] said: this is correct, a very good reason to fix TEXTRELs. another good reason is that since the segment cannot be mapped readonly, the memory cannot be shared across multiple processes ... each will need to have its own copy, thus wasting what could be significant memory resources. Sounds like something nice to do, but not something that should be _required_ for a package to be deemed stable. The constant need to look up the address is what causes the performance degredation. The performance degredation however is of no worry to us. if by constant you mean once everytime the code is loaded, then you are correct ... the relocations need to be rewritten everytime the image is loaded into memory by the dynamic loader (ldso), but after the first fixup, that's it, nothing else to be done. Again, nice to fix, but does not seem necessary. working on it as i said ... i wish this e-mail could have been posted once i had more easier things to read :p You are working on a policy, or just docs to explain the issues? From what was listed above, I'm not sure why we should require that people fix these issues just to have a package deemed stable. -- Mark Loeser - Gentoo Developer (cpp gcc-porting toolchain x86) email - halcy0n AT gentoo DOT org mark AT halcy0n DOT com web - http://dev.gentoo.org/~halcy0n/ http://www.halcy0n.com pgpqnl9LWsDn7.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Tue, 13 Dec 2005 20:02:27 -0500 Mark Loeser [EMAIL PROTECTED] wrote: | You are working on a policy, or just docs to explain the issues? | From what was listed above, I'm not sure why we should require that | people fix these issues just to have a package deemed stable. Some people want no TEXTRELs to be a requirement because they're superstitious, or because they read somewhere that TEXTRELs are naughty and are repeating the conclusion without understanding the underlying reasons. It's approximately on par with trying to ban all code that uses 'goto' because it breaks certain security auditing tools that rely upon call graphs. -- Ciaran McCreesh : Gentoo Developer (I can kill you with my brain) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Tue, Dec 13, 2005 at 08:02:27PM -0500, Mark Loeser wrote: Mike Frysinger [EMAIL PROTECTED] said: working on it as i said ... i wish this e-mail could have been posted once i had more easier things to read :p You are working on a policy, or just docs to explain the issues? documentation on PIC/TEXTRELs/etc... the policy i consider a no-brainer, fix TEXTRELs -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 01:07:53AM +, Ciaran McCreesh wrote: On Wed, 14 Dec 2005 00:22:36 + Mike Frysinger [EMAIL PROTECTED] | another good reason is that since the segment cannot be mapped | readonly, the memory cannot be shared across multiple processes ... | each will need to have its own copy, thus wasting what could be | significant memory resources. Again, that's a big could be. it's more often an is be considering the fact we're talking about shared library code here. We don't avoid marking stable code that, say, mallocs lots of space, then fills it with some calculated numbers (for example, the first million prime numbers), even though a better program would allow for that data to be shared. no one said that broken code with TEXTRELs cannot be marked stable they're something to be fixed down the road as time permits Oh, and don't accept reasons like but they don't work if we enable $obscure_voodoo in the compiler either. If $obscure_voodoo breaks on legitimate TEXTRELs then $obscure_voodoo is broken, not the code using TEXTRELs. majority of the time, if a build process is generating poor code with textrels, it wont work on most architectures. x86 just tends to be pretty lenient when it comes to poor code, so no one notices/cares. -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
Mike Frysinger [EMAIL PROTECTED] said: We don't avoid marking stable code that, say, mallocs lots of space, then fills it with some calculated numbers (for example, the first million prime numbers), even though a better program would allow for that data to be shared. no one said that broken code with TEXTRELs cannot be marked stable they're something to be fixed down the road as time permits This I completely agree with, but was not what was being pushed as policy for the x86 arch team. If a package reintroduces textrels or such, I don't see the problem with marking it stable personally. The issues listed don't seem to warrant that. -- Mark Loeser - Gentoo Developer (cpp gcc-porting toolchain x86) email - halcy0n AT gentoo DOT org mark AT halcy0n DOT com web - http://dev.gentoo.org/~halcy0n/ http://www.halcy0n.com pgpofdVS6jpvI.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, 14 Dec 2005 01:20:29 + Mike Frysinger [EMAIL PROTECTED] wrote: | the policy i consider a no-brainer, fix TEXTRELs So... Say libfoo is a library that decodes files in the foo format. Say also that libfoo-2.1 is currently marked stable, and does not contain any TEXTRELs, but only supports foo files using the foo-1 format, which is rapidly becoming obsolete. Then, if libfoo-2.2 comes along, and introduces support for the rapidly becoming very widely used foo-2 format, should libfoo-2.2 be held back from being marked stable purely because of a TEXTREL? What about if, instead, libfoo-2.1 contains a security bug that is fixed by libfoo-2.2? What about if, instead, libfoo-2.1 has a nasty bug that will cause it to lose data under certain circumstances? See, any policy on TEXTRELs will have to be *very* carefully worded to avoid having people think that they're something important... Otherwise we'll end up with broken packages being left in stable simply because newer versions contain TEXTRELs. Don't overhype the problem -- make it clear that it's a would be nice issue, or else you'll encourage even more superstition than we have already... -- Ciaran McCreesh : Gentoo Developer (I can kill you with my brain) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm signature.asc Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
Mike Frysinger [EMAIL PROTECTED] said: On Tue, Dec 13, 2005 at 08:02:27PM -0500, Mark Loeser wrote: You are working on a policy, or just docs to explain the issues? documentation on PIC/TEXTRELs/etc... the policy i consider a no-brainer, fix TEXTRELs By policy, I mean things to add to the ebuild policy stating, These conditions must be met in order to consider a package stable:. I'll assume that is not what you are working on since nothing you have listed so far seems to imply this. -- Mark Loeser - Gentoo Developer (cpp gcc-porting toolchain x86) email - halcy0n AT gentoo DOT org mark AT halcy0n DOT com web - http://dev.gentoo.org/~halcy0n/ http://www.halcy0n.com pgpRexJ34OhCQ.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Wed, 14 Dec 2005 00:25:57 + Mike Frysinger [EMAIL PROTECTED] wrote: no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. Not to redirect the thread, but can someone point me to stuff on executable stacks (what they are and the background info on the warnings in portage)? If these are anywhere near critical, then I think a lot of us non-x86 arch monkeys have a lot of work on our hands. Cheers, -- Jason Wever Gentoo/Sparc Team Co-Lead signature.asc Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
Jason Wever [EMAIL PROTECTED] said: Not to redirect the thread, but can someone point me to stuff on executable stacks (what they are and the background info on the warnings in portage)? Not really redirecting the thread since this was another thing I wanted to find out about :) Basically, any of the things that stricter may complain about, which are critical, in the sense that they can not be marked stable under any circumstances? Also, what exactly do they mean? :) If these are anywhere near critical, then I think a lot of us non-x86 arch monkeys have a lot of work on our hands. Yea, we need to know what is absolutely critical so we can add this to the ebuild policy if something should not go stable if the problem is present. -- Mark Loeser - Gentoo Developer (cpp gcc-porting toolchain x86) email - halcy0n AT gentoo DOT org mark AT halcy0n DOT com web - http://dev.gentoo.org/~halcy0n/ http://www.halcy0n.com pgpn3AVUNG4Qp.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
On Tue, Dec 13, 2005 at 07:59:02PM -0700, Jason Wever wrote: On Wed, 14 Dec 2005 00:25:57 + Mike Frysinger [EMAIL PROTECTED] wrote: no idea what you mean by override, but here's a crazy idea ... ask upstream to fix the issues. for example, we just reported executable stacks with the ut2004 game and Ryan of epicgames was so kind as to fix it up for us. some upstream peeps dont even know about these sort of things until you point them out. Not to redirect the thread, but can someone point me to stuff on executable stacks (what they are and the background info on the warnings in portage)? my gnu stack docs are actually complete: http://hardened.gentoo.org/gnu-stack.xml If these are anywhere near critical, then I think a lot of us non-x86 arch monkeys have a lot of work on our hands. gnu-stack tends to be broken for everyone, not just x86 or !x86 -mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Textrels in packages policy
On Wed, Dec 14, 2005 at 03:50:16AM +, Mike Frysinger wrote: my gnu stack docs are actually complete: http://hardened.gentoo.org/gnu-stack.xml A question about that: you discourage fixing this with --noexecstack because it's better to be able to submit a patch upstream. What's your take on patches that modify configure scripts or similar files to check for this flag, keeping it out of the ebuild? Is that good, acceptable, or bad, and why? pgpdHOunZ5i9m.pgp Description: PGP signature
Re: [gentoo-dev] Textrels in packages policy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 13 Dec 2005 15:59:03 -0500 Mark Loeser [EMAIL PROTECTED] wrote: Basically what I'm looking for here is an easy to understand explanation of what textrels are, why they are bad, and why they should hold back marking a package stable. The only information I've been able to find states that they could cause a performance hit, but this doesn't seem to warrant banning them completely in my eyes. The seriousness of the textrel issue is different for Hardened Gentoo and normal Gentoo. For Hardened Gentoo they cause real problems and must to be fixed to avoid compromising the overall strategy. For non-hardened Gentoo it's not so serious. I'll focus on non-hardened Gentoo here. Textrels (short for text relocations) are function or data pointers within the code of a shared library that need to be adjusted to take account of the address at which a shared library is loaded. Shared libraries can be loaded at varying addresses depending on how the loader brings shared libraries into a process' memory image. This can be affected by factors including the size of the main executable, which other shared libraries are loaded for the same process and in which order. When a shared library is loaded, any absolute address references within the code have to be changed to reflect the actual address according to the address at which the shared library is loaded, before that code is executed. Since the loader can't tell what will be executed and when, it has to resolve all such references before the process can be permitted to start execution (this is the performance overhead incurred at load time). It also means that the image of the shared library is specific to that process - other processes can only use the same image if they load the library at the same address which is unlikely under normal circumstances. Thus libraries that are truly shared between applications, consume separate memory for each application that is using them (which is therefore a resource overhead). One workaround for this is to use prelink. This examines your whole system, and marks all shared libraries with hints for their load addresses, which the loader follows if it can. The addresses are chosen to minimise the amount of collisions, If a shared library has no text relocations, it is properly Position Independent Code (PIC). Instead of absolute addresses, the code always uses relative addresses which work no matter at which address the code is loaded. Typically this is achieved via an indirection table (the Global Offset Table) which contains the offsets for each symbol. Thus such code can be mapped without write permissions on the code, and you only ever have one copy of the code in memory, no matter how many applications are using that code. The downside of position-independent code is that it does have a performance impact as currently implemented (i.e. it generates slower code) due to the indirection introduced. This is made worse by the bad use of shared libraries, and the fact that the visibility stuff in the GNU tools is so recent (shockingly) and so libraries have far too much stuff unnecessarily in their global offset tables. Shared libraries (.so's) are used for various things: 1) To share code between multiple applications, the original intention of shared libraries. Best example is libc which if it were to be duplicated for each running application would have a significant impact on memory usage. For such libraries, TEXTRELs are a problem; the seriousness of the problem depends on how many applications use them. 2) To modularise an application; good examples here are X and web-browser plugins. Here, code is loaded if it is used, but not otherwise. Loading all the device drivers into X would be wasteful, for example. This is a secondary use of shared libraries that arises from the way shared libraries are implemented, making them easy to use for this purpose. TEXTRELs here cause a penalty on load, but 3) To break an application up into bits. This is the worst reason to create a shared library, as it gains nothing but loses much. It means a developer abstraction has leaked unnecessarily into the implementation. These simply shouldn't be shared libraries, instead the code should have been linked into the executable. Textrels can be caused by: 1) asm code that isn't PIC 2) incorrect makefiles etc (if a build uses libtool properly, it'll create shared libraries with no textrels) Frequently these are easy to fix, and are simply oversights upstream. There's no reason not to fix these. Occasionally they're a pain to fix, especially non-PIC asm, and if upstream are non-PIC on purpose (usually because they want to extract every last cycle from the processor) we end up having to maintain large patches which amongst other things is risky. Getting a clear cut policy on exactly what issues should hold a package back from being marked stable is what I'm
Re: [gentoo-dev] Textrels in packages policy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 14 Dec 2005 07:59:23 +0100 Harald van Dijk [EMAIL PROTECTED] wrote: On Wed, Dec 14, 2005 at 03:50:16AM +, Mike Frysinger wrote: my gnu stack docs are actually complete: http://hardened.gentoo.org/gnu-stack.xml A question about that: you discourage fixing this with --noexecstack because it's better to be able to submit a patch upstream. What's your take on patches that modify configure scripts or similar files to check for this flag, keeping it out of the ebuild? Is that good, acceptable, or bad, and why? Using '--noexecstack' overrides anything the compiler works out for itself, so applying it indiscriminately is a bad idea. For example, if an application contains asm code with no markings, but also contains code that creates trampolines, it should be marked for executable stack even if the asm code is fixed. Applying '--noexecstack' via LDFLAGS would break such an application. Regarding patches, it's usually much simpler to patch asm source code compared to patching an application's make process. Patching asm source code just means appending a few lines depending on the type of assembler used. As far as ebuilds are concerned, if you add it to LDFLAGS you will need to re-check the application every time you bump the ebuild, and it's difficult to find new occurrences of nested functions for example if you've applied '--noexecstack'. - -- Kevin F. Quinn -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDn88O9G2S8dekcG0RAsdDAJ9bhfqc44mtQgBsPu5OFjfNGG0GWQCg0eTA vU+j9b8nxMtodf5MSXgkfsE= =nVnR -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
