Re: PIE and static libraries
* Jiri Jaburek: > On 09/12/16 13:42, Jakub Wilk wrote: >> * Bálint Réczey , 2016-09-12, 13:21: Reading up on the subject so far, I got the impression that most static libraries should be built with PIE, but not necessarily PIC (to allow building PIE(xecutable)s, but discourage creating shared libraries from those static ones.) >> >> How does it discourage creating shlibs? >> >> Also, is there a reliable way for a static checker (such as Lintian) to >> distinguish between -fPIC and -fPIE static libraries? > > There is (kinda), > https://sourceware.org/ml/libc-alpha/2015-03/msg00605.html Yes, but we have moved on, and PIE has received optimizations since then which reduce the cost on platforms with PC-relative loads and stores. These optimizations eliminate unneeded redirections, but also make the resulting objects incompatible with full PIC mode. Without PIC, ELF dynamic linking allocates global variables referenced from the main program as if defined in the main program, and patches all DSOs to reference those global variables. This mechanism ensures that the objects which go into the main executable do not need to determine, at compile time, whether a variable reference ends up resolving to a definition in a DSO or not.
Re: PIE and static libraries
On 09/12/16 13:42, Jakub Wilk wrote: > * Bálint Réczey , 2016-09-12, 13:21: >>> Reading up on the subject so far, I got the impression that most >>> static libraries should be built with PIE, but not necessarily PIC >>> (to allow building PIE(xecutable)s, but discourage creating shared >>> libraries from those static ones.) > > How does it discourage creating shlibs? > > Also, is there a reliable way for a static checker (such as Lintian) to > distinguish between -fPIC and -fPIE static libraries? There is (kinda), https://sourceware.org/ml/libc-alpha/2015-03/msg00605.html > >> I don't see why should not we encourage using static libs in shared >> libs and recommending PIC would also simplify the currently mandated >> build process: > > Meh, this complexity lies in upstream build system most of the time. >
Re: PIE and static libraries
On 09/12/2016 01:42 PM, Jakub Wilk wrote: > * Bálint Réczey , 2016-09-12, 13:21: >>> Reading up on the subject so far, I got the impression that most >>> static libraries should be built with PIE, but not necessarily PIC >>> (to allow building PIE(xecutable)s, but discourage creating shared >>> libraries from those static ones.) > > How does it discourage creating shlibs? AFAIUI it's impossible to build a PIC-enabled shared library from a PIE-enabled static one. That may be an intentional effect (from a packager's view). Granted, building a non-PIC shared library would still be possible from that static library, I guess, so this isn't bullet-proof. However, using a static library to build a shared one hopefully isn't a terribly frequent use case, anyways (in most cases you should better just use the original shared library). (This argument had nothing to do with shipping shared vs static libraries in Debian packages, nor the fact that we often have to compile things twice.) I guess my point mainly is that for static libraries, I currently see valid use cases for all of the three variants: no-nothing-enabled, PIE and PIC. Justification for the first case seems to vanish with time, while the policy currently mandates it (modulo discussion on this list). Regards Markus Wanner signature.asc Description: OpenPGP digital signature
Re: PIE and static libraries
* Bálint Réczey , 2016-09-12, 13:21: Reading up on the subject so far, I got the impression that most static libraries should be built with PIE, but not necessarily PIC (to allow building PIE(xecutable)s, but discourage creating shared libraries from those static ones.) How does it discourage creating shlibs? Also, is there a reliable way for a static checker (such as Lintian) to distinguish between -fPIC and -fPIE static libraries? I don't see why should not we encourage using static libs in shared libs and recommending PIC would also simplify the currently mandated build process: Meh, this complexity lies in upstream build system most of the time. -- Jakub Wilk
Re: PIE and static libraries
On Mon, Sep 12, 2016 at 01:21:55PM +0200, Bálint Réczey wrote: > There is a (still growing:-)) list in here which includes other PIE > releated issues: > https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=pie-bindnow-20160906&user=balint%40balintreczey.hu btw, I saw that in at least one of them I received you linked to a ubuntu patch using patches.ubuntu.com. I discourage you to do so, as patches.ubuntu.com urls are not stable, and only presents the diff of the current ubuntu merge against the base common version. Instead I suggest you go through launchpad and pick the diff of an upload introducing the PIE-related patch, which could be found in https://launchpad.net/ubuntu/+source// under a label "diff from to " in the "Available diffs" section. It might be harder to find a proper diff, and sometimes it's actually impossible to find something acceptable, but at least you'll get durable URLs. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Re: PIE and static libraries
Hi Markus, 2016-09-12 8:51 GMT+02:00 Markus Wanner : > On 09/12/2016 01:47 AM, Bálint Réczey wrote: >> I have opened a bug to encourage PIC for static libraries in Policy, too.: >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478 > > Thanks, cool. > > Is there any specific reason for not mentioning -fPIE in that request? > That seems like a good middle-ground for static libraries. > > Reading up on the subject so far, I got the impression that most static > libraries should be built with PIE, but not necessarily PIC (to allow > building PIE(xecutable)s, but discourage creating shared libraries from > those static ones.) I don't see why should not we encourage using static libs in shared libs and recommending PIC would also simplify the currently mandated build process: 10.2 Libraries ... (paragraph about shared libs requiring PIC) As to the static libraries, the common case is not to have relocatable code, since there is no benefit, unless in specific cases; therefore the static version must not be compiled with the -fPIC flag. Any exception to this rule should be discussed on the mailing list debian-devel@lists.debian.org, and the reasons for compiling with the -fPIC flag must be recorded in the file README.Debian. [86] In other words, if both a shared and a static library is being built, each source unit (*.c, for example, for C files) will need to be compiled twice, for the normal case. ... > > To be honest, I didn't really check any use-case other than > libsimgear-dev, which I'm concerned about. There is a (still growing:-)) list in here which includes other PIE releated issues: https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=pie-bindnow-20160906&user=balint%40balintreczey.hu Cheers, Balint
Re: PIE and static libraries
On 09/12/2016 01:47 AM, Bálint Réczey wrote: > I have opened a bug to encourage PIC for static libraries in Policy, too.: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478 Thanks, cool. Is there any specific reason for not mentioning -fPIE in that request? That seems like a good middle-ground for static libraries. Reading up on the subject so far, I got the impression that most static libraries should be built with PIE, but not necessarily PIC (to allow building PIE(xecutable)s, but discourage creating shared libraries from those static ones.) To be honest, I didn't really check any use-case other than libsimgear-dev, which I'm concerned about. Kind Regards Markus Wanner signature.asc Description: OpenPGP digital signature
Re: PIE and static libraries
Hi All, 2016-05-22 11:26 GMT+02:00 Christian Seiler : > On 05/22/2016 10:50 AM, Andrey Rahmatullin wrote: >> On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote: ... > >>> B. From a performance perspective, using non-PIC/PIE code is >>>faster, though not necessarily by much anymore. >> It was worth mentioning only for i386 anyway. > > Well, there's not only amd64 and i386 - and some other platforms > also show some differences here. But as I said: I would recommend > to use PIE/PIC anyway. I have opened a bug to encourage PIC for static libraries in Policy, too.: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478 Cheers, Balint
Re: PIE and static libraries
On 05/22/2016 07:31 PM, Guillem Jover wrote: > I've tried to condense this and the other message on the other thread > to extend the dpkg-buildflags(1) man page. Great, thanks! > Attached the patch I'm intending to apply. Let me know if you have > other suggestions, improvements, wording tweaks, etc. > +.B \-fPIC > +Can be linked into any program and shared library. > +The resulting program might be slightly slower and bigger in case > +\fBbindnow\fP is not in use. Well, that is only for the difference between -fPIC and -fPIE, and TBH, that's not really that relevant. dpkg-buildflags does discuss the impact of PIE later on (your patch quotes part of it), so I don't think you need to mention that here, also because the description is already condensed. So I would simply remove the sentence "The resulting... not in use.", otherwise I think your patch looks good. Regards, Christian signature.asc Description: OpenPGP digital signature
Re: PIE and static libraries
Hi! On Sun, 2016-05-22 at 10:41:56 +0200, Christian Seiler wrote: [… useful overview …] I've tried to condense this and the other message on the other thread to extend the dpkg-buildflags(1) man page. Attached the patch I'm intending to apply. Let me know if you have other suggestions, improvements, wording tweaks, etc. Thanks, Guillem From d3a6f97736bade62a8945f91c22593f22f3e2ae4 Mon Sep 17 00:00:00 2001 From: Guillem Jover Date: Sun, 22 May 2016 19:20:04 +0200 Subject: [PATCH] man: Document interaction between PIE and libraries Prompted-by: Christian Seiler --- man/dpkg-buildflags.1 | 30 ++ 1 file changed, 30 insertions(+) diff --git a/man/dpkg-buildflags.1 b/man/dpkg-buildflags.1 index ac2489e..47b77d8 100644 --- a/man/dpkg-buildflags.1 +++ b/man/dpkg-buildflags.1 @@ -362,6 +362,36 @@ locations to bounce off of during a memory corruption attack. This is not compatible with \fB\-fPIC\fP so care must be taken when building shared objects. +Static libraries can be used by programs or other shared libraries. +Depending on the flags used to compile all the objects within a static +library, these libraries will be usable by different sets of objects: + +.RS +.TP +none +Cannot be linked into a PIE program, nor a shared library. +.TP +.B \-fPIE +Can be linked into any program, but not a shared library. +.TP +.B \-fPIC +Can be linked into any program and shared library. +The resulting program might be slightly slower and bigger in case +\fBbindnow\fP is not in use. +.RE + +.IP +Unconditionally passing \fB\-fPIE\fP, \fB\-fpie\fP or \fB\-pie\fP to a +build-system using libtool is safe as these flags will get stripped when +building shared libraries. +Otherwise on projects that build both programs and shared libraries you +might need to make sure that when building the shared libraries \fB\-fPIC\fP +is always passed last (so that it overrides any previous \fB\-PIE\fP) to +compilation flags such as \fBCFLAGS\fP, and \fB\-shared\fP is passed last +(so that it overrides any previous \fB\-pie\fP) to linking flags such as +\fBLDFLAGS\fP. + +.IP Additionally, since PIE is implemented via a general register, some register starved architectures (but not including i386 anymore since optimizations implemented in gcc >= 5) can see performance losses of up to -- 2.8.1
Re: PIE and static libraries
On 2016-05-22 Christian Seiler wrote: [extensive explanation] > Therefore, I would recommend to use at least -fPIE for static > libraries, and possibly -fPIC if you think they might be used > in other dynamic libraries. > Hope that helps. Yes it does. Thanks for taking the time to explain this. cu Andreas
Re: PIE and static libraries
On 05/22/2016 10:50 AM, Andrey Rahmatullin wrote: > On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote: >>=> however, -fPIC code is again slightly slower and >> larger than -fPIE code. > Really? I thought the idea is the same in both modes. Ok, thinking about it again, if you use relro + bindnow, there will probably be no difference anymore. (But PIC uses PLT when bindnow is not used, whereas PIE never uses PLT.) >> So in the end in boils down to the following: >> >> A. From a hardening perspective, any code that is added to >>static libraries should be compiled with -fPIE if the static >>library will only ever be used in executables, and with >>-fPIC if it also might be used in shared libraries. >>(Although, to be honest, that use case is a bit rarer.) > This, of course, assumes all executables using that library will be > compiled with -fPIE. No. You can combine -fPIE code with non-PIE code into a non-PIE executable: a.c: const char *hello = "Hello World"; const char **p_a() { return &hello; } b.c: const char **p_a(); const char *p_b() { return *p_a(); } m.c: #include const char *p_b(); int main() { puts(p_b()); return 0; } gcc -Wall -fno-PIE -c -o a.o a.c gcc -Wall -fPIE -c -o b.o b.c gcc -Wall -fno-PIE -c -o m.o m.c gcc -o m m.o a.o b.o ./m Hello World >> B. From a performance perspective, using non-PIC/PIE code is >>faster, though not necessarily by much anymore. > It was worth mentioning only for i386 anyway. Well, there's not only amd64 and i386 - and some other platforms also show some differences here. But as I said: I would recommend to use PIE/PIC anyway. Regards, Christian signature.asc Description: OpenPGP digital signature
Re: PIE and static libraries
On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote: >=> however, -fPIC code is again slightly slower and > larger than -fPIE code. Really? I thought the idea is the same in both modes. > So in the end in boils down to the following: > > A. From a hardening perspective, any code that is added to >static libraries should be compiled with -fPIE if the static >library will only ever be used in executables, and with >-fPIC if it also might be used in shared libraries. >(Although, to be honest, that use case is a bit rarer.) This, of course, assumes all executables using that library will be compiled with -fPIE. > B. From a performance perspective, using non-PIC/PIE code is >faster, though not necessarily by much anymore. It was worth mentioning only for i386 anyway. -- WBR, wRAR signature.asc Description: PGP signature
Re: PIE and static libraries
On 05/22/2016 08:48 AM, Andreas Metzler wrote: > https://lintian.debian.org/tags/hardening-no-pie.html says "It is > unlikely to work when compiling static libraries or executables (gcc > -static)." For static libraries, it really depends on what you want to do with them. A static library is just an archive of .o files (similar to tar), nothing more, and linking against a static library is roughly the same as just adding more .o files to the linker line. You can also link a static library into a shared library - the code in the static library is then just copied into the shared library (but the code then must be compiled with -fPIC, as with all other code that is used in shared libraries). It's therefore in possible to generate a static library with code that was compiled with either -fPIC or -fPIE. The question is what you want to use the code for: - code compiled without -fPIC and without -fPIE can be used only in non-PIE executables => a static library containing such code may not be linked into a PIE executable and may not be linked into a shared library - code compiled with -fPIE can be used in any executable but not in shared libraries => if you link -fPIE code into a non-PIE executable, it will work. Note, however, that PIE code is slightly larger and slower than non-PIE code. (The difference is tiny on modern platforms, though, and IMHO worth it w.r.t. hardening.) - code compiled with -fPIC can be used in anything, from shared libraries to executables (PIE or not) => however, -fPIC code is again slightly slower and larger than -fPIE code. On the other hand, if you look at how much functionality of modern systems is in shared libs anyway, IMHO this doesn't matter as much. Also, gcc starting with version 5 has optimized performance of -fPIC even on older x86_32 quite a bit, see [1]. So starting with Stretch, I don't think the performance argument against using PIC code holds much water anymore, even on older platforms. Note, however, that you cannot currently create PIE executables (-pie LDFLAGS) that are fully statically linked (-static linker flag). But you _can_ create executables that use PIE, are dynamically linked binaries (i.e. uses libc.so), but which do link against some static libraries - as long as the code in those static libraries was compiled with either -fPIE or -fPIC. So in the end in boils down to the following: A. From a hardening perspective, any code that is added to static libraries should be compiled with -fPIE if the static library will only ever be used in executables, and with -fPIC if it also might be used in shared libraries. (Although, to be honest, that use case is a bit rarer.) B. From a performance perspective, using non-PIC/PIE code is faster, though not necessarily by much anymore. Personally, I think the benefits gained from using PIE/PIC in hardening trump these performance considerations in a general-purpose distribution such as Debian anytime - but you should be aware of them. Therefore, I would recommend to use at least -fPIE for static libraries, and possibly -fPIC if you think they might be used in other dynamic libraries. Hope that helps. Regards, Christian [1] https://software.intel.com/en-us/blogs/2014/12/26/new-optimizations-for-x86-in-upcoming-gcc-50-32bit-pic-mode signature.asc Description: OpenPGP digital signature
Re: PIE and static libraries
On Sun, May 22, 2016 at 12:23:59PM +0500, Andrey Rahmatullin wrote: > On Sun, May 22, 2016 at 08:48:19AM +0200, Andreas Metzler wrote: > > https://lintian.debian.org/tags/hardening-no-pie.html says "It is > > unlikely to work when compiling static libraries or executables (gcc > > -static)." > > > > However e.g. https://wiki.debian.org/Hardening does not mention this > > problem. Is the former incorrect or the latter incomplete? > dpkg-buildflags(1) mentions it. https://wiki.debian.org/Hardening doesn't > seem like a good place to repeat all things discussed in that manpage. > > Also, this is a third thread on d-devel@ about PIE and shared libraries > for the past week. Sorry, I was assuming that the problem mentioned is about not being able to compile such static libraries into both shared objects and executables. The part about static executables may be worth mentioning in dpkg-buildflags(1) too (though static executables are not a common thing in the archive). -- WBR, wRAR signature.asc Description: PGP signature
Re: PIE and static libraries
On Sun, May 22, 2016 at 08:48:19AM +0200, Andreas Metzler wrote: > https://lintian.debian.org/tags/hardening-no-pie.html says "It is > unlikely to work when compiling static libraries or executables (gcc > -static)." > > However e.g. https://wiki.debian.org/Hardening does not mention this > problem. Is the former incorrect or the latter incomplete? dpkg-buildflags(1) mentions it. https://wiki.debian.org/Hardening doesn't seem like a good place to repeat all things discussed in that manpage. Also, this is a third thread on d-devel@ about PIE and shared libraries for the past week. -- WBR, wRAR signature.asc Description: PGP signature