Re: PIE and static libraries

[Florian Weimer]

* Jiri Jaburek:

> On 09/12/16 13:42, Jakub Wilk wrote:
>> * Bálint Réczey , 2016-09-12, 13:21:
 Reading up on the subject so far, I got the impression that most
 static libraries should be built with PIE, but not necessarily PIC
 (to allow building PIE(xecutable)s, but discourage creating shared
 libraries from those static ones.)
>> 
>> How does it discourage creating shlibs?
>> 
>> Also, is there a reliable way for a static checker (such as Lintian) to
>> distinguish between -fPIC and -fPIE static libraries?
>
> There is (kinda),
> https://sourceware.org/ml/libc-alpha/2015-03/msg00605.html

Yes, but we have moved on, and PIE has received optimizations since
then which reduce the cost on platforms with PC-relative loads and
stores.  These optimizations eliminate unneeded redirections, but also
make the resulting objects incompatible with full PIC mode.

Without PIC, ELF dynamic linking allocates global variables referenced
from the main program as if defined in the main program, and patches
all DSOs to reference those global variables.  This mechanism ensures
that the objects which go into the main executable do not need to
determine, at compile time, whether a variable reference ends up
resolving to a definition in a DSO or not.

Re: PIE and static libraries

[Jiri Jaburek]

On 09/12/16 13:42, Jakub Wilk wrote:
> * Bálint Réczey , 2016-09-12, 13:21:
>>> Reading up on the subject so far, I got the impression that most
>>> static libraries should be built with PIE, but not necessarily PIC
>>> (to allow building PIE(xecutable)s, but discourage creating shared
>>> libraries from those static ones.)
> 
> How does it discourage creating shlibs?
> 
> Also, is there a reliable way for a static checker (such as Lintian) to
> distinguish between -fPIC and -fPIE static libraries?

There is (kinda),
https://sourceware.org/ml/libc-alpha/2015-03/msg00605.html

> 
>> I don't see why should not we encourage using static libs in shared
>> libs and recommending PIC would also simplify the currently mandated
>> build process:
> 
> Meh, this complexity lies in upstream build system most of the time.
> 

Re: PIE and static libraries

[Markus Wanner]

On 09/12/2016 01:42 PM, Jakub Wilk wrote:
> * Bálint Réczey , 2016-09-12, 13:21:
>>> Reading up on the subject so far, I got the impression that most
>>> static libraries should be built with PIE, but not necessarily PIC
>>> (to allow building PIE(xecutable)s, but discourage creating shared
>>> libraries from those static ones.)
> 
> How does it discourage creating shlibs?

AFAIUI it's impossible to build a PIC-enabled shared library from a
PIE-enabled static one. That may be an intentional effect (from a
packager's view). Granted, building a non-PIC shared library would still
be possible from that static library, I guess, so this isn't
bullet-proof. However, using a static library to build a shared one
hopefully isn't a terribly frequent use case, anyways (in most cases you
should better just use the original shared library).

(This argument had nothing to do with shipping shared vs static
libraries in Debian packages, nor the fact that we often have to compile
things twice.)

I guess my point mainly is that for static libraries, I currently see
valid use cases for all of the three variants: no-nothing-enabled, PIE
and PIC. Justification for the first case seems to vanish with time,
while the policy currently mandates it (modulo discussion on this list).

Regards

Markus Wanner




signature.asc
Description: OpenPGP digital signature

Re: PIE and static libraries

[Jakub Wilk]

* Bálint Réczey , 2016-09-12, 13:21:
Reading up on the subject so far, I got the impression that most 
static libraries should be built with PIE, but not necessarily PIC (to 
allow building PIE(xecutable)s, but discourage creating shared 
libraries from those static ones.)


How does it discourage creating shlibs?

Also, is there a reliable way for a static checker (such as Lintian) to 
distinguish between -fPIC and -fPIE static libraries?


I don't see why should not we encourage using static libs in shared 
libs and recommending PIC would also simplify the currently mandated 
build process:


Meh, this complexity lies in upstream build system most of the time.

--
Jakub Wilk

Re: PIE and static libraries

[Mattia Rizzolo]

On Mon, Sep 12, 2016 at 01:21:55PM +0200, Bálint Réczey wrote:
> There is a (still growing:-)) list in here which includes other PIE
> releated issues:
> https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=pie-bindnow-20160906=balint%40balintreczey.hu

btw, I saw that in at least one of them I received you linked to a
ubuntu patch using patches.ubuntu.com.  I discourage you to do so, as
patches.ubuntu.com urls are not stable, and only presents the diff of
the current ubuntu merge against the base common version.

Instead I suggest you go through launchpad and pick the diff of an
upload introducing the PIE-related patch, which could be found in
https://launchpad.net/ubuntu/+source//
under a label "diff from  to " in the "Available diffs"
section.

It might be harder to find a proper diff, and sometimes it's actually
impossible to find something acceptable, but at least you'll get durable
URLs.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: PIE and static libraries

[Bálint Réczey]

Hi Markus,

2016-09-12 8:51 GMT+02:00 Markus Wanner :
> On 09/12/2016 01:47 AM, Bálint Réczey wrote:
>> I have opened a bug to encourage PIC for static libraries in Policy, too.:
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478
>
> Thanks, cool.
>
> Is there any specific reason for not mentioning -fPIE in that request?
> That seems like a good middle-ground for static libraries.
>
> Reading up on the subject so far, I got the impression that most static
> libraries should be built with PIE, but not necessarily PIC (to allow
> building PIE(xecutable)s, but discourage creating shared libraries from
> those static ones.)

I don't see why should not we encourage using static libs in shared libs
and recommending PIC would also simplify the currently mandated build process:

10.2 Libraries
... (paragraph about shared libs requiring PIC)

As to the static libraries, the common case is not to have relocatable
code, since there is no benefit, unless in specific cases; therefore the
static version must not be compiled with the -fPIC flag. Any exception
to this rule should be discussed on the mailing list
debian-devel@lists.debian.org, and the reasons for compiling with the
-fPIC flag must be recorded in the file README.Debian. [86]

In other words, if both a shared and a static library is being built,
each source unit (*.c, for example, for C files) will need to be
compiled twice, for the normal case.
...

>
> To be honest, I didn't really check any use-case other than
> libsimgear-dev, which I'm concerned about.

There is a (still growing:-)) list in here which includes other PIE
releated issues:
https://udd.debian.org/cgi-bin/bts-usertags.cgi?tag=pie-bindnow-20160906=balint%40balintreczey.hu

Cheers,
Balint

Re: PIE and static libraries

[Markus Wanner]

On 09/12/2016 01:47 AM, Bálint Réczey wrote:
> I have opened a bug to encourage PIC for static libraries in Policy, too.:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478

Thanks, cool.

Is there any specific reason for not mentioning -fPIE in that request?
That seems like a good middle-ground for static libraries.

Reading up on the subject so far, I got the impression that most static
libraries should be built with PIE, but not necessarily PIC (to allow
building PIE(xecutable)s, but discourage creating shared libraries from
those static ones.)

To be honest, I didn't really check any use-case other than
libsimgear-dev, which I'm concerned about.

Kind Regards

Markus Wanner




signature.asc
Description: OpenPGP digital signature

Re: PIE and static libraries

[Bálint Réczey]

Hi All,

2016-05-22 11:26 GMT+02:00 Christian Seiler :
> On 05/22/2016 10:50 AM, Andrey Rahmatullin wrote:
>> On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote:
...
>
>>> B. From a performance perspective, using non-PIC/PIE code is
>>>faster, though not necessarily by much anymore.
>> It was worth mentioning only for i386 anyway.
>
> Well, there's not only amd64 and i386 - and some other platforms
> also show some differences here. But as I said: I would recommend
> to use PIE/PIC anyway.

I have opened a bug to encourage PIC for static libraries in Policy, too.:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837478

Cheers,
Balint

Re: PIE and static libraries

[Christian Seiler]

On 05/22/2016 07:31 PM, Guillem Jover wrote:
> I've tried to condense this and the other message on the other thread
> to extend the dpkg-buildflags(1) man page.

Great, thanks!

> Attached the patch I'm intending to apply. Let me know if you have
> other suggestions, improvements, wording tweaks, etc.

> +.B \-fPIC
> +Can be linked into any program and shared library.
> +The resulting program might be slightly slower and bigger in case
> +\fBbindnow\fP is not in use.

Well, that is only for the difference between -fPIC and -fPIE,
and TBH, that's not really that relevant. dpkg-buildflags does
discuss the impact of PIE later on (your patch quotes part of
it), so I don't think you need to mention that here, also
because the description is already condensed. So I would
simply remove the sentence "The resulting... not in use.",
otherwise I think your patch looks good.

Regards,
Christian



signature.asc
Description: OpenPGP digital signature

Re: PIE and static libraries

[Guillem Jover]

Hi!

On Sun, 2016-05-22 at 10:41:56 +0200, Christian Seiler wrote:
[… useful overview …]

I've tried to condense this and the other message on the other thread
to extend the dpkg-buildflags(1) man page. Attached the patch I'm
intending to apply. Let me know if you have other suggestions,
improvements, wording tweaks, etc.

Thanks,
Guillem
From d3a6f97736bade62a8945f91c22593f22f3e2ae4 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Sun, 22 May 2016 19:20:04 +0200
Subject: [PATCH] man: Document interaction between PIE and libraries

Prompted-by: Christian Seiler 
---
 man/dpkg-buildflags.1 | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/man/dpkg-buildflags.1 b/man/dpkg-buildflags.1
index ac2489e..47b77d8 100644
--- a/man/dpkg-buildflags.1
+++ b/man/dpkg-buildflags.1
@@ -362,6 +362,36 @@ locations to bounce off of during a memory corruption attack.
 This is not compatible with \fB\-fPIC\fP so care must be taken when
 building shared objects.
 
+Static libraries can be used by programs or other shared libraries.
+Depending on the flags used to compile all the objects within a static
+library, these libraries will be usable by different sets of objects:
+
+.RS
+.TP
+none
+Cannot be linked into a PIE program, nor a shared library.
+.TP
+.B \-fPIE
+Can be linked into any program, but not a shared library.
+.TP
+.B \-fPIC
+Can be linked into any program and shared library.
+The resulting program might be slightly slower and bigger in case
+\fBbindnow\fP is not in use.
+.RE
+
+.IP
+Unconditionally passing \fB\-fPIE\fP, \fB\-fpie\fP or \fB\-pie\fP to a
+build-system using libtool is safe as these flags will get stripped when
+building shared libraries.
+Otherwise on projects that build both programs and shared libraries you
+might need to make sure that when building the shared libraries \fB\-fPIC\fP
+is always passed last (so that it overrides any previous \fB\-PIE\fP) to
+compilation flags such as \fBCFLAGS\fP, and \fB\-shared\fP is passed last
+(so that it overrides any previous \fB\-pie\fP) to linking flags such as
+\fBLDFLAGS\fP.
+
+.IP
 Additionally, since PIE is implemented via a general register, some
 register starved architectures (but not including i386 anymore since
 optimizations implemented in gcc >= 5) can see performance losses of up to
-- 
2.8.1

Re: PIE and static libraries

[Andreas Metzler]

On 2016-05-22 Christian Seiler  wrote:
[extensive explanation]
> Therefore, I would recommend to use at least -fPIE for static
> libraries, and possibly -fPIC if you think they might be used
> in other dynamic libraries.

> Hope that helps.

Yes it does. Thanks for taking the time to explain this.

cu Andreas

Re: PIE and static libraries

[Christian Seiler]

On 05/22/2016 10:50 AM, Andrey Rahmatullin wrote:
> On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote:
>>=> however, -fPIC code is again slightly slower and
>>   larger than -fPIE code. 
> Really? I thought the idea is the same in both modes.

Ok, thinking about it again, if you use relro + bindnow, there will
probably be no difference anymore. (But PIC uses PLT when bindnow
is not used, whereas PIE never uses PLT.)

>> So in the end in boils down to the following:
>>
>> A. From a hardening perspective, any code that is added to
>>static libraries should be compiled with -fPIE if the static
>>library will only ever be used in executables, and with
>>-fPIC if it also might be used in shared libraries.
>>(Although, to be honest, that use case is a bit rarer.)
> This, of course, assumes all executables using that library will be
> compiled with -fPIE.

No. You can combine -fPIE code with non-PIE code into a non-PIE
executable:

a.c:
const char *hello = "Hello World";
const char **p_a() { return  }
b.c:
const char **p_a();
const char *p_b() { return *p_a(); }
m.c:
#include 
const char *p_b();
int main() { puts(p_b()); return 0; }

gcc -Wall -fno-PIE -c -o a.o a.c
gcc -Wall -fPIE -c -o b.o b.c
gcc -Wall -fno-PIE -c -o m.o m.c
gcc -o m m.o a.o b.o
./m
Hello World

>> B. From a performance perspective, using non-PIC/PIE code is
>>faster, though not necessarily by much anymore.
> It was worth mentioning only for i386 anyway.

Well, there's not only amd64 and i386 - and some other platforms
also show some differences here. But as I said: I would recommend
to use PIE/PIC anyway.

Regards,
Christian



signature.asc
Description: OpenPGP digital signature

Re: PIE and static libraries

[Andrey Rahmatullin]

On Sun, May 22, 2016 at 10:41:56AM +0200, Christian Seiler wrote:
>=> however, -fPIC code is again slightly slower and
>   larger than -fPIE code. 
Really? I thought the idea is the same in both modes.

> So in the end in boils down to the following:
> 
> A. From a hardening perspective, any code that is added to
>static libraries should be compiled with -fPIE if the static
>library will only ever be used in executables, and with
>-fPIC if it also might be used in shared libraries.
>(Although, to be honest, that use case is a bit rarer.)
This, of course, assumes all executables using that library will be
compiled with -fPIE.

> B. From a performance perspective, using non-PIC/PIE code is
>faster, though not necessarily by much anymore.
It was worth mentioning only for i386 anyway.


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: PIE and static libraries

[Christian Seiler]

On 05/22/2016 08:48 AM, Andreas Metzler wrote:
> https://lintian.debian.org/tags/hardening-no-pie.html says "It is
> unlikely to work when compiling static libraries or executables (gcc
> -static)."

For static libraries, it really depends on what you want to do with
them. A static library is just an archive of .o files (similar to
tar), nothing more, and linking against a static library is roughly
the same as just adding more .o files to the linker line. You can
also link a static library into a shared library - the code in the
static library is then just copied into the shared library (but the
code then must be compiled with -fPIC, as with all other code that
is used in shared libraries).

It's therefore in possible to generate a static library with code
that was compiled with either -fPIC or -fPIE.

The question is what you want to use the code for:

 - code compiled without -fPIC and without -fPIE can be used only
   in non-PIE executables

   => a static library containing such code may not be linked
  into a PIE executable and may not be linked into a
  shared library

 - code compiled with -fPIE can be used in any executable but not
   in shared libraries

   => if you link -fPIE code into a non-PIE executable, it
  will work. Note, however, that PIE code is slightly
  larger and slower than non-PIE code. (The difference is
  tiny on modern platforms, though, and IMHO worth it
  w.r.t. hardening.)

 - code compiled with -fPIC can be used in anything, from shared
   libraries to executables (PIE or not)

   => however, -fPIC code is again slightly slower and
  larger than -fPIE code. On the other hand, if you look
  at how much functionality of modern systems is in
  shared libs anyway, IMHO this doesn't matter as much.
  Also, gcc starting with version 5 has optimized
  performance of -fPIC even on older x86_32 quite a bit,
  see [1]. So starting with Stretch, I don't think the
  performance argument against using PIC code holds much
  water anymore, even on older platforms.

Note, however, that you cannot currently create PIE executables
(-pie LDFLAGS) that are fully statically linked (-static linker
flag). But you _can_ create executables that use PIE, are
dynamically linked binaries (i.e. uses libc.so), but which do
link against some static libraries - as long as the code in
those static libraries was compiled with either -fPIE or -fPIC.

So in the end in boils down to the following:

A. From a hardening perspective, any code that is added to
   static libraries should be compiled with -fPIE if the static
   library will only ever be used in executables, and with
   -fPIC if it also might be used in shared libraries.
   (Although, to be honest, that use case is a bit rarer.)

B. From a performance perspective, using non-PIC/PIE code is
   faster, though not necessarily by much anymore. Personally,
   I think the benefits gained from using PIE/PIC in hardening
   trump these performance considerations in a general-purpose
   distribution such as Debian anytime - but you should be
   aware of them.

Therefore, I would recommend to use at least -fPIE for static
libraries, and possibly -fPIC if you think they might be used
in other dynamic libraries.

Hope that helps.

Regards,
Christian

[1] 
https://software.intel.com/en-us/blogs/2014/12/26/new-optimizations-for-x86-in-upcoming-gcc-50-32bit-pic-mode



signature.asc
Description: OpenPGP digital signature

Re: PIE and static libraries

[Andrey Rahmatullin]

On Sun, May 22, 2016 at 12:23:59PM +0500, Andrey Rahmatullin wrote:
> On Sun, May 22, 2016 at 08:48:19AM +0200, Andreas Metzler wrote:
> > https://lintian.debian.org/tags/hardening-no-pie.html says "It is
> > unlikely to work when compiling static libraries or executables (gcc
> > -static)."
> > 
> > However e.g. https://wiki.debian.org/Hardening does not mention this
> > problem. Is the former incorrect or the latter incomplete?
> dpkg-buildflags(1) mentions it. https://wiki.debian.org/Hardening doesn't
> seem like a good place to repeat all things discussed in that manpage.
> 
> Also, this is a third thread on d-devel@ about PIE and shared libraries
> for the past week.
Sorry, I was assuming that the problem mentioned is about not being able
to compile such static libraries into both shared objects and executables.
The part about static executables may be worth mentioning in
dpkg-buildflags(1) too (though static executables are not a common thing
in the archive).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: PIE and static libraries

[Andrey Rahmatullin]

On Sun, May 22, 2016 at 08:48:19AM +0200, Andreas Metzler wrote:
> https://lintian.debian.org/tags/hardening-no-pie.html says "It is
> unlikely to work when compiling static libraries or executables (gcc
> -static)."
> 
> However e.g. https://wiki.debian.org/Hardening does not mention this
> problem. Is the former incorrect or the latter incomplete?
dpkg-buildflags(1) mentions it. https://wiki.debian.org/Hardening doesn't
seem like a good place to repeat all things discussed in that manpage.

Also, this is a third thread on d-devel@ about PIE and shared libraries
for the past week.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

PIE and static libraries

[Andreas Metzler]

Hello,

https://lintian.debian.org/tags/hardening-no-pie.html says "It is
unlikely to work when compiling static libraries or executables (gcc
-static)."

However e.g. https://wiki.debian.org/Hardening does not mention this
problem. Is the former incorrect or the latter incomplete?

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'