Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Tue, 15 Aug 2017, Ian Lance Taylor wrote:

> On Tue, Aug 15, 2017 at 5:51 AM, Richard Biener  wrote:
> > On Tue, 15 Aug 2017, Richard Biener wrote:
> >
> >> On Mon, 14 Aug 2017, Ian Lance Taylor wrote:
> >>
> >> > On Mon, Aug 14, 2017 at 6:17 AM, Richard Biener  
> >> > wrote:
> >> > > On Fri, 4 Aug 2017, Richard Biener wrote:
> >> > >
> >> > >> On Fri, 28 Jul 2017, Jason Merrill wrote:
> >> > >>
> >> > >> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
> >> > >> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener 
> >> > >> > >  wrote:
> >> > >> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
> >> > >> > > >
> >> > >> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
> >> > >> > > > >
> >> > >> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
> >> > >> > > > > >
> >> > >> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
> >> > >> > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > > This is a repost (unchanged) of the simple-object ELF 
> >> > >> > > > > > > > support for
> >> > >> > > > > > > > early LTO debug transfer from IL object to a separate 
> >> > >> > > > > > > > debug-only
> >> > >> > > > > > > > object
> >> > >> > > > > > > > file.
> >> > >> > > > > > > >
> >> > >> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> >> > >> > > > > > >
> >> > >> > > > > > > Ping.
> >> > >> > > > > >
> >> > >> > > > > > Ping^2.
> >> > >> > > > >
> >> > >> > > > > Ping^3.
> >> > >> > > >
> >> > >> > > > Ping^4.  Adding some more global reviewers to CC.
> >> > >> > >
> >> > >> > > Looking at it now, sorry for the delay.
> >> > >> >
> >> > >> > Actually, the simple-object stuff is more Ian's area.  Looking at 
> >> > >> > the other
> >> > >> > part.
> >> > >>
> >> > >> Ian, ping -- we're through with the other part
> >> > >> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
> >> > >> simple-object part is unchanged at
> >> > >>
> >> > >> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> >> > >>
> >> > >> nearly unchanged from the post last year (which also includes
> >> > >> some description):
> >> > >>
> >> > >> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html
> >> > >
> >> > > Ian, ping again!  (trying now also with the e-mail listed in
> >> > > MAINTAINERS)
> >> > >
> >> > > https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> >> >
> >> > Sorry about that, for some reason GMail didn't flag this message for me.
> >> >
> >> >
> >> > > +  for (ent = buf + entsize; ent < buf + length; ent += entsize)
> >> > > +{
> >> > > +  unsigned sec = type_functions->fetch_Elf_Word (ent);
> >> > > +  if (pfnret[sec - 1] == 0)
> >> > > + keep = 1;
> >> > > +}
> >> > > +  if (keep)
> >> > > +{
> >> > > +  pfnret[sh_link - 1] = 0;
> >> > > +  pfnret[i - 1] = 0;
> >> > > +}
> >> > > + }
> >> >
> >> > It seems to me that if you keep any section of an SHT_GROUP, you need
> >> > to keep all the sections.  Otherwise the section indexes in the group
> >> > will be left dangling.
> >>
> >> Note that all sections are "kept", removed sections are just marked
> >> as SHT_NULL.  Which means that indeed some group members might
> >> be SHT_NULL.
> >>
> >> Note that I don't remember if I actually run into any SHT_GROUP
> >> member being necessary when another is not (IIRC this was with
> >> debug types sections or so).  I'll double-check and add a comment
> >> where this matters for early-debug (the simple-object code tries
> >> to be more generic but obviously all testing was done with
> >> early-debug section copying in mind).
> >
> > That said, the callback is supposed to mark interesting stuff as not
> > deleted.  The simple-object interface is only giving some leeway
> > to simplify things (pulling in the container, directly dependent
> > sections) to generate a valid ELF output.
> >
> >> > The symbol table handling is pretty awful.  Can't you just remove the
> >> > symbols you don't want?  You will have to update the sh_info field of
> >> > the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
> >> > symbol in the table.
> >>
> >> Maybe that's what I was missing, let me see if I can make it work.
> >> It looks like if there's no STB_GLOBAL symbol the index is one after
> >> the last symbol and it seems it is the first ! STB_LOCAL symbol
> >> ("One greater than the symbol table index of the last local symbol
> >> (binding STB_LOCAL)").
> >>
> >> Ah, and implementing this now I remember why I chose the "awkward"
> >> way.  This was to preserve symtab indices to not need to rewrite
> >> relocation sections...  There isn't a documented way to make
> >> an "none" symtab entry (eventually duplicating UND might work
> >> but IIRC it messes up ordering as UND is STB_LOCAL and thus may
> >> not appear after the first ! STB_LOCAL symbol).  As comments in
> >> the code indicate I tried to mangle things in a way that makes
> >> both GNU ld and gold happy...
> >>
> 

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Ian Lance Taylor via gcc-patches]

On Tue, Aug 15, 2017 at 5:51 AM, Richard Biener  wrote:
> On Tue, 15 Aug 2017, Richard Biener wrote:
>
>> On Mon, 14 Aug 2017, Ian Lance Taylor wrote:
>>
>> > On Mon, Aug 14, 2017 at 6:17 AM, Richard Biener  wrote:
>> > > On Fri, 4 Aug 2017, Richard Biener wrote:
>> > >
>> > >> On Fri, 28 Jul 2017, Jason Merrill wrote:
>> > >>
>> > >> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
>> > >> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  
>> > >> > > wrote:
>> > >> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
>> > >> > > >
>> > >> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
>> > >> > > > >
>> > >> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
>> > >> > > > > >
>> > >> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
>> > >> > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > > This is a repost (unchanged) of the simple-object ELF 
>> > >> > > > > > > > support for
>> > >> > > > > > > > early LTO debug transfer from IL object to a separate 
>> > >> > > > > > > > debug-only
>> > >> > > > > > > > object
>> > >> > > > > > > > file.
>> > >> > > > > > > >
>> > >> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
>> > >> > > > > > >
>> > >> > > > > > > Ping.
>> > >> > > > > >
>> > >> > > > > > Ping^2.
>> > >> > > > >
>> > >> > > > > Ping^3.
>> > >> > > >
>> > >> > > > Ping^4.  Adding some more global reviewers to CC.
>> > >> > >
>> > >> > > Looking at it now, sorry for the delay.
>> > >> >
>> > >> > Actually, the simple-object stuff is more Ian's area.  Looking at the 
>> > >> > other
>> > >> > part.
>> > >>
>> > >> Ian, ping -- we're through with the other part
>> > >> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
>> > >> simple-object part is unchanged at
>> > >>
>> > >> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
>> > >>
>> > >> nearly unchanged from the post last year (which also includes
>> > >> some description):
>> > >>
>> > >> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html
>> > >
>> > > Ian, ping again!  (trying now also with the e-mail listed in
>> > > MAINTAINERS)
>> > >
>> > > https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
>> >
>> > Sorry about that, for some reason GMail didn't flag this message for me.
>> >
>> >
>> > > +  for (ent = buf + entsize; ent < buf + length; ent += entsize)
>> > > +{
>> > > +  unsigned sec = type_functions->fetch_Elf_Word (ent);
>> > > +  if (pfnret[sec - 1] == 0)
>> > > + keep = 1;
>> > > +}
>> > > +  if (keep)
>> > > +{
>> > > +  pfnret[sh_link - 1] = 0;
>> > > +  pfnret[i - 1] = 0;
>> > > +}
>> > > + }
>> >
>> > It seems to me that if you keep any section of an SHT_GROUP, you need
>> > to keep all the sections.  Otherwise the section indexes in the group
>> > will be left dangling.
>>
>> Note that all sections are "kept", removed sections are just marked
>> as SHT_NULL.  Which means that indeed some group members might
>> be SHT_NULL.
>>
>> Note that I don't remember if I actually run into any SHT_GROUP
>> member being necessary when another is not (IIRC this was with
>> debug types sections or so).  I'll double-check and add a comment
>> where this matters for early-debug (the simple-object code tries
>> to be more generic but obviously all testing was done with
>> early-debug section copying in mind).
>
> That said, the callback is supposed to mark interesting stuff as not
> deleted.  The simple-object interface is only giving some leeway
> to simplify things (pulling in the container, directly dependent
> sections) to generate a valid ELF output.
>
>> > The symbol table handling is pretty awful.  Can't you just remove the
>> > symbols you don't want?  You will have to update the sh_info field of
>> > the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
>> > symbol in the table.
>>
>> Maybe that's what I was missing, let me see if I can make it work.
>> It looks like if there's no STB_GLOBAL symbol the index is one after
>> the last symbol and it seems it is the first ! STB_LOCAL symbol
>> ("One greater than the symbol table index of the last local symbol
>> (binding STB_LOCAL)").
>>
>> Ah, and implementing this now I remember why I chose the "awkward"
>> way.  This was to preserve symtab indices to not need to rewrite
>> relocation sections...  There isn't a documented way to make
>> an "none" symtab entry (eventually duplicating UND might work
>> but IIRC it messes up ordering as UND is STB_LOCAL and thus may
>> not appear after the first ! STB_LOCAL symbol).  As comments in
>> the code indicate I tried to mangle things in a way that makes
>> both GNU ld and gold happy...
>>
>> Any suggestion how to improve things when not removing symbols?
>> (I could of course remove things from the end of the symtab)
>> Like replacing local symbols with UND and globals with
>> a COMMON of zero size (and no name)?
>>
>> I suppose rewriting relocation sections is possible but 

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Tue, 15 Aug 2017, Richard Biener wrote:

> On Mon, 14 Aug 2017, Ian Lance Taylor wrote:
> 
> > On Mon, Aug 14, 2017 at 6:17 AM, Richard Biener  wrote:
> > > On Fri, 4 Aug 2017, Richard Biener wrote:
> > >
> > >> On Fri, 28 Jul 2017, Jason Merrill wrote:
> > >>
> > >> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
> > >> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  
> > >> > > wrote:
> > >> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
> > >> > > >
> > >> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
> > >> > > > >
> > >> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
> > >> > > > > >
> > >> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > This is a repost (unchanged) of the simple-object ELF 
> > >> > > > > > > > support for
> > >> > > > > > > > early LTO debug transfer from IL object to a separate 
> > >> > > > > > > > debug-only
> > >> > > > > > > > object
> > >> > > > > > > > file.
> > >> > > > > > > >
> > >> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> > >> > > > > > >
> > >> > > > > > > Ping.
> > >> > > > > >
> > >> > > > > > Ping^2.
> > >> > > > >
> > >> > > > > Ping^3.
> > >> > > >
> > >> > > > Ping^4.  Adding some more global reviewers to CC.
> > >> > >
> > >> > > Looking at it now, sorry for the delay.
> > >> >
> > >> > Actually, the simple-object stuff is more Ian's area.  Looking at the 
> > >> > other
> > >> > part.
> > >>
> > >> Ian, ping -- we're through with the other part
> > >> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
> > >> simple-object part is unchanged at
> > >>
> > >> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> > >>
> > >> nearly unchanged from the post last year (which also includes
> > >> some description):
> > >>
> > >> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html
> > >
> > > Ian, ping again!  (trying now also with the e-mail listed in
> > > MAINTAINERS)
> > >
> > > https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> > 
> > Sorry about that, for some reason GMail didn't flag this message for me.
> > 
> > 
> > > +  for (ent = buf + entsize; ent < buf + length; ent += entsize)
> > > +{
> > > +  unsigned sec = type_functions->fetch_Elf_Word (ent);
> > > +  if (pfnret[sec - 1] == 0)
> > > + keep = 1;
> > > +}
> > > +  if (keep)
> > > +{
> > > +  pfnret[sh_link - 1] = 0;
> > > +  pfnret[i - 1] = 0;
> > > +}
> > > + }
> > 
> > It seems to me that if you keep any section of an SHT_GROUP, you need
> > to keep all the sections.  Otherwise the section indexes in the group
> > will be left dangling.
> 
> Note that all sections are "kept", removed sections are just marked
> as SHT_NULL.  Which means that indeed some group members might
> be SHT_NULL.
> 
> Note that I don't remember if I actually run into any SHT_GROUP
> member being necessary when another is not (IIRC this was with
> debug types sections or so).  I'll double-check and add a comment
> where this matters for early-debug (the simple-object code tries
> to be more generic but obviously all testing was done with
> early-debug section copying in mind).

That said, the callback is supposed to mark interesting stuff as not
deleted.  The simple-object interface is only giving some leeway
to simplify things (pulling in the container, directly dependent
sections) to generate a valid ELF output.

> > The symbol table handling is pretty awful.  Can't you just remove the
> > symbols you don't want?  You will have to update the sh_info field of
> > the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
> > symbol in the table.
> 
> Maybe that's what I was missing, let me see if I can make it work.
> It looks like if there's no STB_GLOBAL symbol the index is one after
> the last symbol and it seems it is the first ! STB_LOCAL symbol
> ("One greater than the symbol table index of the last local symbol 
> (binding STB_LOCAL)").
> 
> Ah, and implementing this now I remember why I chose the "awkward"
> way.  This was to preserve symtab indices to not need to rewrite
> relocation sections...  There isn't a documented way to make
> an "none" symtab entry (eventually duplicating UND might work
> but IIRC it messes up ordering as UND is STB_LOCAL and thus may
> not appear after the first ! STB_LOCAL symbol).  As comments in
> the code indicate I tried to mangle things in a way that makes
> both GNU ld and gold happy...
> 
> Any suggestion how to improve things when not removing symbols?
> (I could of course remove things from the end of the symtab)
> Like replacing local symbols with UND and globals with
> a COMMON of zero size (and no name)?
> 
> I suppose rewriting relocation sections is possible but I tried
> to do as little as necessary (like not actually removing sections).

So the following is an update which changes how I remove symbols
by making removed locals copies of UND and 

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Ian Lance Taylor via gcc-patches]

On Tue, Aug 15, 2017 at 1:30 AM, Richard Biener  wrote:
> On Mon, 14 Aug 2017, Ian Lance Taylor wrote:
>
>> It seems to me that if you keep any section of an SHT_GROUP, you need
>> to keep all the sections.  Otherwise the section indexes in the group
>> will be left dangling.
>
> Note that all sections are "kept", removed sections are just marked
> as SHT_NULL.  Which means that indeed some group members might
> be SHT_NULL.
>
> Note that I don't remember if I actually run into any SHT_GROUP
> member being necessary when another is not (IIRC this was with
> debug types sections or so).  I'll double-check and add a comment
> where this matters for early-debug (the simple-object code tries
> to be more generic but obviously all testing was done with
> early-debug section copying in mind).

In general they are SHT_GROUP for a reason, that reason being that if
you keep one of the group sections you need to keep all of them.  If
that didn't matter, there wouldn't be a SHT_GROUP section at all.  So
I think it would be appropriate that if you keep one of a group you
keep all of the group.


>> The symbol table handling is pretty awful.  Can't you just remove the
>> symbols you don't want?  You will have to update the sh_info field of
>> the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
>> symbol in the table.
>
> Maybe that's what I was missing, let me see if I can make it work.
> It looks like if there's no STB_GLOBAL symbol the index is one after
> the last symbol and it seems it is the first ! STB_LOCAL symbol
> ("One greater than the symbol table index of the last local symbol
> (binding STB_LOCAL)").
>
> Ah, and implementing this now I remember why I chose the "awkward"
> way.  This was to preserve symtab indices to not need to rewrite
> relocation sections...  There isn't a documented way to make
> an "none" symtab entry (eventually duplicating UND might work
> but IIRC it messes up ordering as UND is STB_LOCAL and thus may
> not appear after the first ! STB_LOCAL symbol).  As comments in
> the code indicate I tried to mangle things in a way that makes
> both GNU ld and gold happy...

Bother.

> Any suggestion how to improve things when not removing symbols?
> (I could of course remove things from the end of the symtab)
> Like replacing local symbols with UND and globals with
> a COMMON of zero size (and no name)?

You can have a UND global symbol, after all.  You just shouldn't
simply zero out the entire symbol, as that will indeed change the
binding to STB_LOCAL.  So probably the best approach is to change
every unwanted symbol to SHN_UNDEF, keep the binding as is, and zero
all the other fields.  That should let the linker simply discard the
symbol, I hope.

> I suppose rewriting relocation sections is possible but I tried
> to do as little as necessary (like not actually removing sections).

The patch is OK with the above suggestions, assuming they work.

Ian

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Mon, 14 Aug 2017, Ian Lance Taylor wrote:

> On Mon, Aug 14, 2017 at 6:17 AM, Richard Biener  wrote:
> > On Fri, 4 Aug 2017, Richard Biener wrote:
> >
> >> On Fri, 28 Jul 2017, Jason Merrill wrote:
> >>
> >> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
> >> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  
> >> > > wrote:
> >> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
> >> > > >
> >> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
> >> > > > >
> >> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
> >> > > > > >
> >> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > This is a repost (unchanged) of the simple-object ELF 
> >> > > > > > > > support for
> >> > > > > > > > early LTO debug transfer from IL object to a separate 
> >> > > > > > > > debug-only
> >> > > > > > > > object
> >> > > > > > > > file.
> >> > > > > > > >
> >> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> >> > > > > > >
> >> > > > > > > Ping.
> >> > > > > >
> >> > > > > > Ping^2.
> >> > > > >
> >> > > > > Ping^3.
> >> > > >
> >> > > > Ping^4.  Adding some more global reviewers to CC.
> >> > >
> >> > > Looking at it now, sorry for the delay.
> >> >
> >> > Actually, the simple-object stuff is more Ian's area.  Looking at the 
> >> > other
> >> > part.
> >>
> >> Ian, ping -- we're through with the other part
> >> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
> >> simple-object part is unchanged at
> >>
> >> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> >>
> >> nearly unchanged from the post last year (which also includes
> >> some description):
> >>
> >> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html
> >
> > Ian, ping again!  (trying now also with the e-mail listed in
> > MAINTAINERS)
> >
> > https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> 
> Sorry about that, for some reason GMail didn't flag this message for me.
> 
> 
> > +  for (ent = buf + entsize; ent < buf + length; ent += entsize)
> > +{
> > +  unsigned sec = type_functions->fetch_Elf_Word (ent);
> > +  if (pfnret[sec - 1] == 0)
> > + keep = 1;
> > +}
> > +  if (keep)
> > +{
> > +  pfnret[sh_link - 1] = 0;
> > +  pfnret[i - 1] = 0;
> > +}
> > + }
> 
> It seems to me that if you keep any section of an SHT_GROUP, you need
> to keep all the sections.  Otherwise the section indexes in the group
> will be left dangling.

Note that all sections are "kept", removed sections are just marked
as SHT_NULL.  Which means that indeed some group members might
be SHT_NULL.

Note that I don't remember if I actually run into any SHT_GROUP
member being necessary when another is not (IIRC this was with
debug types sections or so).  I'll double-check and add a comment
where this matters for early-debug (the simple-object code tries
to be more generic but obviously all testing was done with
early-debug section copying in mind).

> The symbol table handling is pretty awful.  Can't you just remove the
> symbols you don't want?  You will have to update the sh_info field of
> the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
> symbol in the table.

Maybe that's what I was missing, let me see if I can make it work.
It looks like if there's no STB_GLOBAL symbol the index is one after
the last symbol and it seems it is the first ! STB_LOCAL symbol
("One greater than the symbol table index of the last local symbol 
(binding STB_LOCAL)").

Ah, and implementing this now I remember why I chose the "awkward"
way.  This was to preserve symtab indices to not need to rewrite
relocation sections...  There isn't a documented way to make
an "none" symtab entry (eventually duplicating UND might work
but IIRC it messes up ordering as UND is STB_LOCAL and thus may
not appear after the first ! STB_LOCAL symbol).  As comments in
the code indicate I tried to mangle things in a way that makes
both GNU ld and gold happy...

Any suggestion how to improve things when not removing symbols?
(I could of course remove things from the end of the symtab)
Like replacing local symbols with UND and globals with
a COMMON of zero size (and no name)?

I suppose rewriting relocation sections is possible but I tried
to do as little as necessary (like not actually removing sections).

Thanks,
Richard.

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Ian Lance Taylor via gcc-patches]

On Mon, Aug 14, 2017 at 6:17 AM, Richard Biener  wrote:
> On Fri, 4 Aug 2017, Richard Biener wrote:
>
>> On Fri, 28 Jul 2017, Jason Merrill wrote:
>>
>> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
>> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  
>> > > wrote:
>> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
>> > > >
>> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
>> > > > >
>> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
>> > > > > >
>> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > This is a repost (unchanged) of the simple-object ELF support 
>> > > > > > > > for
>> > > > > > > > early LTO debug transfer from IL object to a separate 
>> > > > > > > > debug-only
>> > > > > > > > object
>> > > > > > > > file.
>> > > > > > > >
>> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
>> > > > > > >
>> > > > > > > Ping.
>> > > > > >
>> > > > > > Ping^2.
>> > > > >
>> > > > > Ping^3.
>> > > >
>> > > > Ping^4.  Adding some more global reviewers to CC.
>> > >
>> > > Looking at it now, sorry for the delay.
>> >
>> > Actually, the simple-object stuff is more Ian's area.  Looking at the other
>> > part.
>>
>> Ian, ping -- we're through with the other part
>> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
>> simple-object part is unchanged at
>>
>> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
>>
>> nearly unchanged from the post last year (which also includes
>> some description):
>>
>> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html
>
> Ian, ping again!  (trying now also with the e-mail listed in
> MAINTAINERS)
>
> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html

Sorry about that, for some reason GMail didn't flag this message for me.


> +  for (ent = buf + entsize; ent < buf + length; ent += entsize)
> +{
> +  unsigned sec = type_functions->fetch_Elf_Word (ent);
> +  if (pfnret[sec - 1] == 0)
> + keep = 1;
> +}
> +  if (keep)
> +{
> +  pfnret[sh_link - 1] = 0;
> +  pfnret[i - 1] = 0;
> +}
> + }

It seems to me that if you keep any section of an SHT_GROUP, you need
to keep all the sections.  Otherwise the section indexes in the group
will be left dangling.


The symbol table handling is pretty awful.  Can't you just remove the
symbols you don't want?  You will have to update the sh_info field of
the SHT_SYMTAB section, which holds the index of the first STB_GLOBAL
symbol in the table.

Ian

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Fri, 4 Aug 2017, Richard Biener wrote:

> On Fri, 28 Jul 2017, Jason Merrill wrote:
> 
> > On 07/28/2017 05:55 PM, Jason Merrill wrote:
> > > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  wrote:
> > > > On Tue, 4 Jul 2017, Richard Biener wrote:
> > > > 
> > > > > On Tue, 20 Jun 2017, Richard Biener wrote:
> > > > > 
> > > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
> > > > > > 
> > > > > > > On Fri, 19 May 2017, Richard Biener wrote:
> > > > > > > 
> > > > > > > > 
> > > > > > > > This is a repost (unchanged) of the simple-object ELF support 
> > > > > > > > for
> > > > > > > > early LTO debug transfer from IL object to a separate debug-only
> > > > > > > > object
> > > > > > > > file.
> > > > > > > > 
> > > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> > > > > > > 
> > > > > > > Ping.
> > > > > > 
> > > > > > Ping^2.
> > > > > 
> > > > > Ping^3.
> > > > 
> > > > Ping^4.  Adding some more global reviewers to CC.
> > > 
> > > Looking at it now, sorry for the delay.
> > 
> > Actually, the simple-object stuff is more Ian's area.  Looking at the other
> > part.
> 
> Ian, ping -- we're through with the other part
> (https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
> simple-object part is unchanged at
> 
> https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html
> 
> nearly unchanged from the post last year (which also includes 
> some description):
> 
> https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html

Ian, ping again!  (trying now also with the e-mail listed in
MAINTAINERS)

https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html

Thanks,
Richard.

> Thanks,
> Richard.

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Fri, 28 Jul 2017, Jason Merrill wrote:

> On 07/28/2017 05:55 PM, Jason Merrill wrote:
> > On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  wrote:
> > > On Tue, 4 Jul 2017, Richard Biener wrote:
> > > 
> > > > On Tue, 20 Jun 2017, Richard Biener wrote:
> > > > 
> > > > > On Wed, 7 Jun 2017, Richard Biener wrote:
> > > > > 
> > > > > > On Fri, 19 May 2017, Richard Biener wrote:
> > > > > > 
> > > > > > > 
> > > > > > > This is a repost (unchanged) of the simple-object ELF support for
> > > > > > > early LTO debug transfer from IL object to a separate debug-only
> > > > > > > object
> > > > > > > file.
> > > > > > > 
> > > > > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> > > > > > 
> > > > > > Ping.
> > > > > 
> > > > > Ping^2.
> > > > 
> > > > Ping^3.
> > > 
> > > Ping^4.  Adding some more global reviewers to CC.
> > 
> > Looking at it now, sorry for the delay.
> 
> Actually, the simple-object stuff is more Ian's area.  Looking at the other
> part.

Ian, ping -- we're through with the other part
(https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00374.html).  The
simple-object part is unchanged at

https://gcc.gnu.org/ml/gcc-patches/2017-05/msg01541.html

nearly unchanged from the post last year (which also includes 
some description):

https://gcc.gnu.org/ml/gcc-patches/2016-09/msg01292.html

Thanks,
Richard.

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Jason Merrill]

On 07/28/2017 05:55 PM, Jason Merrill wrote:

On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  wrote:

On Tue, 4 Jul 2017, Richard Biener wrote:


On Tue, 20 Jun 2017, Richard Biener wrote:


On Wed, 7 Jun 2017, Richard Biener wrote:


On Fri, 19 May 2017, Richard Biener wrote:



This is a repost (unchanged) of the simple-object ELF support for
early LTO debug transfer from IL object to a separate debug-only object
file.

Bootstrapped and tested on x86_64-unknown-linux-gnu.


Ping.


Ping^2.


Ping^3.


Ping^4.  Adding some more global reviewers to CC.


Looking at it now, sorry for the delay.


Actually, the simple-object stuff is more Ian's area.  Looking at the 
other part.


Jason

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Jason Merrill]

On Fri, Jul 28, 2017 at 8:54 AM, Richard Biener  wrote:
> On Tue, 4 Jul 2017, Richard Biener wrote:
>
>> On Tue, 20 Jun 2017, Richard Biener wrote:
>>
>> > On Wed, 7 Jun 2017, Richard Biener wrote:
>> >
>> > > On Fri, 19 May 2017, Richard Biener wrote:
>> > >
>> > > >
>> > > > This is a repost (unchanged) of the simple-object ELF support for
>> > > > early LTO debug transfer from IL object to a separate debug-only object
>> > > > file.
>> > > >
>> > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
>> > >
>> > > Ping.
>> >
>> > Ping^2.
>>
>> Ping^3.
>
> Ping^4.  Adding some more global reviewers to CC.

Looking at it now, sorry for the delay.

Jason

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Tue, 4 Jul 2017, Richard Biener wrote:

> On Tue, 20 Jun 2017, Richard Biener wrote:
> 
> > On Wed, 7 Jun 2017, Richard Biener wrote:
> > 
> > > On Fri, 19 May 2017, Richard Biener wrote:
> > > 
> > > > 
> > > > This is a repost (unchanged) of the simple-object ELF support for
> > > > early LTO debug transfer from IL object to a separate debug-only object 
> > > > file.
> > > > 
> > > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> > > 
> > > Ping.
> > 
> > Ping^2.
> 
> Ping^3.

Ping^4.  Adding some more global reviewers to CC.

Richard.

> > > > Richard.
> > > > 
> > > > 2017-05-19  Richard Biener  
> > > > 
> > > > include/
> > > > * simple-object.h (simple_object_copy_lto_debug_sections): New
> > > > function.
> > > > 
> > > > libiberty/
> > > > * simple-object-common.h (struct simple_object_functions): Add
> > > > copy_lto_debug_sections hook.
> > > > * simple-object.c: Include fcntl.h.
> > > > (handle_lto_debug_sections): New helper function.
> > > > (simple_object_copy_lto_debug_sections): New function copying
> > > > early LTO debug sections to regular debug sections in a new 
> > > > file.
> > > > (simple_object_start_write): Handle NULL segment_name.
> > > > * simple-object-coff.c (simple_object_coff_functions): Adjust
> > > > for not implemented copy_lto_debug_sections hook.
> > > > * simple-object-mach-o.c (simple_object_mach_o_functions): 
> > > > Likewise.
> > > > * simple-object-xcoff.c (simple_object_xcoff_functions): 
> > > > Likewise.
> > > > * simple-object-elf.c (SHT_NULL, SHT_SYMTAB, SHT_RELA, SHT_REL,
> > > > SHT_GROUP): Add various sectopn header types.
> > > > (SHF_EXCLUDE): Add flag.
> > > > (Elf32_External_Sym, Elf64_External_Sym): Add symbol struct.
> > > > (ELF_ST_BIND, ELF_ST_TYPE, ELF_ST_INFO): Add accessors.
> > > > (STT_OBJECT, STT_FUNC, STT_TLS, STT_GNU_IFUNC): Add Symbol 
> > > > types.
> > > > (STV_DEFAULT): Add symbol visibility.
> > > > (SHN_COMMON): Add special section index name.
> > > > (struct simple_object_elf_write): New.
> > > > (simple_object_elf_start_write): Adjust for new private data.
> > > > (simple_object_elf_write_shdr): Pass in values for all fields
> > > > we write.
> > > > (simple_object_elf_write_to_file): Adjust.  Copy from recorded
> > > > section headers if requested.
> > > > (simple_object_elf_release_write): Release private data.
> > > > (simple_object_elf_copy_lto_debug_sections): Copy and rename 
> > > > sections
> > > > as denoted by PFN and all their dependences, symbols and 
> > > > relocations
> > > > to the empty destination file.
> > > > (simple_object_elf_functions): Adjust for 
> > > > copy_lto_debug_sections hook.
> > > > 
> > > > Index: early-lto-debug/include/simple-object.h
> > > > ===
> > > > --- early-lto-debug.orig/include/simple-object.h2016-10-19 
> > > > 13:19:58.012326431 +0200
> > > > +++ early-lto-debug/include/simple-object.h 2016-10-20 
> > > > 10:51:49.861722998 +0200
> > > > @@ -197,6 +197,14 @@ simple_object_write_to_file (simple_obje
> > > >  extern void
> > > >  simple_object_release_write (simple_object_write *);
> > > >  
> > > > +/* Copy LTO debug sections from SRC_OBJECT to DEST.
> > > > +   If an error occurs, return the errno value in ERR and an error 
> > > > string.  */
> > > > +
> > > > +extern const char *
> > > > +simple_object_copy_lto_debug_sections (simple_object_read *src_object,
> > > > +  const char *dest,
> > > > +  int *err);
> > > > +
> > > >  #ifdef __cplusplus
> > > >  }
> > > >  #endif
> > > > Index: early-lto-debug/libiberty/simple-object-common.h
> > > > ===
> > > > --- early-lto-debug.orig/libiberty/simple-object-common.h   
> > > > 2016-10-19 13:19:58.012326431 +0200
> > > > +++ early-lto-debug/libiberty/simple-object-common.h2016-10-20 
> > > > 10:51:49.865723045 +0200
> > > > @@ -141,6 +141,12 @@ struct simple_object_functions
> > > >  
> > > >/* Release the private data for an simple_object_write.  */
> > > >void (*release_write) (void *);
> > > > +
> > > > +  /* Copy LTO debug sections.  */
> > > > +  const char *(*copy_lto_debug_sections) (simple_object_read *sobj,
> > > > + simple_object_write *dobj,
> > > > + int (*pfn) (const char **),
> > > > + int *err);
> > > >  };
> > > >  
> > > >  /* The known object file formats.  */
> > > > Index: early-lto-debug/libiberty/simple-object-elf.c
> > > > 

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Tue, 20 Jun 2017, Richard Biener wrote:

> On Wed, 7 Jun 2017, Richard Biener wrote:
> 
> > On Fri, 19 May 2017, Richard Biener wrote:
> > 
> > > 
> > > This is a repost (unchanged) of the simple-object ELF support for
> > > early LTO debug transfer from IL object to a separate debug-only object 
> > > file.
> > > 
> > > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> > 
> > Ping.
> 
> Ping^2.

Ping^3.

> > > Richard.
> > > 
> > > 2017-05-19  Richard Biener  
> > > 
> > >   include/
> > >   * simple-object.h (simple_object_copy_lto_debug_sections): New
> > >   function.
> > > 
> > >   libiberty/
> > >   * simple-object-common.h (struct simple_object_functions): Add
> > >   copy_lto_debug_sections hook.
> > >   * simple-object.c: Include fcntl.h.
> > >   (handle_lto_debug_sections): New helper function.
> > >   (simple_object_copy_lto_debug_sections): New function copying
> > >   early LTO debug sections to regular debug sections in a new file.
> > >   (simple_object_start_write): Handle NULL segment_name.
> > >   * simple-object-coff.c (simple_object_coff_functions): Adjust
> > >   for not implemented copy_lto_debug_sections hook.
> > >   * simple-object-mach-o.c (simple_object_mach_o_functions): Likewise.
> > >   * simple-object-xcoff.c (simple_object_xcoff_functions): Likewise.
> > >   * simple-object-elf.c (SHT_NULL, SHT_SYMTAB, SHT_RELA, SHT_REL,
> > >   SHT_GROUP): Add various sectopn header types.
> > >   (SHF_EXCLUDE): Add flag.
> > >   (Elf32_External_Sym, Elf64_External_Sym): Add symbol struct.
> > >   (ELF_ST_BIND, ELF_ST_TYPE, ELF_ST_INFO): Add accessors.
> > >   (STT_OBJECT, STT_FUNC, STT_TLS, STT_GNU_IFUNC): Add Symbol types.
> > >   (STV_DEFAULT): Add symbol visibility.
> > >   (SHN_COMMON): Add special section index name.
> > >   (struct simple_object_elf_write): New.
> > >   (simple_object_elf_start_write): Adjust for new private data.
> > >   (simple_object_elf_write_shdr): Pass in values for all fields
> > >   we write.
> > >   (simple_object_elf_write_to_file): Adjust.  Copy from recorded
> > >   section headers if requested.
> > >   (simple_object_elf_release_write): Release private data.
> > >   (simple_object_elf_copy_lto_debug_sections): Copy and rename sections
> > >   as denoted by PFN and all their dependences, symbols and relocations
> > >   to the empty destination file.
> > >   (simple_object_elf_functions): Adjust for copy_lto_debug_sections hook.
> > > 
> > > Index: early-lto-debug/include/simple-object.h
> > > ===
> > > --- early-lto-debug.orig/include/simple-object.h  2016-10-19 
> > > 13:19:58.012326431 +0200
> > > +++ early-lto-debug/include/simple-object.h   2016-10-20 
> > > 10:51:49.861722998 +0200
> > > @@ -197,6 +197,14 @@ simple_object_write_to_file (simple_obje
> > >  extern void
> > >  simple_object_release_write (simple_object_write *);
> > >  
> > > +/* Copy LTO debug sections from SRC_OBJECT to DEST.
> > > +   If an error occurs, return the errno value in ERR and an error 
> > > string.  */
> > > +
> > > +extern const char *
> > > +simple_object_copy_lto_debug_sections (simple_object_read *src_object,
> > > +const char *dest,
> > > +int *err);
> > > +
> > >  #ifdef __cplusplus
> > >  }
> > >  #endif
> > > Index: early-lto-debug/libiberty/simple-object-common.h
> > > ===
> > > --- early-lto-debug.orig/libiberty/simple-object-common.h 2016-10-19 
> > > 13:19:58.012326431 +0200
> > > +++ early-lto-debug/libiberty/simple-object-common.h  2016-10-20 
> > > 10:51:49.865723045 +0200
> > > @@ -141,6 +141,12 @@ struct simple_object_functions
> > >  
> > >/* Release the private data for an simple_object_write.  */
> > >void (*release_write) (void *);
> > > +
> > > +  /* Copy LTO debug sections.  */
> > > +  const char *(*copy_lto_debug_sections) (simple_object_read *sobj,
> > > +   simple_object_write *dobj,
> > > +   int (*pfn) (const char **),
> > > +   int *err);
> > >  };
> > >  
> > >  /* The known object file formats.  */
> > > Index: early-lto-debug/libiberty/simple-object-elf.c
> > > ===
> > > --- early-lto-debug.orig/libiberty/simple-object-elf.c2016-10-19 
> > > 13:19:58.012326431 +0200
> > > +++ early-lto-debug/libiberty/simple-object-elf.c 2016-10-20 
> > > 10:51:49.865723045 +0200
> > > @@ -183,8 +183,55 @@ typedef struct {
> > >  
> > >  /* Values for sh_type field.  */
> > >  
> > > +#define SHT_NULL 0   /* Section header table entry unused */
> > >  #define SHT_PROGBITS 1   /* Program data */
> > > +#define SHT_SYMTAB   2   /* Link editing symbol table */
> > >  #define SHT_STRTAB   3   /* A string 

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Wed, 7 Jun 2017, Richard Biener wrote:

> On Fri, 19 May 2017, Richard Biener wrote:
> 
> > 
> > This is a repost (unchanged) of the simple-object ELF support for
> > early LTO debug transfer from IL object to a separate debug-only object 
> > file.
> > 
> > Bootstrapped and tested on x86_64-unknown-linux-gnu.
> 
> Ping.

Ping^2.

> > Richard.
> > 
> > 2017-05-19  Richard Biener  
> > 
> > include/
> > * simple-object.h (simple_object_copy_lto_debug_sections): New
> > function.
> > 
> > libiberty/
> > * simple-object-common.h (struct simple_object_functions): Add
> > copy_lto_debug_sections hook.
> > * simple-object.c: Include fcntl.h.
> > (handle_lto_debug_sections): New helper function.
> > (simple_object_copy_lto_debug_sections): New function copying
> > early LTO debug sections to regular debug sections in a new file.
> > (simple_object_start_write): Handle NULL segment_name.
> > * simple-object-coff.c (simple_object_coff_functions): Adjust
> > for not implemented copy_lto_debug_sections hook.
> > * simple-object-mach-o.c (simple_object_mach_o_functions): Likewise.
> > * simple-object-xcoff.c (simple_object_xcoff_functions): Likewise.
> > * simple-object-elf.c (SHT_NULL, SHT_SYMTAB, SHT_RELA, SHT_REL,
> > SHT_GROUP): Add various sectopn header types.
> > (SHF_EXCLUDE): Add flag.
> > (Elf32_External_Sym, Elf64_External_Sym): Add symbol struct.
> > (ELF_ST_BIND, ELF_ST_TYPE, ELF_ST_INFO): Add accessors.
> > (STT_OBJECT, STT_FUNC, STT_TLS, STT_GNU_IFUNC): Add Symbol types.
> > (STV_DEFAULT): Add symbol visibility.
> > (SHN_COMMON): Add special section index name.
> > (struct simple_object_elf_write): New.
> > (simple_object_elf_start_write): Adjust for new private data.
> > (simple_object_elf_write_shdr): Pass in values for all fields
> > we write.
> > (simple_object_elf_write_to_file): Adjust.  Copy from recorded
> > section headers if requested.
> > (simple_object_elf_release_write): Release private data.
> > (simple_object_elf_copy_lto_debug_sections): Copy and rename sections
> > as denoted by PFN and all their dependences, symbols and relocations
> > to the empty destination file.
> > (simple_object_elf_functions): Adjust for copy_lto_debug_sections hook.
> > 
> > Index: early-lto-debug/include/simple-object.h
> > ===
> > --- early-lto-debug.orig/include/simple-object.h2016-10-19 
> > 13:19:58.012326431 +0200
> > +++ early-lto-debug/include/simple-object.h 2016-10-20 10:51:49.861722998 
> > +0200
> > @@ -197,6 +197,14 @@ simple_object_write_to_file (simple_obje
> >  extern void
> >  simple_object_release_write (simple_object_write *);
> >  
> > +/* Copy LTO debug sections from SRC_OBJECT to DEST.
> > +   If an error occurs, return the errno value in ERR and an error string.  
> > */
> > +
> > +extern const char *
> > +simple_object_copy_lto_debug_sections (simple_object_read *src_object,
> > +  const char *dest,
> > +  int *err);
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > Index: early-lto-debug/libiberty/simple-object-common.h
> > ===
> > --- early-lto-debug.orig/libiberty/simple-object-common.h   2016-10-19 
> > 13:19:58.012326431 +0200
> > +++ early-lto-debug/libiberty/simple-object-common.h2016-10-20 
> > 10:51:49.865723045 +0200
> > @@ -141,6 +141,12 @@ struct simple_object_functions
> >  
> >/* Release the private data for an simple_object_write.  */
> >void (*release_write) (void *);
> > +
> > +  /* Copy LTO debug sections.  */
> > +  const char *(*copy_lto_debug_sections) (simple_object_read *sobj,
> > + simple_object_write *dobj,
> > + int (*pfn) (const char **),
> > + int *err);
> >  };
> >  
> >  /* The known object file formats.  */
> > Index: early-lto-debug/libiberty/simple-object-elf.c
> > ===
> > --- early-lto-debug.orig/libiberty/simple-object-elf.c  2016-10-19 
> > 13:19:58.012326431 +0200
> > +++ early-lto-debug/libiberty/simple-object-elf.c   2016-10-20 
> > 10:51:49.865723045 +0200
> > @@ -183,8 +183,55 @@ typedef struct {
> >  
> >  /* Values for sh_type field.  */
> >  
> > +#define SHT_NULL   0   /* Section header table entry unused */
> >  #define SHT_PROGBITS   1   /* Program data */
> > +#define SHT_SYMTAB 2   /* Link editing symbol table */
> >  #define SHT_STRTAB 3   /* A string table */
> > +#define SHT_RELA   4   /* Relocation entries with addends */
> > +#define SHT_REL9   /* Relocation entries, no 
> > addends */
> > +#define SHT_GROUP  

Re: [PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

On Fri, 19 May 2017, Richard Biener wrote:

> 
> This is a repost (unchanged) of the simple-object ELF support for
> early LTO debug transfer from IL object to a separate debug-only object 
> file.
> 
> Bootstrapped and tested on x86_64-unknown-linux-gnu.

Ping.

> Richard.
> 
> 2017-05-19  Richard Biener  
> 
>   include/
>   * simple-object.h (simple_object_copy_lto_debug_sections): New
>   function.
> 
>   libiberty/
>   * simple-object-common.h (struct simple_object_functions): Add
>   copy_lto_debug_sections hook.
>   * simple-object.c: Include fcntl.h.
>   (handle_lto_debug_sections): New helper function.
>   (simple_object_copy_lto_debug_sections): New function copying
>   early LTO debug sections to regular debug sections in a new file.
>   (simple_object_start_write): Handle NULL segment_name.
>   * simple-object-coff.c (simple_object_coff_functions): Adjust
>   for not implemented copy_lto_debug_sections hook.
>   * simple-object-mach-o.c (simple_object_mach_o_functions): Likewise.
>   * simple-object-xcoff.c (simple_object_xcoff_functions): Likewise.
>   * simple-object-elf.c (SHT_NULL, SHT_SYMTAB, SHT_RELA, SHT_REL,
>   SHT_GROUP): Add various sectopn header types.
>   (SHF_EXCLUDE): Add flag.
>   (Elf32_External_Sym, Elf64_External_Sym): Add symbol struct.
>   (ELF_ST_BIND, ELF_ST_TYPE, ELF_ST_INFO): Add accessors.
>   (STT_OBJECT, STT_FUNC, STT_TLS, STT_GNU_IFUNC): Add Symbol types.
>   (STV_DEFAULT): Add symbol visibility.
>   (SHN_COMMON): Add special section index name.
>   (struct simple_object_elf_write): New.
>   (simple_object_elf_start_write): Adjust for new private data.
>   (simple_object_elf_write_shdr): Pass in values for all fields
>   we write.
>   (simple_object_elf_write_to_file): Adjust.  Copy from recorded
>   section headers if requested.
>   (simple_object_elf_release_write): Release private data.
>   (simple_object_elf_copy_lto_debug_sections): Copy and rename sections
>   as denoted by PFN and all their dependences, symbols and relocations
>   to the empty destination file.
>   (simple_object_elf_functions): Adjust for copy_lto_debug_sections hook.
> 
> Index: early-lto-debug/include/simple-object.h
> ===
> --- early-lto-debug.orig/include/simple-object.h  2016-10-19 
> 13:19:58.012326431 +0200
> +++ early-lto-debug/include/simple-object.h   2016-10-20 10:51:49.861722998 
> +0200
> @@ -197,6 +197,14 @@ simple_object_write_to_file (simple_obje
>  extern void
>  simple_object_release_write (simple_object_write *);
>  
> +/* Copy LTO debug sections from SRC_OBJECT to DEST.
> +   If an error occurs, return the errno value in ERR and an error string.  */
> +
> +extern const char *
> +simple_object_copy_lto_debug_sections (simple_object_read *src_object,
> +const char *dest,
> +int *err);
> +
>  #ifdef __cplusplus
>  }
>  #endif
> Index: early-lto-debug/libiberty/simple-object-common.h
> ===
> --- early-lto-debug.orig/libiberty/simple-object-common.h 2016-10-19 
> 13:19:58.012326431 +0200
> +++ early-lto-debug/libiberty/simple-object-common.h  2016-10-20 
> 10:51:49.865723045 +0200
> @@ -141,6 +141,12 @@ struct simple_object_functions
>  
>/* Release the private data for an simple_object_write.  */
>void (*release_write) (void *);
> +
> +  /* Copy LTO debug sections.  */
> +  const char *(*copy_lto_debug_sections) (simple_object_read *sobj,
> +   simple_object_write *dobj,
> +   int (*pfn) (const char **),
> +   int *err);
>  };
>  
>  /* The known object file formats.  */
> Index: early-lto-debug/libiberty/simple-object-elf.c
> ===
> --- early-lto-debug.orig/libiberty/simple-object-elf.c2016-10-19 
> 13:19:58.012326431 +0200
> +++ early-lto-debug/libiberty/simple-object-elf.c 2016-10-20 
> 10:51:49.865723045 +0200
> @@ -183,8 +183,55 @@ typedef struct {
>  
>  /* Values for sh_type field.  */
>  
> +#define SHT_NULL 0   /* Section header table entry unused */
>  #define SHT_PROGBITS 1   /* Program data */
> +#define SHT_SYMTAB   2   /* Link editing symbol table */
>  #define SHT_STRTAB   3   /* A string table */
> +#define SHT_RELA 4   /* Relocation entries with addends */
> +#define SHT_REL  9   /* Relocation entries, no 
> addends */
> +#define SHT_GROUP17  /* Section contains a section group */
> +
> +/* Values for sh_flags field.  */
> +
> +#define SHF_EXCLUDE  0x8000  /* Link editor is to exclude this
> +

[PATCH][1/2] Early LTO debug, simple-object part

[Richard Biener]

This is a repost (unchanged) of the simple-object ELF support for
early LTO debug transfer from IL object to a separate debug-only object 
file.

Bootstrapped and tested on x86_64-unknown-linux-gnu.

Richard.

2017-05-19  Richard Biener  

include/
* simple-object.h (simple_object_copy_lto_debug_sections): New
function.

libiberty/
* simple-object-common.h (struct simple_object_functions): Add
copy_lto_debug_sections hook.
* simple-object.c: Include fcntl.h.
(handle_lto_debug_sections): New helper function.
(simple_object_copy_lto_debug_sections): New function copying
early LTO debug sections to regular debug sections in a new file.
(simple_object_start_write): Handle NULL segment_name.
* simple-object-coff.c (simple_object_coff_functions): Adjust
for not implemented copy_lto_debug_sections hook.
* simple-object-mach-o.c (simple_object_mach_o_functions): Likewise.
* simple-object-xcoff.c (simple_object_xcoff_functions): Likewise.
* simple-object-elf.c (SHT_NULL, SHT_SYMTAB, SHT_RELA, SHT_REL,
SHT_GROUP): Add various sectopn header types.
(SHF_EXCLUDE): Add flag.
(Elf32_External_Sym, Elf64_External_Sym): Add symbol struct.
(ELF_ST_BIND, ELF_ST_TYPE, ELF_ST_INFO): Add accessors.
(STT_OBJECT, STT_FUNC, STT_TLS, STT_GNU_IFUNC): Add Symbol types.
(STV_DEFAULT): Add symbol visibility.
(SHN_COMMON): Add special section index name.
(struct simple_object_elf_write): New.
(simple_object_elf_start_write): Adjust for new private data.
(simple_object_elf_write_shdr): Pass in values for all fields
we write.
(simple_object_elf_write_to_file): Adjust.  Copy from recorded
section headers if requested.
(simple_object_elf_release_write): Release private data.
(simple_object_elf_copy_lto_debug_sections): Copy and rename sections
as denoted by PFN and all their dependences, symbols and relocations
to the empty destination file.
(simple_object_elf_functions): Adjust for copy_lto_debug_sections hook.

Index: early-lto-debug/include/simple-object.h
===
--- early-lto-debug.orig/include/simple-object.h2016-10-19 
13:19:58.012326431 +0200
+++ early-lto-debug/include/simple-object.h 2016-10-20 10:51:49.861722998 
+0200
@@ -197,6 +197,14 @@ simple_object_write_to_file (simple_obje
 extern void
 simple_object_release_write (simple_object_write *);
 
+/* Copy LTO debug sections from SRC_OBJECT to DEST.
+   If an error occurs, return the errno value in ERR and an error string.  */
+
+extern const char *
+simple_object_copy_lto_debug_sections (simple_object_read *src_object,
+  const char *dest,
+  int *err);
+
 #ifdef __cplusplus
 }
 #endif
Index: early-lto-debug/libiberty/simple-object-common.h
===
--- early-lto-debug.orig/libiberty/simple-object-common.h   2016-10-19 
13:19:58.012326431 +0200
+++ early-lto-debug/libiberty/simple-object-common.h2016-10-20 
10:51:49.865723045 +0200
@@ -141,6 +141,12 @@ struct simple_object_functions
 
   /* Release the private data for an simple_object_write.  */
   void (*release_write) (void *);
+
+  /* Copy LTO debug sections.  */
+  const char *(*copy_lto_debug_sections) (simple_object_read *sobj,
+ simple_object_write *dobj,
+ int (*pfn) (const char **),
+ int *err);
 };
 
 /* The known object file formats.  */
Index: early-lto-debug/libiberty/simple-object-elf.c
===
--- early-lto-debug.orig/libiberty/simple-object-elf.c  2016-10-19 
13:19:58.012326431 +0200
+++ early-lto-debug/libiberty/simple-object-elf.c   2016-10-20 
10:51:49.865723045 +0200
@@ -183,8 +183,55 @@ typedef struct {
 
 /* Values for sh_type field.  */
 
+#define SHT_NULL   0   /* Section header table entry unused */
 #define SHT_PROGBITS   1   /* Program data */
+#define SHT_SYMTAB 2   /* Link editing symbol table */
 #define SHT_STRTAB 3   /* A string table */
+#define SHT_RELA   4   /* Relocation entries with addends */
+#define SHT_REL9   /* Relocation entries, no 
addends */
+#define SHT_GROUP  17  /* Section contains a section group */
+
+/* Values for sh_flags field.  */
+
+#define SHF_EXCLUDE0x8000  /* Link editor is to exclude this
+  section from executable and
+  shared library that it builds
+