Re: [PATCH][RFC] Early LTO debug
On 09/06/2016 05:17 AM, Richard Biener wrote: On Fri, 26 Aug 2016, Richard Biener wrote: The patch below is my current development state of Early LTO debug lumped into one big patch (no changelog, sorry). It contains previously posted https://gcc.gnu.org/ml/gcc-patches/2015-08/msg01089.html which extends dwarf2asm.h and adjusts a target hook which means the patch carries the usual tm.texi burden. From previous iterations the patch has removed most of the dwarf2out refactoring to make it easier to review. It does get away with dwarf2out_abstract_function (we have that by means of early debug for all functions) and it moves quite a bunch of stuff from late dwarf processing to early. For actual submission I will try to factor out parts of the latter at least. So the main idea is to emit the DWARF tree we have at dwarf2out_early_finish into "Early LTO DWARF" sections (using .gnu.debuglto_ section prefixes, not .gnu.lto_ because GNU as spits out warnings about bogus section flags used for .gnu.lto_.debug_str). If we produce fat LTO objects the full DWARF info will be emit alongside regular assembly again at dwarf2out_finish time. When streaming out LTO bytecode we record for each interesting tree (decls at the moment) a reference to the "early DIE" generated via a $symbol + offset pair. There is a single symbol emitted for each early debug-info section. At WPA time those references are simply carried through. At LTRANS time those references are read back in and dwarf2out is told about the existence of the early DIE for tree X and it creates a shadow DIE for that, simply refering to the early DIE via a symbolic reference. At dwarf2out_finish time we then annotate the early DIE by means of the shadow DIE which has a DW_AT_abstract_origin refering to the early DIE. This mostly adds location information to DIEs. On the driver side things get a little complicated as we need to funnel through the "Early LTO DWARF" sections to the final link, presenting them as regular debug info sections. To make a smooth transition possible I choose to write a Early LTO DWARF copier in the simple-object framework which basically takes all the early LTO DWARF sections and writes them into a new object file, renaming them on-the-fly. I implemented this for ELF only sofar and the API needs to be changed to reflect the fact that this is really only suited for use as LTO early DWARF section copier (tried to make it a little more generic but that didn't work out very well). So for each LTO input file you get a LTO early debug object which are then partially linked to a single LTO early debug object (we could simply omit that step, it doesn't save very much due to the fact that GNU ld with partial linking doesn't optimize mergeable string sections...). This object is then fed back as LTO link output to the linker. Ideally the linker plugin API would be extended to allow claiming only a subset of sections and allow telling the linker to rename sections that we didn't claim. That would avoid some I/O and disk-space usage. The main motivation for this is to increase the quality of debug information you get when using LTO and the most promising part of testing is [ ... ] [ ... ] Feedback (and actual experience debugging LTO programs with this patch) appreciated. No time to do anything with this directly, but I will say that the work on this is greatly appreciated. Improving the LTO debugging situation is something I know many would like to see happen. jeff
Re: [PATCH][RFC] Early LTO debug
On Tue, 6 Sep 2016, Markus Trippelsdorf wrote:
> On 2016.09.06 at 14:26 +0200, Richard Biener wrote:
> > On Tue, 6 Sep 2016, Markus Trippelsdorf wrote:
> >
> > > On 2016.09.06 at 13:17 +0200, Richard Biener wrote:
> > > >
> > > > The following is an updated patch, mainly stripped out old unnecessary
> > > > cruft and some fixes for an issue that arised when LTOing Firefox.
> > >
> > > One minor issue that I've noticed is that the patch generates a lot of
> > > empty *debugobj* files in $TMPDIR, e.g.:
> > >
> > > % echo 'int main(){}' | g++ -flto -o /dev/null-x c++ -
> > > % ls -l $TMPDIR
> > > total 0
> > > -rw---. 1 trippels trippels 0 Sep 6 12:11 ccenD5Tcdebugobj
> > > -rw---. 1 trippels trippels 0 Sep 6 12:11 ccXzvE4udebugobjtem
> >
> > Ah, make_temp_file actually _creates_ the files already...
> >
> > Fixed with the patch below.
>
> Thanks, it takes care of the debugobjtems, but still creates bogus
> debugobjs.
Ah, yeah. The following incremental patch fixes that as well.
Richard.
Index: early-lto-debug/gcc/lto-wrapper.c
===
--- early-lto-debug.orig/gcc/lto-wrapper.c 2016-09-07 09:12:25.339333496
+0200
+++ early-lto-debug/gcc/lto-wrapper.c 2016-09-07 09:09:19.017138725 +0200
@@ -1435,17 +1435,24 @@ cont1:
}
}
else
-skip_debug = true;
+{
+ unlink_if_ordinary (debug_obj);
+ free (debug_obj);
+ debug_obj = NULL;
+ skip_debug = true;
+}
if (lto_mode == LTO_MODE_LTO)
{
printf ("%s\n", flto_out);
if (!skip_debug)
- printf ("%s\n", debug_obj);
+ {
+ printf ("%s\n", debug_obj);
+ free (debug_obj);
+ debug_obj = NULL;
+ }
free (flto_out);
flto_out = NULL;
- free (debug_obj);
- debug_obj = NULL;
}
else
{
@@ -1593,9 +1600,11 @@ cont:
maybe_unlink (input_names[i]);
}
if (!skip_debug)
- printf ("%s\n", debug_obj);
- free (debug_obj);
- debug_obj = NULL;
+ {
+ printf ("%s\n", debug_obj);
+ free (debug_obj);
+ debug_obj = NULL;
+ }
for (i = 0; i < nr; ++i)
{
fputs (output_names[i], stdout);
Re: [PATCH][RFC] Early LTO debug
On 2016.09.06 at 14:26 +0200, Richard Biener wrote:
> On Tue, 6 Sep 2016, Markus Trippelsdorf wrote:
>
> > On 2016.09.06 at 13:17 +0200, Richard Biener wrote:
> > >
> > > The following is an updated patch, mainly stripped out old unnecessary
> > > cruft and some fixes for an issue that arised when LTOing Firefox.
> >
> > One minor issue that I've noticed is that the patch generates a lot of
> > empty *debugobj* files in $TMPDIR, e.g.:
> >
> > % echo 'int main(){}' | g++ -flto -o /dev/null-x c++ -
> > % ls -l $TMPDIR
> > total 0
> > -rw---. 1 trippels trippels 0 Sep 6 12:11 ccenD5Tcdebugobj
> > -rw---. 1 trippels trippels 0 Sep 6 12:11 ccXzvE4udebugobjtem
>
> Ah, make_temp_file actually _creates_ the files already...
>
> Fixed with the patch below.
Thanks, it takes care of the debugobjtems, but still creates bogus
debugobjs.
--
Markus
Re: [PATCH][RFC] Early LTO debug
On Tue, 6 Sep 2016, Markus Trippelsdorf wrote:
> On 2016.09.06 at 13:17 +0200, Richard Biener wrote:
> >
> > The following is an updated patch, mainly stripped out old unnecessary
> > cruft and some fixes for an issue that arised when LTOing Firefox.
>
> One minor issue that I've noticed is that the patch generates a lot of
> empty *debugobj* files in $TMPDIR, e.g.:
>
> % echo 'int main(){}' | g++ -flto -o /dev/null-x c++ -
> % ls -l $TMPDIR
> total 0
> -rw---. 1 trippels trippels 0 Sep 6 12:11 ccenD5Tcdebugobj
> -rw---. 1 trippels trippels 0 Sep 6 12:11 ccXzvE4udebugobjtem
Ah, make_temp_file actually _creates_ the files already...
Fixed with the patch below.
> The new patch builds LLVM fine with "-flto -g".
Great.
Richard.
Index: early-lto-debug/gcc/lto-wrapper.c
===
--- early-lto-debug.orig/gcc/lto-wrapper.c 2016-09-06 14:26:29.233142490
+0200
+++ early-lto-debug/gcc/lto-wrapper.c 2016-09-06 14:25:44.316618249 +0200
@@ -943,9 +943,10 @@ find_and_merge_options (int fd, off_t fi
return true;
}
-int
-debug_objcopy (const char *infile, const char *outfile)
+const char *
+debug_objcopy (const char *infile)
{
+ const char *outfile;
const char *errmsg;
int err;
@@ -966,12 +967,12 @@ debug_objcopy (const char *infile, const
}
int infd = open (infile, O_RDONLY);
if (infd == -1)
-return 1;
+return NULL;
simple_object_read *inobj = simple_object_start_read (infd, inoff,
"__GNU_LTO",
, );
if (!inobj)
-return 1;
+return NULL;
off_t off, len;
if (simple_object_find_section (inobj, ".gnu.debuglto_.debug_info",
@@ -982,17 +983,21 @@ debug_objcopy (const char *infile, const
simple_object_release_read (inobj);
close (infd);
- return 1;
+ return NULL;
}
+ outfile = make_temp_file ("debugobjtem");
errmsg = simple_object_copy_lto_debug_sections (inobj, outfile, );
if (errmsg)
-fatal_error (0, "%s: %s\n", errmsg, xstrerror (err));
+{
+ unlink_if_ordinary (outfile);
+ fatal_error (0, "%s: %s\n", errmsg, xstrerror (err));
+}
simple_object_release_read (inobj);
close (infd);
- return 0;
+ return outfile;
}
@@ -1401,8 +1406,8 @@ cont1:
if (! skip_debug)
for (i = 0; i < ltoobj_argc; ++i)
{
- char *tem = make_temp_file ("debugobjtem");
- if (!debug_objcopy (ltoobj_argv[i], tem))
+ const char *tem;
+ if ((tem = debug_objcopy (ltoobj_argv[i])))
{
obstack_ptr_grow (_obstack, tem);
n_debugobj++;
Re: [PATCH][RFC] Early LTO debug
On 2016.09.06 at 13:17 +0200, Richard Biener wrote:
>
> The following is an updated patch, mainly stripped out old unnecessary
> cruft and some fixes for an issue that arised when LTOing Firefox.
One minor issue that I've noticed is that the patch generates a lot of
empty *debugobj* files in $TMPDIR, e.g.:
% echo 'int main(){}' | g++ -flto -o /dev/null-x c++ -
% ls -l $TMPDIR
total 0
-rw---. 1 trippels trippels 0 Sep 6 12:11 ccenD5Tcdebugobj
-rw---. 1 trippels trippels 0 Sep 6 12:11 ccXzvE4udebugobjtem
The new patch builds LLVM fine with "-flto -g".
--
Markus
Re: [PATCH][RFC] Early LTO debug
On Fri, 26 Aug 2016, Richard Biener wrote: > > The patch below is my current development state of Early LTO debug lumped > into one big patch (no changelog, sorry). It contains previously posted > > https://gcc.gnu.org/ml/gcc-patches/2015-08/msg01089.html > > which extends dwarf2asm.h and adjusts a target hook which means the > patch carries the usual tm.texi burden. > > From previous iterations the patch has removed most of the dwarf2out > refactoring to make it easier to review. It does get away with > dwarf2out_abstract_function (we have that by means of early debug > for all functions) and it moves quite a bunch of stuff from late > dwarf processing to early. For actual submission I will try to > factor out parts of the latter at least. > > > So the main idea is to emit the DWARF tree we have at > dwarf2out_early_finish into "Early LTO DWARF" sections (using > .gnu.debuglto_ section prefixes, not .gnu.lto_ because GNU as > spits out warnings about bogus section flags used for > .gnu.lto_.debug_str). If we produce fat LTO objects the full > DWARF info will be emit alongside regular assembly again at > dwarf2out_finish time. When streaming out LTO bytecode we > record for each interesting tree (decls at the moment) a reference > to the "early DIE" generated via a $symbol + offset pair. There > is a single symbol emitted for each early debug-info section. > > At WPA time those references are simply carried through. > > At LTRANS time those references are read back in and dwarf2out > is told about the existence of the early DIE for tree X and > it creates a shadow DIE for that, simply refering to the early > DIE via a symbolic reference. At dwarf2out_finish time we then > annotate the early DIE by means of the shadow DIE which has > a DW_AT_abstract_origin refering to the early DIE. This mostly > adds location information to DIEs. > > > On the driver side things get a little complicated as we need to > funnel through the "Early LTO DWARF" sections to the final link, > presenting them as regular debug info sections. To make a smooth > transition possible I choose to write a Early LTO DWARF copier > in the simple-object framework which basically takes all the > early LTO DWARF sections and writes them into a new object file, > renaming them on-the-fly. I implemented this for ELF only sofar > and the API needs to be changed to reflect the fact that this is > really only suited for use as LTO early DWARF section copier > (tried to make it a little more generic but that didn't work > out very well). So for each LTO input file you get a LTO early > debug object which are then partially linked to a single LTO > early debug object (we could simply omit that step, it doesn't > save very much due to the fact that GNU ld with partial linking > doesn't optimize mergeable string sections...). This object > is then fed back as LTO link output to the linker. > > Ideally the linker plugin API would be extended to allow claiming > only a subset of sections and allow telling the linker to rename > sections that we didn't claim. That would avoid some I/O and > disk-space usage. > > > The main motivation for this is to increase the quality of > debug information you get when using LTO and the most promising > part of testing is > > === libstdc++ tests === > > Running target unix/-flto/-g/ > FAIL: 23_containers/deque/cons/clear_allocator.cc execution test > -FAIL: libstdc++-prettyprinters/48362.cc print t1 > -FAIL: libstdc++-prettyprinters/48362.cc print t2 > -FAIL: libstdc++-prettyprinters/cxx11.cc print efl > ... > -FAIL: libstdc++-prettyprinters/whatis.cc whatis unord2_holder > -FAIL: libstdc++-prettyprinters/whatis.cc whatis ustring_holder > > === libstdc++ Summary for unix/-flto/-g/ === > > # of expected passes 11423 > -# of unexpected failures 140 > +# of unexpected failures 1 > # of expected failures 65 > -# of unsupported tests 234 > +# of unsupported tests 243 > > which means all libstdc++ pretty printer tests now pass! > > We get a few guality improvements (mind you, unconditionally added > -flto -g): > > -FAIL: gcc.dg/guality/inline-params.c -O1 execution test > +XPASS: gcc.dg/guality/inline-params.c -O2 execution test > XPASS: gcc.dg/guality/inline-params.c -O2 -flto -fno-use-linker-plugin > -flto-partition=none execution test > +XPASS: gcc.dg/guality/inline-params.c -O2 -flto -fuse-linker-plugin > -fno-fat-lto-objects execution test > +XPASS: gcc.dg/guality/inline-params.c -O3 -g execution test > +XPASS: gcc.dg/guality/inline-params.c -Os execution test > > but also the issue that gdb doesn't cope with the way we represent > VLAs with the new scheme: > > +FAIL: gcc.dg/guality/vla-1.c -O0 line 17 sizeof (a) == 6 > +FAIL: gcc.dg/guality/vla-1.c -O0 line 24 sizeof (a) == 17 * sizeof > (short) > +FAIL: gcc.dg/guality/vla-1.c -O1 line 17 sizeof (a) == 6 > +FAIL:
[PATCH][RFC] Early LTO debug
The patch below is my current development state of Early LTO debug lumped into one big patch (no changelog, sorry). It contains previously posted https://gcc.gnu.org/ml/gcc-patches/2015-08/msg01089.html which extends dwarf2asm.h and adjusts a target hook which means the patch carries the usual tm.texi burden. >From previous iterations the patch has removed most of the dwarf2out refactoring to make it easier to review. It does get away with dwarf2out_abstract_function (we have that by means of early debug for all functions) and it moves quite a bunch of stuff from late dwarf processing to early. For actual submission I will try to factor out parts of the latter at least. So the main idea is to emit the DWARF tree we have at dwarf2out_early_finish into "Early LTO DWARF" sections (using .gnu.debuglto_ section prefixes, not .gnu.lto_ because GNU as spits out warnings about bogus section flags used for .gnu.lto_.debug_str). If we produce fat LTO objects the full DWARF info will be emit alongside regular assembly again at dwarf2out_finish time. When streaming out LTO bytecode we record for each interesting tree (decls at the moment) a reference to the "early DIE" generated via a $symbol + offset pair. There is a single symbol emitted for each early debug-info section. At WPA time those references are simply carried through. At LTRANS time those references are read back in and dwarf2out is told about the existence of the early DIE for tree X and it creates a shadow DIE for that, simply refering to the early DIE via a symbolic reference. At dwarf2out_finish time we then annotate the early DIE by means of the shadow DIE which has a DW_AT_abstract_origin refering to the early DIE. This mostly adds location information to DIEs. On the driver side things get a little complicated as we need to funnel through the "Early LTO DWARF" sections to the final link, presenting them as regular debug info sections. To make a smooth transition possible I choose to write a Early LTO DWARF copier in the simple-object framework which basically takes all the early LTO DWARF sections and writes them into a new object file, renaming them on-the-fly. I implemented this for ELF only sofar and the API needs to be changed to reflect the fact that this is really only suited for use as LTO early DWARF section copier (tried to make it a little more generic but that didn't work out very well). So for each LTO input file you get a LTO early debug object which are then partially linked to a single LTO early debug object (we could simply omit that step, it doesn't save very much due to the fact that GNU ld with partial linking doesn't optimize mergeable string sections...). This object is then fed back as LTO link output to the linker. Ideally the linker plugin API would be extended to allow claiming only a subset of sections and allow telling the linker to rename sections that we didn't claim. That would avoid some I/O and disk-space usage. The main motivation for this is to increase the quality of debug information you get when using LTO and the most promising part of testing is === libstdc++ tests === Running target unix/-flto/-g/ FAIL: 23_containers/deque/cons/clear_allocator.cc execution test -FAIL: libstdc++-prettyprinters/48362.cc print t1 -FAIL: libstdc++-prettyprinters/48362.cc print t2 -FAIL: libstdc++-prettyprinters/cxx11.cc print efl ... -FAIL: libstdc++-prettyprinters/whatis.cc whatis unord2_holder -FAIL: libstdc++-prettyprinters/whatis.cc whatis ustring_holder === libstdc++ Summary for unix/-flto/-g/ === # of expected passes 11423 -# of unexpected failures 140 +# of unexpected failures 1 # of expected failures 65 -# of unsupported tests 234 +# of unsupported tests 243 which means all libstdc++ pretty printer tests now pass! We get a few guality improvements (mind you, unconditionally added -flto -g): -FAIL: gcc.dg/guality/inline-params.c -O1 execution test +XPASS: gcc.dg/guality/inline-params.c -O2 execution test XPASS: gcc.dg/guality/inline-params.c -O2 -flto -fno-use-linker-plugin -flto-partition=none execution test +XPASS: gcc.dg/guality/inline-params.c -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects execution test +XPASS: gcc.dg/guality/inline-params.c -O3 -g execution test +XPASS: gcc.dg/guality/inline-params.c -Os execution test but also the issue that gdb doesn't cope with the way we represent VLAs with the new scheme: +FAIL: gcc.dg/guality/vla-1.c -O0 line 17 sizeof (a) == 6 +FAIL: gcc.dg/guality/vla-1.c -O0 line 24 sizeof (a) == 17 * sizeof (short) +FAIL: gcc.dg/guality/vla-1.c -O1 line 17 sizeof (a) == 6 +FAIL: gcc.dg/guality/vla-1.c -O1 line 24 sizeof (a) == 17 * sizeof (short) FAIL: gcc.dg/guality/vla-1.c -O2 line 17 sizeof (a) == 6 +FAIL: gcc.dg/guality/vla-1.c -O2 line 24 sizeof (a) == 17 * sizeof (short) +FAIL: gcc.dg/guality/vla-1.c -O2 -flto
