Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Leonard Mosescu via lldb-dev]

For core -> minidump conversion, a Python tool might end up as trivial as:

process = target.LoadCore(source_core_file)
process.SaveMinidump(output_minidump_file)


Another angle on reusing existing minidump writing code: we could reuse
just the minidump file creation from Crashpad (the part which constructs
the minidump datastructures and puts together the final minidump file)
without the "process parsing" part of Crashpad (since LLDB already covers
that)

If anyone is interested in exploring that, the interface to this low level
minidump writing part is crashpad::ProcessSnapshot

& the rest of the xxxSnapshot interfaces.


On Wed, Jun 13, 2018 at 3:33 PM, Jim Ingham  wrote:

> Greg already wrote a "save_crashlog" Python command that writes the state
> of the program as a macOS flavor Crashlog file.  It's in
> examples/Python/crashlog.py.  My guess is he had something similar to that
> in mind, but writing a mini dump file instead.
>
> Jim
>
>
> > On Jun 13, 2018, at 3:20 PM, Leonard Mosescu via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> >
> > What about the case where you already have a Unix core file and you
> aren't in a debugger but just want to convert it?
> >
> > Just curious, would a small Python script using the LLDB SB API satisfy
> this requirement?
> >
> >  We could move all the code for consuming and producing Windows
> minidumps and Unix / Mach-O corefiles from LLDB down into LLVMCoreFile,
> write a library like llvm-core that can manipulate or inspect them, then
> have LLDB use it.  Kill 2 birds with one stone that way IMO.
> >
> > I like the idea of factoring out reusable subsystems, and I'd love to
> see something along these lines. Just a word of caution though: the hard
> part may not be the generation of a "structurally valid" minidump file, but
> "parsing" and modeling the process state (figuring out the list of modules
> & memory regions, etc. See the Crashpad implementation for details).
> >
> > On Wed, Jun 13, 2018 at 3:01 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > Yea, I think something like this would actually make a useful llvm
> utility.  Call it llvm-core or something, and it links against the library
> LLVMCoreFile.  We could move all the code for consuming and producing
> Windows minidumps and Unix / Mach-O corefiles from LLDB down into
> LLVMCoreFile, write a library like llvm-core that can manipulate or inspect
> them, then have LLDB use it.  Kill 2 birds with one stone that way IMO.
> >
> > On Wed, Jun 13, 2018 at 2:56 PM Jason Molenda 
> wrote:
> > fwiw I had to prototype a new LC_NOTE load command a year ago in Mach-O
> core files, to specify where the kernel binary was located.  I wrote a
> utility to add the data to an existing corefile - both load command and
> payload - and it was only about five hundred lines of C++.  I didn't link
> against anything but libc, it's such  a simple task I didn't sweat trying
> to find an object-file-reader/writer library.  ELF may be more complicated
> though.
> >
> > > On Jun 13, 2018, at 2:51 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > >
> > > What about the case where you already have a Unix core file and you
> aren't in a debugger but just want to convert it?  It seems like we could
> have a standalone utility that did that (one could imagine doing the
> reverse too).  I'm wondering if it wouldn't be possible to do this as a
> library or something that didn't have any dependencies on LLDB, that way a
> standalone tool could link against this library, and so could LLDB.  I
> think this would improve its usefulness quite a bit.
> > >
> > > On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton 
> wrote:
> > > The goal is to take a live process (regular process just stopped, or a
> core file) and run "save_minidump ..." as a command and export a minidump
> file that can be sent elsewhere. Unix core files are too large to always
> send and they are less useful if they are not examined in the machine that
> they were produced on. So LLDB gives us the connection to the live process,
> and we can then create a minidump file. I am going to create a python
> module that can do this for us.
> > >
> > > Greg
> > >
> > >
> > >> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > >>
> > >> Also, if the goal is to have this upstream somewhere, it would be
> nice to have a tool this be a standalone tool.  This seems like something
> that you shouldn't be required to start up a debugger to do, and probably
> doesn't have many (or any for that matters) on the rest of LLDB.
> > >>
> > >> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu 
> wrote:
> > >> That being said, it's not exactly trivial to produce a good minidump.
> Crashpad has a native & cross-platform minidump writer, that's what I'd
> start with.
> > >>
> > >> Addendum: I realized after sending the email 

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Jim Ingham via lldb-dev]

Greg already wrote a "save_crashlog" Python command that writes the state of 
the program as a macOS flavor Crashlog file.  It's in 
examples/Python/crashlog.py.  My guess is he had something similar to that in 
mind, but writing a mini dump file instead.

Jim


> On Jun 13, 2018, at 3:20 PM, Leonard Mosescu via lldb-dev 
>  wrote:
> 
> What about the case where you already have a Unix core file and you aren't in 
> a debugger but just want to convert it?  
> 
> Just curious, would a small Python script using the LLDB SB API satisfy this 
> requirement? 
> 
>  We could move all the code for consuming and producing Windows minidumps and 
> Unix / Mach-O corefiles from LLDB down into LLVMCoreFile, write a library 
> like llvm-core that can manipulate or inspect them, then have LLDB use it.  
> Kill 2 birds with one stone that way IMO.
> 
> I like the idea of factoring out reusable subsystems, and I'd love to see 
> something along these lines. Just a word of caution though: the hard part may 
> not be the generation of a "structurally valid" minidump file, but "parsing" 
> and modeling the process state (figuring out the list of modules & memory 
> regions, etc. See the Crashpad implementation for details).
> 
> On Wed, Jun 13, 2018 at 3:01 PM, Zachary Turner via lldb-dev 
>  wrote:
> Yea, I think something like this would actually make a useful llvm utility.  
> Call it llvm-core or something, and it links against the library 
> LLVMCoreFile.  We could move all the code for consuming and producing Windows 
> minidumps and Unix / Mach-O corefiles from LLDB down into LLVMCoreFile, write 
> a library like llvm-core that can manipulate or inspect them, then have LLDB 
> use it.  Kill 2 birds with one stone that way IMO.
> 
> On Wed, Jun 13, 2018 at 2:56 PM Jason Molenda  wrote:
> fwiw I had to prototype a new LC_NOTE load command a year ago in Mach-O core 
> files, to specify where the kernel binary was located.  I wrote a utility to 
> add the data to an existing corefile - both load command and payload - and it 
> was only about five hundred lines of C++.  I didn't link against anything but 
> libc, it's such  a simple task I didn't sweat trying to find an 
> object-file-reader/writer library.  ELF may be more complicated though.  
> 
> > On Jun 13, 2018, at 2:51 PM, Zachary Turner via lldb-dev 
> >  wrote:
> > 
> > What about the case where you already have a Unix core file and you aren't 
> > in a debugger but just want to convert it?  It seems like we could have a 
> > standalone utility that did that (one could imagine doing the reverse too). 
> >  I'm wondering if it wouldn't be possible to do this as a library or 
> > something that didn't have any dependencies on LLDB, that way a standalone 
> > tool could link against this library, and so could LLDB.  I think this 
> > would improve its usefulness quite a bit.
> > 
> > On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton  wrote:
> > The goal is to take a live process (regular process just stopped, or a core 
> > file) and run "save_minidump ..." as a command and export a minidump file 
> > that can be sent elsewhere. Unix core files are too large to always send 
> > and they are less useful if they are not examined in the machine that they 
> > were produced on. So LLDB gives us the connection to the live process, and 
> > we can then create a minidump file. I am going to create a python module 
> > that can do this for us.
> > 
> > Greg 
> > 
> > 
> >> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev 
> >>  wrote:
> >> 
> >> Also, if the goal is to have this upstream somewhere, it would be nice to 
> >> have a tool this be a standalone tool.  This seems like something that you 
> >> shouldn't be required to start up a debugger to do, and probably doesn't 
> >> have many (or any for that matters) on the rest of LLDB.
> >> 
> >> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu  wrote:
> >> That being said, it's not exactly trivial to produce a good minidump. 
> >> Crashpad has a native & cross-platform minidump writer, that's what I'd 
> >> start with. 
> >> 
> >> Addendum: I realized after sending the email that if the goal is to 
> >> convert core files -> LLDB -> minidump a lot of the complexity found in 
> >> Crashpad can be avoided, so perhaps writing an LLDB minidump writer from 
> >> scratch would not be too bad.
> >> 
> >> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu  
> >> wrote:
> >> The minidump format is more or less documented in MSDN. 
> >> 
> >> That being said, it's not exactly trivial to produce a good minidump. 
> >> Crashpad has a native & cross-platform minidump writer, that's what I'd 
> >> start with.
> >> 
> >> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev 
> >>  wrote:
> >> Zach's right.  On Windows, lldb can produce a minidump, but it just calls 
> >> out to a Microsoft library to do so.  We don't have any platform-agnostic 
> >> code for producing a minidump.
> >> 
> >> I've also pinged another Googler who 

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Leonard Mosescu via lldb-dev]

>
> What about the case where you already have a Unix core file and you aren't
> in a debugger but just want to convert it?


Just curious, would a small Python script using the LLDB SB API satisfy
this requirement?

 We could move all the code for consuming and producing Windows minidumps
> and Unix / Mach-O corefiles from LLDB down into LLVMCoreFile, write a
> library like llvm-core that can manipulate or inspect them, then have LLDB
> use it.  Kill 2 birds with one stone that way IMO.
>

I like the idea of factoring out reusable subsystems, and I'd love to see
something along these lines. Just a word of caution though: the hard part
may not be the generation of a "structurally valid" minidump file, but
"parsing" and modeling the process state (figuring out the list of modules
& memory regions, etc. See the Crashpad implementation for details).

On Wed, Jun 13, 2018 at 3:01 PM, Zachary Turner via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Yea, I think something like this would actually make a useful llvm
> utility.  Call it llvm-core or something, and it links against the library
> LLVMCoreFile.  We could move all the code for consuming and producing
> Windows minidumps and Unix / Mach-O corefiles from LLDB down into
> LLVMCoreFile, write a library like llvm-core that can manipulate or inspect
> them, then have LLDB use it.  Kill 2 birds with one stone that way IMO.
>
> On Wed, Jun 13, 2018 at 2:56 PM Jason Molenda  wrote:
>
>> fwiw I had to prototype a new LC_NOTE load command a year ago in Mach-O
>> core files, to specify where the kernel binary was located.  I wrote a
>> utility to add the data to an existing corefile - both load command and
>> payload - and it was only about five hundred lines of C++.  I didn't link
>> against anything but libc, it's such  a simple task I didn't sweat trying
>> to find an object-file-reader/writer library.  ELF may be more complicated
>> though.
>>
>> > On Jun 13, 2018, at 2:51 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> >
>> > What about the case where you already have a Unix core file and you
>> aren't in a debugger but just want to convert it?  It seems like we could
>> have a standalone utility that did that (one could imagine doing the
>> reverse too).  I'm wondering if it wouldn't be possible to do this as a
>> library or something that didn't have any dependencies on LLDB, that way a
>> standalone tool could link against this library, and so could LLDB.  I
>> think this would improve its usefulness quite a bit.
>> >
>> > On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton 
>> wrote:
>> > The goal is to take a live process (regular process just stopped, or a
>> core file) and run "save_minidump ..." as a command and export a minidump
>> file that can be sent elsewhere. Unix core files are too large to always
>> send and they are less useful if they are not examined in the machine that
>> they were produced on. So LLDB gives us the connection to the live process,
>> and we can then create a minidump file. I am going to create a python
>> module that can do this for us.
>> >
>> > Greg
>> >
>> >
>> >> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> >>
>> >> Also, if the goal is to have this upstream somewhere, it would be nice
>> to have a tool this be a standalone tool.  This seems like something that
>> you shouldn't be required to start up a debugger to do, and probably
>> doesn't have many (or any for that matters) on the rest of LLDB.
>> >>
>> >> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu 
>> wrote:
>> >> That being said, it's not exactly trivial to produce a good minidump.
>> Crashpad has a native & cross-platform minidump writer, that's what I'd
>> start with.
>> >>
>> >> Addendum: I realized after sending the email that if the goal is to
>> convert core files -> LLDB -> minidump a lot of the complexity found in
>> Crashpad can be avoided, so perhaps writing an LLDB minidump writer from
>> scratch would not be too bad.
>> >>
>> >> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu 
>> wrote:
>> >> The minidump format is more or less documented in MSDN.
>> >>
>> >> That being said, it's not exactly trivial to produce a good minidump.
>> Crashpad has a native & cross-platform minidump writer, that's what I'd
>> start with.
>> >>
>> >> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> >> Zach's right.  On Windows, lldb can produce a minidump, but it just
>> calls out to a Microsoft library to do so.  We don't have any
>> platform-agnostic code for producing a minidump.
>> >>
>> >> I've also pinged another Googler who I know might be interested in
>> converting between minidumps and core files (the opposite direction) to see
>> if he has any additional info.  I don't think he's on lldb-dev, though, so
>> I'll act as a relay if necessary.
>> >>
>> >> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
>> 

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Jason Molenda via lldb-dev]

fwiw I had to prototype a new LC_NOTE load command a year ago in Mach-O core 
files, to specify where the kernel binary was located.  I wrote a utility to 
add the data to an existing corefile - both load command and payload - and it 
was only about five hundred lines of C++.  I didn't link against anything but 
libc, it's such  a simple task I didn't sweat trying to find an 
object-file-reader/writer library.  ELF may be more complicated though.  

> On Jun 13, 2018, at 2:51 PM, Zachary Turner via lldb-dev 
>  wrote:
> 
> What about the case where you already have a Unix core file and you aren't in 
> a debugger but just want to convert it?  It seems like we could have a 
> standalone utility that did that (one could imagine doing the reverse too).  
> I'm wondering if it wouldn't be possible to do this as a library or something 
> that didn't have any dependencies on LLDB, that way a standalone tool could 
> link against this library, and so could LLDB.  I think this would improve its 
> usefulness quite a bit.
> 
> On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton  wrote:
> The goal is to take a live process (regular process just stopped, or a core 
> file) and run "save_minidump ..." as a command and export a minidump file 
> that can be sent elsewhere. Unix core files are too large to always send and 
> they are less useful if they are not examined in the machine that they were 
> produced on. So LLDB gives us the connection to the live process, and we can 
> then create a minidump file. I am going to create a python module that can do 
> this for us.
> 
> Greg 
> 
> 
>> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev 
>>  wrote:
>> 
>> Also, if the goal is to have this upstream somewhere, it would be nice to 
>> have a tool this be a standalone tool.  This seems like something that you 
>> shouldn't be required to start up a debugger to do, and probably doesn't 
>> have many (or any for that matters) on the rest of LLDB.
>> 
>> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu  wrote:
>> That being said, it's not exactly trivial to produce a good minidump. 
>> Crashpad has a native & cross-platform minidump writer, that's what I'd 
>> start with. 
>> 
>> Addendum: I realized after sending the email that if the goal is to convert 
>> core files -> LLDB -> minidump a lot of the complexity found in Crashpad can 
>> be avoided, so perhaps writing an LLDB minidump writer from scratch would 
>> not be too bad.
>> 
>> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu  wrote:
>> The minidump format is more or less documented in MSDN. 
>> 
>> That being said, it's not exactly trivial to produce a good minidump. 
>> Crashpad has a native & cross-platform minidump writer, that's what I'd 
>> start with.
>> 
>> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev 
>>  wrote:
>> Zach's right.  On Windows, lldb can produce a minidump, but it just calls 
>> out to a Microsoft library to do so.  We don't have any platform-agnostic 
>> code for producing a minidump.
>> 
>> I've also pinged another Googler who I know might be interested in 
>> converting between minidumps and core files (the opposite direction) to see 
>> if he has any additional info.  I don't think he's on lldb-dev, though, so 
>> I'll act as a relay if necessary.
>> 
>> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev 
>>  wrote:
>> We can’t produce them, but you should check out the source code of google 
>> breakpad / crashpad which can.
>> 
>> That said it’s a pretty simple format, there may be enough in our consumer 
>> code that should allow you to produce them
>> 
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> 
>> 
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> 
>> 
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Zachary Turner via lldb-dev]

Also one could imagine using it for many other things too.  For example,
given a windows dump file, strip it down (i.e. remove heap, etc), or
similar types of options for operating on Unix core files to remove certain
types of info etc

On Wed, Jun 13, 2018 at 2:51 PM Zachary Turner  wrote:

> What about the case where you already have a Unix core file and you aren't
> in a debugger but just want to convert it?  It seems like we could have a
> standalone utility that did that (one could imagine doing the reverse
> too).  I'm wondering if it wouldn't be possible to do this as a library or
> something that didn't have any dependencies on LLDB, that way a standalone
> tool could link against this library, and so could LLDB.  I think this
> would improve its usefulness quite a bit.
>
> On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton  wrote:
>
>> The goal is to take a live process (regular process just stopped, or a
>> core file) and run "save_minidump ..." as a command and export a minidump
>> file that can be sent elsewhere. Unix core files are too large to always
>> send and they are less useful if they are not examined in the machine that
>> they were produced on. So LLDB gives us the connection to the live process,
>> and we can then create a minidump file. I am going to create a python
>> module that can do this for us.
>>
>> Greg
>>
>>
>> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>> Also, if the goal is to have this upstream somewhere, it would be nice to
>> have a tool this be a standalone tool.  This seems like something that you
>> shouldn't be required to start up a debugger to do, and probably doesn't
>> have many (or any for that matters) on the rest of LLDB.
>>
>> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu 
>> wrote:
>>
>>> That being said, it's not exactly trivial to produce a good minidump.
 Crashpad  has a
 native & cross-platform minidump writer, that's what I'd start with.

>>>
>>> Addendum: I realized after sending the email that if the goal is to
>>> convert core files -> LLDB -> minidump a lot of the complexity found in
>>> Crashpad can be avoided, so perhaps writing an LLDB minidump writer from
>>> scratch would not be too bad.
>>>
>>> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu 
>>> wrote:
>>>
 The minidump format is more or less documented in MSDN
 
 .

 That being said, it's not exactly trivial to produce a good minidump. 
 Crashpad
 has a native &
 cross-platform minidump writer, that's what I'd start with.

 On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> Zach's right.  On Windows, lldb can produce a minidump, but it just
> calls out to a Microsoft library to do so.  We don't have any
> platform-agnostic code for producing a minidump.
>
> I've also pinged another Googler who I know might be interested in
> converting between minidumps and core files (the opposite direction) to 
> see
> if he has any additional info.  I don't think he's on lldb-dev, though, so
> I'll act as a relay if necessary.
>
> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> We can’t produce them, but you should check out the source code of
>> google breakpad / crashpad which can.
>>
>> That said it’s a pretty simple format, there may be enough in our
>> consumer code that should allow you to produce them
>>
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>

>>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Zachary Turner via lldb-dev]

What about the case where you already have a Unix core file and you aren't
in a debugger but just want to convert it?  It seems like we could have a
standalone utility that did that (one could imagine doing the reverse
too).  I'm wondering if it wouldn't be possible to do this as a library or
something that didn't have any dependencies on LLDB, that way a standalone
tool could link against this library, and so could LLDB.  I think this
would improve its usefulness quite a bit.

On Wed, Jun 13, 2018 at 2:42 PM Greg Clayton  wrote:

> The goal is to take a live process (regular process just stopped, or a
> core file) and run "save_minidump ..." as a command and export a minidump
> file that can be sent elsewhere. Unix core files are too large to always
> send and they are less useful if they are not examined in the machine that
> they were produced on. So LLDB gives us the connection to the live process,
> and we can then create a minidump file. I am going to create a python
> module that can do this for us.
>
> Greg
>
>
> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
> Also, if the goal is to have this upstream somewhere, it would be nice to
> have a tool this be a standalone tool.  This seems like something that you
> shouldn't be required to start up a debugger to do, and probably doesn't
> have many (or any for that matters) on the rest of LLDB.
>
> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu 
> wrote:
>
>> That being said, it's not exactly trivial to produce a good minidump.
>>> Crashpad  has a
>>> native & cross-platform minidump writer, that's what I'd start with.
>>>
>>
>> Addendum: I realized after sending the email that if the goal is to
>> convert core files -> LLDB -> minidump a lot of the complexity found in
>> Crashpad can be avoided, so perhaps writing an LLDB minidump writer from
>> scratch would not be too bad.
>>
>> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu 
>> wrote:
>>
>>> The minidump format is more or less documented in MSDN
>>> 
>>> .
>>>
>>> That being said, it's not exactly trivial to produce a good minidump. 
>>> Crashpad
>>> has a native &
>>> cross-platform minidump writer, that's what I'd start with.
>>>
>>> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 Zach's right.  On Windows, lldb can produce a minidump, but it just
 calls out to a Microsoft library to do so.  We don't have any
 platform-agnostic code for producing a minidump.

 I've also pinged another Googler who I know might be interested in
 converting between minidumps and core files (the opposite direction) to see
 if he has any additional info.  I don't think he's on lldb-dev, though, so
 I'll act as a relay if necessary.

 On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
 lldb-dev@lists.llvm.org> wrote:

> We can’t produce them, but you should check out the source code of
> google breakpad / crashpad which can.
>
> That said it’s a pretty simple format, there may be enough in our
> consumer code that should allow you to produce them
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>

 ___
 lldb-dev mailing list
 lldb-dev@lists.llvm.org
 http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


>>>
>> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Greg Clayton via lldb-dev]

The goal is to take a live process (regular process just stopped, or a core 
file) and run "save_minidump ..." as a command and export a minidump file that 
can be sent elsewhere. Unix core files are too large to always send and they 
are less useful if they are not examined in the machine that they were produced 
on. So LLDB gives us the connection to the live process, and we can then create 
a minidump file. I am going to create a python module that can do this for us.

Greg 

> On Jun 13, 2018, at 2:29 PM, Zachary Turner via lldb-dev 
>  wrote:
> 
> Also, if the goal is to have this upstream somewhere, it would be nice to 
> have a tool this be a standalone tool.  This seems like something that you 
> shouldn't be required to start up a debugger to do, and probably doesn't have 
> many (or any for that matters) on the rest of LLDB.
> 
> On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu  > wrote:
> That being said, it's not exactly trivial to produce a good minidump. 
> Crashpad  has a native & 
> cross-platform minidump writer, that's what I'd start with. 
> 
> Addendum: I realized after sending the email that if the goal is to convert 
> core files -> LLDB -> minidump a lot of the complexity found in Crashpad can 
> be avoided, so perhaps writing an LLDB minidump writer from scratch would not 
> be too bad.
> 
> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu  > wrote:
> The minidump format is more or less documented in MSDN 
> .
>  
> 
> That being said, it's not exactly trivial to produce a good minidump. 
> Crashpad  has a native & 
> cross-platform minidump writer, that's what I'd start with.
> 
> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev 
> mailto:lldb-dev@lists.llvm.org>> wrote:
> Zach's right.  On Windows, lldb can produce a minidump, but it just calls out 
> to a Microsoft library to do so.  We don't have any platform-agnostic code 
> for producing a minidump.
> 
> I've also pinged another Googler who I know might be interested in converting 
> between minidumps and core files (the opposite direction) to see if he has 
> any additional info.  I don't think he's on lldb-dev, though, so I'll act as 
> a relay if necessary.
> 
> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev 
> mailto:lldb-dev@lists.llvm.org>> wrote:
> We can’t produce them, but you should check out the source code of google 
> breakpad / crashpad which can.
> 
> That said it’s a pretty simple format, there may be enough in our consumer 
> code that should allow you to produce them
> 
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org 
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev 
> 
> 
> 
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org 
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev 
> 
> 
> 
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Zachary Turner via lldb-dev]

Also, if the goal is to have this upstream somewhere, it would be nice to
have a tool this be a standalone tool.  This seems like something that you
shouldn't be required to start up a debugger to do, and probably doesn't
have many (or any for that matters) on the rest of LLDB.

On Wed, Jun 13, 2018 at 1:58 PM Leonard Mosescu  wrote:

> That being said, it's not exactly trivial to produce a good minidump.
>> Crashpad  has a
>> native & cross-platform minidump writer, that's what I'd start with.
>>
>
> Addendum: I realized after sending the email that if the goal is to
> convert core files -> LLDB -> minidump a lot of the complexity found in
> Crashpad can be avoided, so perhaps writing an LLDB minidump writer from
> scratch would not be too bad.
>
> On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu 
> wrote:
>
>> The minidump format is more or less documented in MSDN
>> 
>> .
>>
>> That being said, it's not exactly trivial to produce a good minidump. 
>> Crashpad
>> has a native &
>> cross-platform minidump writer, that's what I'd start with.
>>
>> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Zach's right.  On Windows, lldb can produce a minidump, but it just
>>> calls out to a Microsoft library to do so.  We don't have any
>>> platform-agnostic code for producing a minidump.
>>>
>>> I've also pinged another Googler who I know might be interested in
>>> converting between minidumps and core files (the opposite direction) to see
>>> if he has any additional info.  I don't think he's on lldb-dev, though, so
>>> I'll act as a relay if necessary.
>>>
>>> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 We can’t produce them, but you should check out the source code of
 google breakpad / crashpad which can.

 That said it’s a pretty simple format, there may be enough in our
 consumer code that should allow you to produce them


 ___
 lldb-dev mailing list
 lldb-dev@lists.llvm.org
 http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


>>>
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>>
>>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] DWARF v5 - compiler work update

[via lldb-dev]

At EuroLLVM's Debug Info BoF, I made a rash statement about being done
with DWARF v5 "next week." It has been an impressively long week...
and it's not over yet.  But given Pavel Labath's awesome news about
accelerator tables, it seemed like a good moment to review the rest of
the DWARF v5 work.

This really falls into three broad categories:
- the bare minimum needed to be conformant with v5;
- making some optional features conformant with v5;
- other things that are pretty useful or beneficial;
- everything else.

Because of unexpected complexities and other factors, the DWARF v5
work has generally been taking longer than expected. I am still
hopeful to get to the "minimally conformant" stage before we branch
LLVM 7.0 on the first of August.  Some other useful/beneficial things
have already gone in, and that's also really positive.  But there is
quite a bit left to go.

I apologize in advance for not crediting people who did the work,
or (just as important) all the reviewing!  All efforts are greatly 
appreciated!


Minimally Conformant Features
=
By this I mean: It is possible to say `-gdwarf-5` and what comes out
will conform to the DWARF v5 specification.  It does not mean that
all optional debug-info features will be conforming, but you can get
a set of .debug_* sections that all claim to be (and actually are)
compliant with DWARF v5.  This work is primarily about data layout
and in some cases, section names; if you ignore the versioning, 
it's all pretty much NFC.

1. Compile Unit headers.  DONE.
   Compile units have v5 header formats.  Nearly all the content
   of the debugging information entries (DIEs) within a compile
   unit remain unchanged, it's just the header that got tweaked.

2. Line Table header.  DONE.
   As with compile units, the header layout follows v5.

3. Range lists.  IN PROGRESS.
   DWARF v5 changes the format and section name (.debug_rnglists).

4. Location lists.  NOT STARTED
   DWARF v5 changes the format and section name (.debug_loclists).


Optional Features Conformant

This collects a few things that Clang/LLVM know how to do now,
but which need some fiddling to make them work as DWARF v5 wants.

1. Type units in .debug_info section. NOT STARTED.
   DWARF v4 defined type units, and gave them their own section
   (.debug_types).  In DWARF v5, type units go in .debug_info
   alongside the regular compile units.
   ** This is "not started" although I have made a run at it.
   Emitting type units in .debug_info is a snap, but getting the
   DWARF parser to understand what's going on requires a moderate
   amount of refactoring.  Shelved until the minimally conformant
   stuff is done.

2. Split DWARF.  NOT STARTED.
   Puts most debug info into a separate "DWARF object" file, to
   reduce the size of the raw data that the linker has to copy around.
   This was not actually part of DWARF v4, although both gcc and
   Clang support it.  Needs some fiddling for DWARF v5, although
   possibly no more than putting type units into .debug_info.

3. String Offsets table.  DONE.
   GNU had a prototype version of this, which Clang supported.
   The new .debug_str_offsets section is an array of offsets into
   the .debug_str section; this reduces the number of relocations
   for strings down from one per reference to one per unique string.


Other Useful or Beneficial Features
===
These help the linker (fewer relocations) or debugger (new info)
or have a goal of reducing debug-info size.

1. New "accelerator" tables. DONE.
   This is an index to speed up how the debugger finds info.

2. Line-table strings.  DONE.
   This moves the pathnames for source files out of the line-table
   header and into a separate string section, which allows the
   linker to deduplicate strings (which it already knows how to do)
   across compilation units, and so reduce the size required for line 
   tables.

3. MD5 Checksum for source files. DONE.
   Use MD5 instead of file size/modtime to allow the debugger to
   detect whether a file had changed since compilation.  Clang
   never provided the size/modtime so this is a real step forward.

4. Address Table.  NOT STARTED.
   There's a new .debug_addr section with an array for relocatable
   addresses, and a variety of ways DWARF can use elements of this
   array to avoid redundant relocations.
   ** This feature can be leveraged by the main debug info section,
   location lists, and range lists.


Everything Else
===
AFAIK only a few small bits of other DWARF v5 features have been
implemented, and as people have interest, there's quite a lot of
"filling in the corners" that could be done.  Some of these are
intended to be helpful for debugging optimized code.

1. Default location entry.
   Instead of sequentially listing all the places a variable might
   reside over its lifetime, specify a default location (e.g., its
   stack slot) and then fill in other 

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Leonard Mosescu via lldb-dev]

>
> That being said, it's not exactly trivial to produce a good minidump.
> Crashpad  has a
> native & cross-platform minidump writer, that's what I'd start with.
>

Addendum: I realized after sending the email that if the goal is to convert
core files -> LLDB -> minidump a lot of the complexity found in Crashpad
can be avoided, so perhaps writing an LLDB minidump writer from scratch
would not be too bad.

On Wed, Jun 13, 2018 at 1:50 PM, Leonard Mosescu  wrote:

> The minidump format is more or less documented in MSDN
> 
> .
>
> That being said, it's not exactly trivial to produce a good minidump. Crashpad
> has a native &
> cross-platform minidump writer, that's what I'd start with.
>
> On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Zach's right.  On Windows, lldb can produce a minidump, but it just calls
>> out to a Microsoft library to do so.  We don't have any platform-agnostic
>> code for producing a minidump.
>>
>> I've also pinged another Googler who I know might be interested in
>> converting between minidumps and core files (the opposite direction) to see
>> if he has any additional info.  I don't think he's on lldb-dev, though, so
>> I'll act as a relay if necessary.
>>
>> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> We can’t produce them, but you should check out the source code of
>>> google breakpad / crashpad which can.
>>>
>>> That said it’s a pretty simple format, there may be enough in our
>>> consumer code that should allow you to produce them
>>>
>>>
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>>
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Leonard Mosescu via lldb-dev]

The minidump format is more or less documented in MSDN

.

That being said, it's not exactly trivial to produce a good minidump. Crashpad
has a native &
cross-platform minidump writer, that's what I'd start with.

On Wed, Jun 13, 2018 at 1:38 PM, Adrian McCarthy via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Zach's right.  On Windows, lldb can produce a minidump, but it just calls
> out to a Microsoft library to do so.  We don't have any platform-agnostic
> code for producing a minidump.
>
> I've also pinged another Googler who I know might be interested in
> converting between minidumps and core files (the opposite direction) to see
> if he has any additional info.  I don't think he's on lldb-dev, though, so
> I'll act as a relay if necessary.
>
> On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> We can’t produce them, but you should check out the source code of google
>> breakpad / crashpad which can.
>>
>> That said it’s a pretty simple format, there may be enough in our
>> consumer code that should allow you to produce them
>>
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Adrian McCarthy via lldb-dev]

Zach's right.  On Windows, lldb can produce a minidump, but it just calls
out to a Microsoft library to do so.  We don't have any platform-agnostic
code for producing a minidump.

I've also pinged another Googler who I know might be interested in
converting between minidumps and core files (the opposite direction) to see
if he has any additional info.  I don't think he's on lldb-dev, though, so
I'll act as a relay if necessary.

On Wed, Jun 13, 2018 at 12:07 PM, Zachary Turner via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> We can’t produce them, but you should check out the source code of google
> breakpad / crashpad which can.
>
> That said it’s a pretty simple format, there may be enough in our consumer
> code that should allow you to produce them
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Zachary Turner via lldb-dev]

We can’t produce them, but you should check out the source code of google
breakpad / crashpad which can.

That said it’s a pretty simple format, there may be enough in our consumer
code that should allow you to produce them
On Wed, Jun 13, 2018 at 11:47 AM Greg Clayton  wrote:

> I am interested in being able to load a core file on linux machines and
> save out a windows minidump file with enough info for backtraces and maybe
> some variables. I know we can ingest minidump files in LLDB, but can we
> produce them? I was planning on writing a python module that adds a new
> save_minidump command, but wanted to check if anyone has anything like this
> already.
>
> Also any URLs for the minidump file format would be appreciated.
>
> Greg Clayton
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] Do we have any infrastructure for creating mini dump files from a loaded process or from a core file?

[Greg Clayton via lldb-dev]

I am interested in being able to load a core file on linux machines and save 
out a windows minidump file with enough info for backtraces and maybe some 
variables. I know we can ingest minidump files in LLDB, but can we produce 
them? I was planning on writing a python module that adds a new save_minidump 
command, but wanted to check if anyone has anything like this already.

Also any URLs for the minidump file format would be appreciated.

Greg Clayton

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm

[Greg Clayton via lldb-dev]

> On Jun 13, 2018, at 11:18 AM, Jonas Devlieghere via lldb-dev 
>  wrote:
> 
> Hi Pavel,
> 
>> On Jun 13, 2018, at 6:56 AM, Pavel Labath > > wrote:
>> 
>> Hello again,
>> 
>> It's been nearly six months since my first email, so it's a good time
>> to recap what has been done here so far. I am happy to report that
>> stages 1-3 (i.e. producer/consumer in llvm and integration with lldb)
>> of my original plan are now complete with one caveat.
> 
> Awesome!
> 
>> The caveat is that the .debug_names section is presently not a full
>> drop-in replacement for the .apple_*** sections. The reason for that
>> is that there is no equivalent to the .apple_objc section (which links
>> an objc class/category name  to all of its methods). I did not
>> implement that, because I do not see a way to embed that kind of
>> information to this section without some sort of an extension. Given
>> that this was not required for my use case, I felt it would be best to
>> leave this to the people working on objc support (*looks at Jonas*) to
>> work out the details of how to represent that.
> 
> Definitely :-) I plan to start working on this (+ dsymutil support) very 
> soon. 

It would be great to add "dsymutil --update" support so it can handle both 
mach-o files and ELF files. What this does is update the accelerator tables of 
any files that are specified. That way the accelerator tables can be added to 
any object file that contains DWARF after the fact.

As David Blaike suggested, maybe we can emit the .apple_objc section as a work 
around for now. 

> 
>> Nonetheless, I believe that the emitted .debug_names section contains
>> all the data that is required by the standard, and it is sufficient to
>> pass all tests in the lldb integration test suite on linux (this
>> doesn't include objc tests).
> 
> How did you (or do you plan to) add this (and DWARF5 in general) in the lldb 
> test suite? It sounds like this might require a new dimension in the test 
> matrix if we want to test all the variants with both Apple and DWARF style 
> accelerator tables. 
> 
>> Simple benchmarks also show a large
>> performance improvement.I have some numbers to illustrate that
>> (measurements taken by using a release build of lldb to debug a debug
>> build of clang, clang was built with -mllvm -accel-tables=Dwarf to
>> enable the accelerator generation, usage of the tables was controlled
>> by a setting in lldb):
>> - setting a breakpoint on a non-existing function without the use of
>> accelerator tables:
>> real0m5.554s
>> user0m43.764s
>> sys 0m6.748s
>> (The majority of this time is spend on building a debug info index,
>> which is a one-shot thing. subsequent breakpoints would be fast)
>> 
>> - setting a breakpoint on a non-existing function with accelerator tables:
>> real0m3.517s
>> user0m3.136s
>> sys 0m0.376s
>> (With the index already present, we are able to quickly determine that
>> there is no match and finish)
>> 
>> - setting a breakpoint on all "dump" functions without the use of
>> accelerator tables:
>> real0m21.544s
>> user0m59.588s
>> sys 0m6.796s
>> (Apart from building the index, now we must also parse a bunch of
>> compile units and line tables to resolve the breakpoint locations)
>> 
>> - setting a breakpoint on all "dump" functions with accelerator tables:
>> real0m23.644s
>> user0m22.692s
>> sys 0m0.948s
>> (Here we see that this extra work is actually the bottleneck now.
>> Preliminary analysis shows that majority of this time is spend
>> inserting line table entries into the middle of a vector, which means
>> it should be possible to fix this with a smarter implementation).
>> 
>> As far as object file sizes go, in the resulting clang binary (2.3GB),
>> the new .debug_names section takes up about 160MB (7%), which isn't
>> negligible, but considering that it supersedes the
>> .debug_pubnames/.debug_pubtypes tables whose combined size is 490MB
>> (21% of the binary), switching to this table (and dropping the other
>> two) will have a positive impact on the binary size. Further
>> reductions can be made by merging the individual indexes into one
>> large index as a part of the link step (which will also increase
>> debugger speed), but it's hard to quantify the exact impact of that.
>> 
>> With all of this in mind, I'd like to encourage you to give the new
>> tables a try. All you need to do is pass -mllvm -accel-tables=Dwarf to
>> clang while building your project. lldb should use the generated
>> tables automatically. I'm particularly interested in the interop
>> scenario. I've checked that readelf is able to make sense of the
>> generated tables, but if you have any other producer/consumer of these
>> tables which is independent of llvm, I'd like to know whether we are
>> compatible with it.
> 
> I know of one internal consumer but it ignores the accelerator tables so I 
> don’t expect any issues there.
> 
>> I'd also like to make the new 

Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm

[Jonas Devlieghere via lldb-dev]

Hi Pavel,

> On Jun 13, 2018, at 6:56 AM, Pavel Labath  wrote:
> 
> Hello again,
> 
> It's been nearly six months since my first email, so it's a good time
> to recap what has been done here so far. I am happy to report that
> stages 1-3 (i.e. producer/consumer in llvm and integration with lldb)
> of my original plan are now complete with one caveat.

Awesome!

> The caveat is that the .debug_names section is presently not a full
> drop-in replacement for the .apple_*** sections. The reason for that
> is that there is no equivalent to the .apple_objc section (which links
> an objc class/category name  to all of its methods). I did not
> implement that, because I do not see a way to embed that kind of
> information to this section without some sort of an extension. Given
> that this was not required for my use case, I felt it would be best to
> leave this to the people working on objc support (*looks at Jonas*) to
> work out the details of how to represent that.

Definitely :-) I plan to start working on this (+ dsymutil support) very soon. 

> Nonetheless, I believe that the emitted .debug_names section contains
> all the data that is required by the standard, and it is sufficient to
> pass all tests in the lldb integration test suite on linux (this
> doesn't include objc tests).

How did you (or do you plan to) add this (and DWARF5 in general) in the lldb 
test suite? It sounds like this might require a new dimension in the test 
matrix if we want to test all the variants with both Apple and DWARF style 
accelerator tables. 

> Simple benchmarks also show a large
> performance improvement.I have some numbers to illustrate that
> (measurements taken by using a release build of lldb to debug a debug
> build of clang, clang was built with -mllvm -accel-tables=Dwarf to
> enable the accelerator generation, usage of the tables was controlled
> by a setting in lldb):
> - setting a breakpoint on a non-existing function without the use of
> accelerator tables:
> real0m5.554s
> user0m43.764s
> sys 0m6.748s
> (The majority of this time is spend on building a debug info index,
> which is a one-shot thing. subsequent breakpoints would be fast)
> 
> - setting a breakpoint on a non-existing function with accelerator tables:
> real0m3.517s
> user0m3.136s
> sys 0m0.376s
> (With the index already present, we are able to quickly determine that
> there is no match and finish)
> 
> - setting a breakpoint on all "dump" functions without the use of
> accelerator tables:
> real0m21.544s
> user0m59.588s
> sys 0m6.796s
> (Apart from building the index, now we must also parse a bunch of
> compile units and line tables to resolve the breakpoint locations)
> 
> - setting a breakpoint on all "dump" functions with accelerator tables:
> real0m23.644s
> user0m22.692s
> sys 0m0.948s
> (Here we see that this extra work is actually the bottleneck now.
> Preliminary analysis shows that majority of this time is spend
> inserting line table entries into the middle of a vector, which means
> it should be possible to fix this with a smarter implementation).
> 
> As far as object file sizes go, in the resulting clang binary (2.3GB),
> the new .debug_names section takes up about 160MB (7%), which isn't
> negligible, but considering that it supersedes the
> .debug_pubnames/.debug_pubtypes tables whose combined size is 490MB
> (21% of the binary), switching to this table (and dropping the other
> two) will have a positive impact on the binary size. Further
> reductions can be made by merging the individual indexes into one
> large index as a part of the link step (which will also increase
> debugger speed), but it's hard to quantify the exact impact of that.
> 
> With all of this in mind, I'd like to encourage you to give the new
> tables a try. All you need to do is pass -mllvm -accel-tables=Dwarf to
> clang while building your project. lldb should use the generated
> tables automatically. I'm particularly interested in the interop
> scenario. I've checked that readelf is able to make sense of the
> generated tables, but if you have any other producer/consumer of these
> tables which is independent of llvm, I'd like to know whether we are
> compatible with it.

I know of one internal consumer but it ignores the accelerator tables so I 
don’t expect any issues there.

> I'd also like to make the new functionality more easily accessible to
> users. I am not sure what our policy here is, but I was thinking of
> either including this functionality in -glldb (on non-apple targets);
> or by adding a separate -g flag for it (-gdebug-names-section?), with
> the goal of eventual inclusion into -glldb. I exclude apple targets
> because: a) they already have a thing that works and the lack of
> .apple_objc would be a pessimization; b) the different debug info
> distribution model means it requires more testing and code (dsymutil).

Changing the default on non-Apple targets sounds good. Once we have Objective-C 

Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm

[via lldb-dev]

> -Original Message-
> From: Pavel Labath [mailto:lab...@google.com]
> Sent: Wednesday, June 13, 2018 9:57 AM
> To: jan.kratoch...@redhat.com; llvm-...@lists.llvm.org;
> jdevliegh...@apple.com; lldb-dev@lists.llvm.org; dblai...@gmail.com;
> Robinson, Paul; apra...@apple.com; echri...@google.com;
> vlesc...@accesssoftek.com; fr...@apple.com
> Subject: Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm
> 
> Hello again,
> 
> It's been nearly six months since my first email, so it's a good time
> to recap what has been done here so far. I am happy to report that
> stages 1-3 (i.e. producer/consumer in llvm and integration with lldb)
> of my original plan are now complete with one caveat.
> 
> The caveat is that the .debug_names section is presently not a full
> drop-in replacement for the .apple_*** sections. The reason for that
> is that there is no equivalent to the .apple_objc section (which links
> an objc class/category name  to all of its methods). I did not
> implement that, because I do not see a way to embed that kind of
> information to this section without some sort of an extension. Given
> that this was not required for my use case, I felt it would be best to
> leave this to the people working on objc support (*looks at Jonas*) to
> work out the details of how to represent that.
> 
> Nonetheless, I believe that the emitted .debug_names section contains
> all the data that is required by the standard, and it is sufficient to
> pass all tests in the lldb integration test suite on linux (this
> doesn't include objc tests). Simple benchmarks also show a large
> performance improvement.I have some numbers to illustrate that
> (measurements taken by using a release build of lldb to debug a debug
> build of clang, clang was built with -mllvm -accel-tables=Dwarf to
> enable the accelerator generation, usage of the tables was controlled
> by a setting in lldb):
> - setting a breakpoint on a non-existing function without the use of
> accelerator tables:
> real0m5.554s
> user0m43.764s
> sys 0m6.748s
> (The majority of this time is spend on building a debug info index,
> which is a one-shot thing. subsequent breakpoints would be fast)
> 
> - setting a breakpoint on a non-existing function with accelerator tables:
> real0m3.517s
> user0m3.136s
> sys 0m0.376s
> (With the index already present, we are able to quickly determine that
> there is no match and finish)
> 
> - setting a breakpoint on all "dump" functions without the use of
> accelerator tables:
> real0m21.544s
> user0m59.588s
> sys 0m6.796s
> (Apart from building the index, now we must also parse a bunch of
> compile units and line tables to resolve the breakpoint locations)
> 
> - setting a breakpoint on all "dump" functions with accelerator tables:
> real0m23.644s
> user0m22.692s
> sys 0m0.948s
> (Here we see that this extra work is actually the bottleneck now.
> Preliminary analysis shows that majority of this time is spend
> inserting line table entries into the middle of a vector, which means
> it should be possible to fix this with a smarter implementation).
> 
> As far as object file sizes go, in the resulting clang binary (2.3GB),
> the new .debug_names section takes up about 160MB (7%), which isn't
> negligible, but considering that it supersedes the
> .debug_pubnames/.debug_pubtypes tables whose combined size is 490MB
> (21% of the binary), switching to this table (and dropping the other
> two) will have a positive impact on the binary size. Further
> reductions can be made by merging the individual indexes into one
> large index as a part of the link step (which will also increase
> debugger speed), but it's hard to quantify the exact impact of that.

Nice.

> 
> With all of this in mind, I'd like to encourage you to give the new
> tables a try. All you need to do is pass -mllvm -accel-tables=Dwarf to
> clang while building your project. lldb should use the generated
> tables automatically. I'm particularly interested in the interop
> scenario. I've checked that readelf is able to make sense of the
> generated tables, but if you have any other producer/consumer of these
> tables which is independent of llvm, I'd like to know whether we are
> compatible with it.

Have you tried dwarfdump (David Anderson's dumper)?  If not I can
probably get somebody to experiment with it on our side.

> 
> I'd also like to make the new functionality more easily accessible to
> users. I am not sure what our policy here is, but I was thinking of
> either including this functionality in -glldb (on non-apple targets);
> or by adding a separate -g flag for it (-gdebug-names-section?), with
> the goal of eventual inclusion into -glldb. I exclude apple targets
> because: a) they already have a thing that works and the lack of
> .apple_objc would be a pessimization; b) the different debug info
> distribution model means it requires more testing and code (dsymutil).
> For other targets this should 

Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm

[David Blaikie via lldb-dev]

Nice! Thanks for the update:

re: ObjC: Perhaps debug_names and .apple_objc could be emitted at the same
time to address that issue at least in the short term?

As for size impact, have you tested this with fission and compressed debug
info enabled? (both in terms of whether debug_names is as compressible as
the pubnames/pubtypes, and whether it's as efficient for the debugger when
it is compressed? (I imagine the decompression might be expensive - maybe
it's worth keeping it decompressed, but then the relative cost may be a
fair bit higher))

On Wed, Jun 13, 2018 at 6:56 AM Pavel Labath  wrote:

> Hello again,
>
> It's been nearly six months since my first email, so it's a good time
> to recap what has been done here so far. I am happy to report that
> stages 1-3 (i.e. producer/consumer in llvm and integration with lldb)
> of my original plan are now complete with one caveat.
>
> The caveat is that the .debug_names section is presently not a full
> drop-in replacement for the .apple_*** sections. The reason for that
> is that there is no equivalent to the .apple_objc section (which links
> an objc class/category name  to all of its methods). I did not
> implement that, because I do not see a way to embed that kind of
> information to this section without some sort of an extension. Given
> that this was not required for my use case, I felt it would be best to
> leave this to the people working on objc support (*looks at Jonas*) to
> work out the details of how to represent that.
>
> Nonetheless, I believe that the emitted .debug_names section contains
> all the data that is required by the standard, and it is sufficient to
> pass all tests in the lldb integration test suite on linux (this
> doesn't include objc tests). Simple benchmarks also show a large
> performance improvement.I have some numbers to illustrate that
> (measurements taken by using a release build of lldb to debug a debug
> build of clang, clang was built with -mllvm -accel-tables=Dwarf to
> enable the accelerator generation, usage of the tables was controlled
> by a setting in lldb):
> - setting a breakpoint on a non-existing function without the use of
> accelerator tables:
> real0m5.554s
> user0m43.764s
> sys 0m6.748s
> (The majority of this time is spend on building a debug info index,
> which is a one-shot thing. subsequent breakpoints would be fast)
>
> - setting a breakpoint on a non-existing function with accelerator tables:
> real0m3.517s
> user0m3.136s
> sys 0m0.376s
> (With the index already present, we are able to quickly determine that
> there is no match and finish)
>
> - setting a breakpoint on all "dump" functions without the use of
> accelerator tables:
> real0m21.544s
> user0m59.588s
> sys 0m6.796s
> (Apart from building the index, now we must also parse a bunch of
> compile units and line tables to resolve the breakpoint locations)
>
> - setting a breakpoint on all "dump" functions with accelerator tables:
> real0m23.644s
> user0m22.692s
> sys 0m0.948s
> (Here we see that this extra work is actually the bottleneck now.
> Preliminary analysis shows that majority of this time is spend
> inserting line table entries into the middle of a vector, which means
> it should be possible to fix this with a smarter implementation).
>
> As far as object file sizes go, in the resulting clang binary (2.3GB),
> the new .debug_names section takes up about 160MB (7%), which isn't
> negligible, but considering that it supersedes the
> .debug_pubnames/.debug_pubtypes tables whose combined size is 490MB
> (21% of the binary), switching to this table (and dropping the other
> two) will have a positive impact on the binary size. Further
> reductions can be made by merging the individual indexes into one
> large index as a part of the link step (which will also increase
> debugger speed), but it's hard to quantify the exact impact of that.
>
> With all of this in mind, I'd like to encourage you to give the new
> tables a try. All you need to do is pass -mllvm -accel-tables=Dwarf to
> clang while building your project. lldb should use the generated
> tables automatically. I'm particularly interested in the interop
> scenario. I've checked that readelf is able to make sense of the
> generated tables, but if you have any other producer/consumer of these
> tables which is independent of llvm, I'd like to know whether we are
> compatible with it.
>
> I'd also like to make the new functionality more easily accessible to
> users. I am not sure what our policy here is, but I was thinking of
> either including this functionality in -glldb (on non-apple targets);
> or by adding a separate -g flag for it (-gdebug-names-section?), with
> the goal of eventual inclusion into -glldb. I exclude apple targets
> because: a) they already have a thing that works and the lack of
> .apple_objc would be a pessimization; b) the different debug info
> distribution model means it requires more testing and code (dsymutil).

Re: [lldb-dev] Adding DWARF5 accelerator table support to llvm

[Pavel Labath via lldb-dev]

Hello again,

It's been nearly six months since my first email, so it's a good time
to recap what has been done here so far. I am happy to report that
stages 1-3 (i.e. producer/consumer in llvm and integration with lldb)
of my original plan are now complete with one caveat.

The caveat is that the .debug_names section is presently not a full
drop-in replacement for the .apple_*** sections. The reason for that
is that there is no equivalent to the .apple_objc section (which links
an objc class/category name  to all of its methods). I did not
implement that, because I do not see a way to embed that kind of
information to this section without some sort of an extension. Given
that this was not required for my use case, I felt it would be best to
leave this to the people working on objc support (*looks at Jonas*) to
work out the details of how to represent that.

Nonetheless, I believe that the emitted .debug_names section contains
all the data that is required by the standard, and it is sufficient to
pass all tests in the lldb integration test suite on linux (this
doesn't include objc tests). Simple benchmarks also show a large
performance improvement.I have some numbers to illustrate that
(measurements taken by using a release build of lldb to debug a debug
build of clang, clang was built with -mllvm -accel-tables=Dwarf to
enable the accelerator generation, usage of the tables was controlled
by a setting in lldb):
- setting a breakpoint on a non-existing function without the use of
accelerator tables:
real0m5.554s
user0m43.764s
sys 0m6.748s
(The majority of this time is spend on building a debug info index,
which is a one-shot thing. subsequent breakpoints would be fast)

- setting a breakpoint on a non-existing function with accelerator tables:
real0m3.517s
user0m3.136s
sys 0m0.376s
(With the index already present, we are able to quickly determine that
there is no match and finish)

- setting a breakpoint on all "dump" functions without the use of
accelerator tables:
real0m21.544s
user0m59.588s
sys 0m6.796s
(Apart from building the index, now we must also parse a bunch of
compile units and line tables to resolve the breakpoint locations)

- setting a breakpoint on all "dump" functions with accelerator tables:
real0m23.644s
user0m22.692s
sys 0m0.948s
(Here we see that this extra work is actually the bottleneck now.
Preliminary analysis shows that majority of this time is spend
inserting line table entries into the middle of a vector, which means
it should be possible to fix this with a smarter implementation).

As far as object file sizes go, in the resulting clang binary (2.3GB),
the new .debug_names section takes up about 160MB (7%), which isn't
negligible, but considering that it supersedes the
.debug_pubnames/.debug_pubtypes tables whose combined size is 490MB
(21% of the binary), switching to this table (and dropping the other
two) will have a positive impact on the binary size. Further
reductions can be made by merging the individual indexes into one
large index as a part of the link step (which will also increase
debugger speed), but it's hard to quantify the exact impact of that.

With all of this in mind, I'd like to encourage you to give the new
tables a try. All you need to do is pass -mllvm -accel-tables=Dwarf to
clang while building your project. lldb should use the generated
tables automatically. I'm particularly interested in the interop
scenario. I've checked that readelf is able to make sense of the
generated tables, but if you have any other producer/consumer of these
tables which is independent of llvm, I'd like to know whether we are
compatible with it.

I'd also like to make the new functionality more easily accessible to
users. I am not sure what our policy here is, but I was thinking of
either including this functionality in -glldb (on non-apple targets);
or by adding a separate -g flag for it (-gdebug-names-section?), with
the goal of eventual inclusion into -glldb. I exclude apple targets
because: a) they already have a thing that works and the lack of
.apple_objc would be a pessimization; b) the different debug info
distribution model means it requires more testing and code (dsymutil).
For other targets this should bring a big performance improvement when
debugging with lldb. The lack of .apple_objc will mean lldb will have
to either index the objc compile units manually, or implement a more
complicated lookup using other information in the section. However,
Objective C is not that widespread outside of apple platforms, so the
impact of this should be minimal.

What do you think?

regards,
pavel
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev