Re: [Dwarf-discuss] [DWARF5] .debug_names + fdebug-types-sections

[Greg Clayton via Dwarf-discuss]

> On Oct 16, 2023, at 9:12 AM, David Blaikie via Dwarf-discuss 
>  wrote:
> 
> 
> 
> On Mon, Oct 16, 2023 at 8:57 AM Alexander Yermolovich  > wrote:
>> For background llvm discussion on how to implement it: 
>> https://discourse.llvm.org/t/debuginfo-dwarfv5-lld-debug-names-with-fdebug-type-sections/73445
>> 
>> Thanks for explaining the issue, and proposing spec change. 
>> The question I have. Is non-bit identical TUs with the same hash a 
>> fundamental issue that needs to be addressed somehow in the next version of 
>> the spec? If we could have such guarantee that should simplify things quite 
>> a bit. The linker can just follow the same path as for functions. Compiler 
>> can generate symbol name unique for the type unit hash. So, when linker 
>> comdats TU sections entries in TU list will point to correct address and no 
>> special logic is needed for tombstone. I guess there is a hashing mechanism 
>> in DWARF spec, but LLVM is not using it. Should we go back to it, is it 
>> enough?
> 
> The hashing mechanism in the spec doesn't guarantee bit-identicality, I 
> believe. It's structural equivalence (eg: if you produce the main type DIE 
> followed by an int DIE that the main type needs, or you emit the int DIE 
> first, followed by the main type DIE - these hash to the same value (because 
> you start from the type DIE and hash outwards/to what it can reach, and has 
> structural equivalence - int is int, no matter what offset it's at)) not bit 
> identical. For a bunch of reasons this is preferable.
> 
> (yes, clang takes this further and hashes based on the C++ ODR - which is 
> off-spec, but workable in our experience)
> 
> I was thinking another direction we could go is that, I think, the only 
> things in a type unit that can be referenced is the type (I think?) then 
> perhaps we could modify how types defined in type units are referenced.
> 
> If only the type can be referenced in a type unit, we could emit a 
> .debug_names entry without a DW_IDX_die_offset - just the DW_IDX_type_unit - 
> and the consumer can use the header of the type unit to find the exact type 
> unit DIE.
> 
> Are there any other things that could be referenced within a type unit?

LLDB will want access to any types contained within the type units. Many 
classes contain type definitions within the class itself. Any CUs wanting to 
access these types of course can't, so they have the duplicate the entire 
declaration context for the type (containing namespaces and the class itself 
with a DW_AT_declaration(true) attribute) then create the copy of the contained 
type if it is simple. 

For example every STL class defines all sorts of "iterator", "const_iterator", 
"reverse_iterator", "size_type", "pointer_type", "reference_type", etc inside 
of the class. If no variables from a CU references these types, then we won't 
have access to them if we only add the main type unit type to the .debug_names 
table. 

So it is correct that the only thing that can be referenced in a type unit is 
the main type itself from a DWARF perspective, but it would be a shame if no 
debugger clients can use any of the extra types in the type units unless they 
are directly referenced (and duplicated) in a CU. 

LLDB notes which CUs and TUs have an entry in the .debug_names table and it 
will manually index any that didn't have entries. If the .debug_names tables 
end up only emitting the main type unit type, we will need to manually index 
each TU to make sure we have access to contained types. 

So I would vote to completely index each TU if possible.

>  
>> 
>> Alex
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> From: David Blaikie mailto:dblai...@gmail.com>>
>> Sent: Monday, September 25, 2023 9:02 AM
>> To: Alexander Yermolovich mailto:ayerm...@meta.com>>
>> Cc: dwarf-discuss@lists.dwarfstd.org 
>>  > >
>> Subject: Re: [Dwarf-discuss] [DWARF5] .debug_names + fdebug-types-sections
>>  
>> 
>> 
>> On Fri, Sep 15, 2023 at 2:45 PM Alexander Yermolovich via
>> Dwarf-discuss > > wrote:
>> >
>> > Hello
>> >
>> > I am trying to enable debug names acceleration table with 
>> > fdebug-types-sections in LLVM. One part I am not sure about is the local 
>> > TU list. It contains an offset into .debug_info section. All the entries 
>> > have an index entry that points to the local TU list. DIEs within entry 
>> > offsets are relative to the TU entry.
>> >
>> > Linker de-duplicates Type Units using COMDAT. So, the final result will 
>> > have less type units. As the result Local Type Unit List will be invalid, 
>> > and all the Entries that point to that TU will not be valid either. Even 
>> > if we Linker is modified so that somehow when it de-duplicates type 
>> > sections Local Type Units will get the right offset, that still leaves all 
>> > the duplicate entries.
>> > Am I missing something in that linker, 

Re: [Dwarf-discuss] Ranges for DW_TAG_namespace

[Greg Clayton via Dwarf-discuss]

When searching for addresses we first see if the compile unit's DW_AT_ranges 
(or low/high pc) attribute contains the address we are looking for. Any CU that 
doesn't contain the address doesn't need to have its child DIEs parsed, just 
the top level DW_TAG_compile_unit DIE. Then we iterate over all the DIEs always 
descending into all of the children looking for DW_TAG_subprogram entries that 
contain the address we are looking for. So if we see a DW_TAG_namespace we just 
call recursively to parse its children. 

> On Sep 14, 2023, at 3:50 PM, rifkin.jer--- via Dwarf-discuss 
>  wrote:
> 
> Hello,
> What is the reasoning for not including range information on DW_TAG_namespace 
> DIEs? Is there a canonical way to check if a DW_TAG_namespace DIE contains a 
> given address?
>  
> Thank you,
> Jeremy
> -- 
> Dwarf-discuss mailing list
> Dwarf-discuss@lists.dwarfstd.org 
> https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss

-- 
Dwarf-discuss mailing list
Dwarf-discuss@lists.dwarfstd.org
https://lists.dwarfstd.org/mailman/listinfo/dwarf-discuss

Re: [Dwarf-Discuss] debug_aranges use and overhead

[Greg Clayton via Dwarf-Discuss]

As long as there is a DW_AT_ranges on the CU the is complete, that is good 
enough for LLDB. No one seems to consistently emit .debug_aranges these days so 
we definitely don't rely on it.

Greg

> On Jun 14, 2022, at 1:10 PM, David Blaikie via Dwarf-Discuss 
>  wrote:
> 
> Given the discussion previously in this thread - does anyone have
> particular objections to removing .debug_aranges? (in favor of/perhaps
> with specific wording that /requires/ CU level ranges to be specified
> (ie: it's not acceptable to have a subprogram with non-empty range in
> a CU which doesn't cover that range) - so a consumer can look at the
> CU and, if it has no ranges, conclude that it has no addresses covered
> and skip the CU for address-related computation purposes)
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] CU-local types

[Greg Clayton via Dwarf-Discuss]

Template types are emitted for C++ in DWARF as specialized instances only, 
there is no generic definition of the type. One of the issues that impedes LLDB 
from functioning correctly in the expression parser for C++ with templates is 
how the accelerator table entries are emitted. If you have a 
"std::vector", the only accelerator table entry, if there is one at all, 
contains this full name ("std::vector"). LLDB uses clang as the expression 
parser, so if someone types this into their expression, the end up with one 
entry that points this full name to the matching entry. LLDB acts and a 
precompiled header for clang when evaluating expressions, so first we will try 
and find "std" in the accelerator tables and we will find the namespace, then 
clang will ask for the name "vector" to be found within the "std" namespace and 
we will never find it since the name of a class is always fully specified. With 
functions we end up with the base name of the function as the DW_AT_name and we 
have the DW_AT_linkage_name for the mangled name, both of which will appear in 
the accelerator tables. But for classes we don't have the base name of the 
class at the DW_AT_name, so we never will find this template class unless we 
again ignore all accelerator tables and generate them ourselves each time we 
debug. Granted this can be fixed in LLDB at great cost of having to parse every 
DIE in all units each time we start debugging so we can make an index that 
works for these lookups, but that cost is prohibitive.

> On Jun 14, 2022, at 1:04 PM, David Blaikie via Dwarf-Discuss 
>  wrote:
> 
> On Wed, May 18, 2022 at 9:53 AM David Blaikie  > wrote:
>> 
>> On Wed, May 18, 2022 at 4:16 AM Robinson, Paul  
>> wrote:
>>> 
 Looks like gdb and lldb both have issues with C++ local types (either
 types defined in anonymous namespaces, or otherwise localized - eg: a
 non-local template with a local type or variable in one of its
 parameters).
 ...
 So... what could/should we do about this?
>>> 
>>> Do you have a strong argument for why these are not debugger bugs?
>>> It sounds to me like gdb/lldb are handling anonymous namespaces
>>> incorrectly, in effect treating their contents as global rather than
>>> CU-local.
>> 
>> Oh, right, sorry forgot to include the trickier examples.
>> 
>> So for a non-template this isn't especially burdensome (check for an
>> anonymous namespace in the parent scopes - it's language specific, but
>> not a ton of weird work to do) - for templates it's a bit harder (you
>> have to check every template parameter, and potentially arbitrarily
>> deep - eg: you might have a template parameter that's a function type
>> and one of the parameters is a pointer type and the type the pointer
>> points to is local - thus the template is local. That seems a bit more
>> of a stretch to ask the consumer to do totally reliably) - but the
>> worst case, that at the moment there's potentially no way to
>> disambiguate whether the type is local or not: A non-type template
>> parameter that points to a local variable.
>> 
>> static int x = 3;
>> template struct t1 { };
>> t1<> v;
>> 
>> Currently both LLVM and GCC name this type "t1<>" and LLVM at least
>> puts a DW_AT_location on the DW_TAG_template_value_parameter which
>> points to the global variable (not the DW_TAG_variable, but to the
>> actual ELF symbol in the file) - though this choice has some negative
>> effects (causes the symbol to be "used" and linked in - which means
>> that enabling debug info can effect the behavior of the program
>> (global ctors in that file execute when they wouldn't've otherwise,
>> etc)).
>> 
>> If the location is provided, arguably the consumer could lookup the
>> symbol and check its linkage (not always accurate - LTO might've
>> internalized a variable that wasn't actually internal in the original
>> source, for instance) - but when the location is not provided there's
>> no way to know whether "t1<>" is local or not.
> 
> Ping - anyone got further ideas about how to address this issue/encode
> this information?
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org 
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org 
> 
___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] How to interpret DW_AT_artificial tag?

[Greg Clayton via Dwarf-Discuss]

> On Feb 28, 2022, at 1:11 PM, David Blaikie via Dwarf-Discuss 
>  wrote:
> 
> 
> 
> On Mon, Feb 28, 2022 at 12:55 PM Greg Clayton via Dwarf-Discuss 
> mailto:dwarf-discuss@lists.dwarfstd.org>> 
> wrote:
> 
> 
>> On Feb 28, 2022, at 5:49 AM, Ron Louzon via Dwarf-Discuss 
>> mailto:dwarf-discuss@lists.dwarfstd.org>> 
>> wrote:
>> 
>> I have an application which uses DwarfLib to extract type information from 
>> debug executable images.  I have found in the DWARF data that some structure 
>> types have a "virtual" pointer added as their first member and this 
>> pointer's DIE contains the tag DW_AT_artificial=true.  How does that pointer 
>> member affect the offsets of the members that follow it in the structure. 
> 
> This will cause all other members to be pushed out by a pointer size.
> 
>> Should this 4-byte pointer be ignored or will its size cause the other 
>> structure members to be pushed out in memory by 4 bytes?
> 
> All offsets in the DWARF should be correct for all members, including 
> artificial members and any members that follow it in memory. So yes, if there 
> is a vtable pointer added as the first member, its offset and all other 
> offsets will be correct, so there is no need to adjust anything when reading 
> this data. 
> 
> You could choose to not show this, but I find it is often easier to show this 
> information in case some compiler change in the future marks something that 
> you might want to see as artificial. For example the "this" parameter to C++ 
> methods is marked as artificial, and people do want to see the "this" in the 
> variables view. Granted that is a variable vs a member variable.
> 
> Probably the important distinction there is that "this", while artificial, is 
> named/usable in the source whereas the vtable isn't.
> 
> Either having some DWARF attribute to communicate that distinction, or a 
> workaround might be to detect that the vtable name uses a reserved identifier 
> could be good - I guess the reality is that DWARF consumers probably hardcode 
> some sort of "this is the vtable pointer" -o so maybe we should have some 
> DWARF attribute that communicates that, instead of relying on the string name 
> of the member? Not sure.

Sounds interesting, yes. Maybe a DW_AT_language_runtime attribute? This could 
let people know that the item is part of the language runtime and not 
necessarily something the user would always want to see.

> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org <mailto:Dwarf-Discuss@lists.dwarfstd.org>
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org 
> <http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org>
___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] How to interpret DW_AT_artificial tag?

[Greg Clayton via Dwarf-Discuss]

> On Feb 28, 2022, at 5:49 AM, Ron Louzon via Dwarf-Discuss 
>  wrote:
> 
> I have an application which uses DwarfLib to extract type information from 
> debug executable images.  I have found in the DWARF data that some structure 
> types have a "virtual" pointer added as their first member and this pointer's 
> DIE contains the tag DW_AT_artificial=true.  How does that pointer member 
> affect the offsets of the members that follow it in the structure. 

This will cause all other members to be pushed out by a pointer size.

> Should this 4-byte pointer be ignored or will its size cause the other 
> structure members to be pushed out in memory by 4 bytes?

All offsets in the DWARF should be correct for all members, including 
artificial members and any members that follow it in memory. So yes, if there 
is a vtable pointer added as the first member, its offset and all other offsets 
will be correct, so there is no need to adjust anything when reading this data. 

You could choose to not show this, but I find it is often easier to show this 
information in case some compiler change in the future marks something that you 
might want to see as artificial. For example the "this" parameter to C++ 
methods is marked as artificial, and people do want to see the "this" in the 
variables view. Granted that is a variable vs a member variable.



> 
> thanks,
> ron
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

[Dwarf-Discuss] Producing .debug_names and questions about this lookup format

[Greg Clayton via Dwarf-Discuss]

I am in the process if writing a tool that can add .debug_names to a file that 
contains DWARF but doesn’t have an accelerator table that is adequate for 
debuggers to use. I was trying to see some example implementations of the 
.debug_names section to see exactly what I should emit from this tool.

I have a question on the following section from the DWARF 5 specification: 
6.1.1.4.7 Abbreviations Table. It contains "Table 6.1: Index attribute 
encodings” which shows the available attributes for the abbreviations that can 
be used for each index entry:

DW_IDX_compile_unit. // Index of CU
DW_IDX_type_unit  // Index of TU
DW_IDX_die_offset // Offset of DIE within CU or TU
DW_IDX_parent // Index of name table entry for parent
DW_IDX_type_hash // Hash of type declaration


My main question is what are the best practices for how producers should emit 
an entry for a specific DIE in this lookup table. One option is to specify both 
a CU index and a relative DIE offset:

DW_IDX_compile_unit + DW_FORM_dataX form
DW_IDX_die_offset + DW_FORM_ref4

“DW_FORM_ref4” is known to be a CU relative offset.

Or we can simply emit a single attribute for the die_offset using an absolute 
DIE offset reference?

DW_IDX_die_offset + DW_FORM_ref_addr

There isn’t much written up in the DWARF spec about these attributes except 
that it seems to imply that DW_IDX_die_offset must also have a CU or TU since 
the comment says "Offset of DIE within CU or TU”

I was looking at the output of dsymutil, a smart DWARF linker maintained by 
Apple, and it can emit .debug_names with an option, but it emits the following 
abbreviation, as seen by using “llvm-dwarfdump --debug-names”:

Abbreviation 0x16 {
  Tag: DW_TAG_typedef
  DW_IDX_die_offset: DW_FORM_ref4
}

And the in the Entry for each type, dsymutil only emits a DIE offset as a 
DW_FORM_ref4, which seems to imply it is an absolute offset:

  Bucket 0 [
Name 1 {
  Hash: 0x4D51C8E0
  String: 0x10c4 "pthread_t"
  Entry @ 0x5a7 {
Abbrev: 0x16
Tag: DW_TAG_typedef
DW_IDX_die_offset: 0x2162
  }
}
Name 2 {
  Hash: 0x8FEC1B20
  String: 0x0323 "long int"
  Entry @ 0x5ad {
Abbrev: 0x24
Tag: DW_TAG_base_type
DW_IDX_die_offset: 0x0920
  }
}
  ]


Should dsymutil be emitting both a CU index and a CU relative offset, or should 
it just switch over to using a DW_FORM_ref_addr as the encoding for the DIE?

Do any current compilers support emitting .debug_names when “-gdwarf-5” is 
produced?

Any clarification or pointers to other producers would be appreciated!

Greg Clayton

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] string reduction techniques

[Greg Clayton via Dwarf-Discuss]

> On Nov 7, 2021, at 12:36 PM, Todd Allen  wrote:
> 
> Just spitballing an idea here, but would there be value in a new DW_FORM (or
> two) that referenced the names from .strtab or .dynstr, instead of .debug_str?
> It would only work if the symbols already were there, but I would expect that
> for many/most/all(?) functions defined in the compilation unit.  It does
> somewhat relegate this to being Someone Else's Problem, but given that the
> .strtab already has the problem of zillions of these huge symbols, maybe 
> that's
> not so bad?
> 
> Maybe, if that's too onerous for tools that need to manipulate .strtab, it 
> could
> reference them indirectly through a .debug_strtab_offsets section similar to
> .debug_str_offsets.

Interesting idea! One issue is if someone strips the binary, this could end up 
stripping local symbols that have mangled names that the DWARF refers to and 
cause the DW_FORM values to point to invalid offsets. 

> 
> On Tue, Nov 02, 2021 at 10:09:16AM -0700, Dwarf Discussion wrote:
>> On Mon, Nov 1, 2021 at 7:14 PM Greg Clayton via Dwarf-Discuss
>> <[1]dwarf-discuss@lists.dwarfstd.org> wrote:
>> 
>>   LLDB also uses mangled names. The clang compiler is our expression
>>   parser and it always tries to resolve symbols during compilation/JIT and
>>   it supplies mangled names when looking for functions to resolve when it
>>   JITs code up. It is nice to be able to do quick name lookups using these
>>   mangled names to find the address of the function. That being said, we
>>   could work around it. Not sure how easy that would be though as mangled
>>   names can end up demangling to the same name with some loss of
>>   information and it would be important to be able to find the right in
>>   charge or out of charge constructor when the compiler asks for a
>>   specific symbol using the mangled name. We have more uses of mangled
>>   names but most of them relate to parsing the symbol tables, so removing
>>   them from DWARF wouldn't affect those areas.
>>   I wonder if these is a way to have a DW_AT_partial_linkage_name that
>>   relies on the decl context of a DIE. Like if you have a class "foo" in
>>   the global namespace it could have a DW_AT_partial_linkage_name with the
>>   value "_Z3foo". A DW_TAG_subprogram that is a child of this "foo" class
>>inside this class could have another partial linkage name "3bari" that
>>   could be put together with the parent "_Z3foo" for a function like:
>>   Void foo::bar(int);
>>   Since many mangled names often start with the same prefix it might help
>>   reduce the string table size.
>> 
>> It's a thought - though I'm not sure how much that would really generalize
>> across different mangling schemes that use different mechanisms for
>> backreferences, etc. Or whether the return type should be included (it's
>> included for function templates in itanium mangling, for instance -
>> presumably also in MSVC mangling, but maybe some manglings include it even
>> in non-templates? I'm not sure) - since the partial linkage name for a
>> type would be context-insensitive (since it'd be attached to the type
>> rather than any use of the type) it'd be up to the consumer to fix that
>> up, eg:
>> 
>> [2]https://godbolt.org/z/TqYjeevqx
>> Itanium:
>>   f1<>(): _Z2f1IJEEvv
>>   f1(): _Z2f1IJ2t1S0_S0_EEvv
>> MSVC:
>>   f1<>(): ??$f1@$$V@@YAXXZ
>>   f1(): ??$f1@Ut1@@U1@U1@@@YAXXZ
>> I'm not sure how much less a consumer would know about mangling if it had
>> to know about how to assemble these things, insert backrefs, insert empty
>> list markers, etc - without having to know how to mangle a specific user
>> defined type or name, like "3foo" versus "@Ut1@"?
>> 
>> On Nov 1, 2021, at 6:52 PM, Daniel Berlin via Dwarf-Discuss
>> <[3]dwarf-discuss@lists.dwarfstd.org> wrote:
>> Finally, a question i know the answer to!
>> It brings us all the way back to when I was the C++ maintainer for
>> GDB, which is the most ancient of history.
>> Unfortunately, this a trip to a horrible place
>> I actually spent a lot of time trying to make it so we didn't need
>> linkage names, because, even then, they took up a *lot* of space.
>> On Mon, Nov 1, 2021 at 8:35 PM Cary Coutant via Dwarf-Discuss
>> <[4]dwarf-discuss@lists.dwarfstd.org> wrote:
>> 
>>>> I can't be sure about this exponential growth.  I don't have the
>>   data to back it
>>>> up.  But I will say, when we created DWARF64, I was skeptica

Re: [Dwarf-Discuss] string reduction techniques

[Greg Clayton via Dwarf-Discuss]

LLDB also uses mangled names. The clang compiler is our expression parser and 
it always tries to resolve symbols during compilation/JIT and it supplies 
mangled names when looking for functions to resolve when it JITs code up. It is 
nice to be able to do quick name lookups using these mangled names to find the 
address of the function. That being said, we could work around it. Not sure how 
easy that would be though as mangled names can end up demangling to the same 
name with some loss of information and it would be important to be able to find 
the right in charge or out of charge constructor when the compiler asks for a 
specific symbol using the mangled name. We have more uses of mangled names but 
most of them relate to parsing the symbol tables, so removing them from DWARF 
wouldn’t affect those areas.

I wonder if these is a way to have a DW_AT_partial_linkage_name that relies on 
the decl context of a DIE. Like if you have a class "foo" in the global 
namespace it could have a DW_AT_partial_linkage_name with the value "_Z3foo". A 
DW_TAG_subprogram that is a child of this "foo" class  inside this class could 
have another partial linkage name "3bari" that could be put together with the 
parent "_Z3foo" for a function like:

Void foo::bar(int);

Since many mangled names often start with the same prefix it might help reduce 
the string table size.


> On Nov 1, 2021, at 6:52 PM, Daniel Berlin via Dwarf-Discuss 
>  wrote:
> 
> Finally, a question i know the answer to!
> 
> It brings us all the way back to when I was the C++ maintainer for GDB, which 
> is the most ancient of history.  
> Unfortunately, this a trip to a horrible place
> I actually spent a lot of time trying to make it so we didn't need linkage 
> names, because, even then, they took up a *lot* of space.
> 
> On Mon, Nov 1, 2021 at 8:35 PM Cary Coutant via Dwarf-Discuss 
> mailto:dwarf-discuss@lists.dwarfstd.org>> 
> wrote:
> >> I can't be sure about this exponential growth.  I don't have the data to 
> >> back it
> >> up.  But I will say, when we created DWARF64, I was skeptical that it 
> >> would be
> >> needed during my career.  And yet here we are...
> >
> > Yep, still got mixed feelings about DWARF64 - partly the pieces that we're 
> > seeing with the need for some solutions for mixed DWARF32/64, etc, makes it 
> > feel like maybe it's not got a bit of "settling in" to do. And I'm still 
> > rather hopeful we might be able to reduce the overheads enough to avoid 
> > widespread use of DWARF64 - but it's not a sure thing by any means.
> 
> Agreed. I'd like to explore as many avenues as we can to eliminate the
> need for DWARF64.
> 
> 
> >> Honestly, I've never been sure why gcc generates DW_AT_linkage_name.  Our
> >> debugger almost never uses it.  (There is one use to detect "GNU indirect"
> >> functions.)  I wonder if it would be possible to avoid them if you provided
> >> enough info about the template parameters, if the debugger had its own name
> >> mangler.  I had to write one for our debugger a couple years ago, and it
> >> definitely was a persnickety beast.  But doable with enough information.  
> >> Mind
> >> you, I'm not sure there is enough information to do it perfectly with the 
> >> state
> >> of DWARF & gcc right now.
> >
> > Yeah, that was/is certainly my first pass - the way I've done the 
> > DW_AT_name one is to have a feature in clang that produces the short name 
> > "t1" but then also embeds the template argument list in the name (like 
> > this: "_STNt1|") - then llvm-dwarfdump will detect this prefix, split 
> > up the name, rebuild the original name as it would if it'd been given only 
> > the simple name ("t1") and compare it to the one from clang. Then I can run 
> > this over large programs and check everything round-trips correctly & in 
> > clang, classify any names we can't roundtrip so they get emitted in full 
> > rather than shortened.
> > We could do something similar with linkage names - since to know there's 
> > some prior art in your work there.
> >
> > I wouldn't be averse to considering what'd take to make DWARF robust enough 
> > to always roundtrip simple and linkage names in this way - I don't think 
> > it'd take a /lot/ of extra DWARF content.
> 
> Fuzzy memory here, but as I recall, GCC didn't generate linkage names
> (or only did in some very specific cases) until the LTO folks
> convinced us they needed it in order to relate profile data back to
> the source. Perhaps if we came up with a better way of doing that, we
> could eliminate the linkage names.
> 
> No, see, that's a mildly reasonable answer.
> If you go far enough back, the linkage names exist for a few reasons:
> 1. Because the debug info wasn't always good enough, and so GDB used to 
> demangle the linkage names and parse them using a hacked up C++-ish parser 
> for type info.
> 2. Even when it didn't, it decoded linkage names to detect things like 
> destructors/constructors, etc.
> 3. Because It used it to do 

Re: [Dwarf-Discuss] debug_aranges use and overhead

[Greg Clayton via Dwarf-Discuss]

> On Mar 19, 2021, at 11:22 AM, Frank Ch. Eigler  wrote:
> 
> Hi -
> 
>> We have seen GSYM files up to 20x smaller than the DWARF file, which
>> to be fair often includes all other sections (.text, .data, etc).
> 
> What information is lost?

GSYM does not contain:
- variable information (locals, parameters, globals, statics)
- type information
- macro information
- unwind information (but will soon be available)
- line entries don't contain anything but function + file + line (no source 
column, is statement, end sequence, end prologue, begin elilogue, etc).


___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] debug_aranges use and overhead

[Greg Clayton via Dwarf-Discuss]

> On Mar 19, 2021, at 9:33 AM, Samy Al Bahra via Dwarf-Discuss 
>  wrote:
> 
> Hi David,
> 
> Sorry I'm a bit late to the game. On the value of having .debug_aranges and 
> the performance impact:
> 
> Our debugger was designed for performance and does end to end lazy evaluation 
> (down to the DIE). This is quite old (excuse the formatting) but numbers are 
> here: 
> https://engineering.backtrace.io/2014-09-15-bt-lightweight-backtrace-tool/ 
>  
> , search for "Chromium".  This is something other debuggers can take 
> advantage of if they run in a non-interactive / batch mode (think bulk 
> processing of millions - billions of dumps a month) and is generally useful 
> when folks are iterating in development (fast feedback for crashes while 
> having some background indexing work going on). I'm also happy to run 
> benchmarks for you with and without .debug_aranges on top of our debugger if 
> it'll be useful.
> 
> One of the crucial optimizations we made is incremental indexing on top of 
> .debug_aranges based on PC values (+ complexities Greg mentions later in the 
> thread). In cases where we lack this, we use our own persistent cache which 
> introduces unnecessary complexity. Now I am considering going as far as 
> adding a multi-threaded indexer for cases where a persistent cache / build 
> system modifications aren't an option (work to begin in the next week or two).


For symbolicating in bulk, millions - billions of dumps a month, I would highly 
suggest you check out converting your DWARF to GSYM. This file format was 
created because of the inefficiencies of using DWARF for symbolication when 
done at a large scale on servers. The file format is designed to be mapped into 
memory and used as is. It is many times smaller than DWARF and the lookup 
speeds are orders of magnitude faster. The file format is designed to be 
mmap'ed into one or more processes as shared memory so that if your 
symbolication servers can route the symbolication requests for the same GSYM 
file to the same server then pages from that GSYM file can be hot in the file 
cache and speed up requests event further. 

Features of the GSYM file format:
- Address table is sorted for fastest lookups and doesn't store full 64 bit 
addresses, just offsets from a base address from the file header. This allows 
you to search for your address and touch as few pages as possible when doing 
the lookup. You are also only searching addresses when doing the search, no 
extra data. The offsets to the data for each address are in a separate table 
that follows the address table.
- Single uniqued file table to store source file paths for any data that is 
references within the GSYM file (like the source files for the line tables). 
This means all compile unit file tables from the DWARF, which each have their 
own file tables in each prologue of the info in .debug_line, are uniqued into a 
single file table which reduces GSYM file size. The file table also splits all 
paths up into directory + basename which allows all files in the same directory 
to shared the same string in the string table for the directory path.
- All function info is stored in one contiguous blob of data and includes:
  - full line table for the function only (no more parsing the entire line 
table of a compile unit just to get the lines for a function)
  - inline call stack information to unwind inline call stacks

The GSYM file format description is contained in the header files in the 
headerdoc. The GSYM file format is open sourced into llvm.org 
 and the code can be found:

Library code:

  llvm/include/llvm/DebugInfo/GSYM
  llvm/lib/DebugInfo/GSYM

Tool code:

  llvm/tools/llvm-gsymutil


If you build llvm-gsymutil you can convert any DWARF to GSYM using:

  $ llvm-gsymutil --convert a.out -o a.out.gsym

We have seen GSYM files up to 20x smaller than the DWARF file, which to be fair 
often includes all other sections (.text, .data, etc).

If you want to see exactly what is encoded in the GSYM file you can dump the 
entire file using:

  $ llvm-gsymutil -o a.out.gsym

This will give you a great idea of exactly what is encoded in the GSYM file as 
the dump accurately mirrors the exact contents of the file.

Then lookups can be done with a the tool:

  $ llvm-gsymutil --address 0x1000 a.out.gsym

Or by linking against the LLVM library code and doing it with a few lines of 
code:

  llvm::Expected Gsym = GsymReader::openFile(GSYMPath);
  if (!Gsym)
error(GSYMPath, Gsym.takeError());


  if (llvm::Expected Result = Gsym->lookup(Addr))
OS << Result.get();
  else
error(Result.takeError());

The LookupResult object has all of the information including function + source 
file + source line for all frames from the inline functions and concrete 
functions from the one address you looked up.

Anyone interested in learning more can feel free to message me directly offline 

Re: [Dwarf-Discuss] debug_aranges use and overhead

[Greg Clayton via Dwarf-Discuss]

> On Mar 11, 2021, at 1:12 PM, Paul Robinson via Dwarf-Discuss 
>  wrote:
> 
> Tom Russell could perhaps speak to this better, but my understanding is that 
> our debugger guys like having .debug_aranges, because parsing the CU DIE does 
> take that extra effort.  I am unfamiliar with their code so I have to take 
> their word on it.  But I can certainly imagine that probing hundreds to 
> thousands of CUs in order to collect range information with lengthy range 
> lists would be more expensive than running through a comparatively compact 
> .debug_aranges list.  If Tom tells me I’m wrong, well, wouldn’t be the first 
> time.

We will use them if they are there, but one interesting issue that we ran into 
with LLDB is some compile units might be in .debug_aranges because the compiler 
made a .debug_aranges section in the .o file, but others might not. So we had 
to add code to LLDB to figure out which compile units have any entries in the 
.debug_aranges section, and read the DW_AT_ranges from the DW_TAG_compile_unit 
if it exist, and if it doesn't, manually index the DWARF to create one on the 
fly each time.

>  
> One thing we have encountered (see issue 210113.1) is that when we’ve done 
> dead-stripping, .debug_aranges entries (one per function, typically, because 
> -ffunction-sections) can end up pointing to nothing.  In our proprietary 
> linker I believe we compress/rewrite .debug_aranges to minimize the number of 
> entries, which by coincidence ends up producing a conforming aranges list; 
> LLD doesn’t do that, which means it produces a non-conforming list (with 
> zero-length entries), hence the issue. 
>  
> I’ll have to think about what a “modern” .debug_aranges might want to look 
> like.

A big issue with any of the DWARF sections is we are subject to making the 
contents work with linkers that just want to concatenate + relocate. This often 
leads to information being kept around when dead stripping occurs because 
anything that is dead stripped will just have its address zero'ed out or -1'ed 
out, but this bogus info is still in the data.

If we don't need a format that can simply be concatenated and relocated, the 
GSYM format, which is open sourced in llvm.org  already, 
might be good inspiration for a .debug_aranges successor section that has very 
efficient lookups. The GSYM format could actually be used as is by adding only 
a new DIE offset IntoType. 

Besides ".debug_names", all other DWARF accelerator tables are really just 
random indexes that must be linearly scanned or pre-indexed prior to being used 
because of the concatenate + relocate style that is used for these DWARF 
sections. It would be great if any future accelerator tables are "map into 
memory and use as is" kind of tables like ".debug_names" and the ".apple_XXX" 
name accelerator tables. 


> Thanks,
> --paulr
>  
> From: David Blaikie  
> Sent: Thursday, March 11, 2021 3:48 PM
> To: Robinson, Paul 
> Cc: Cary Coutant ; DWARF Discuss 
> 
> Subject: debug_aranges use and overhead
>  
> On Thu, Mar 11, 2021 at 5:48 AM  > wrote:
> Hopefully not to side-track things too much... maybe wants its own
> thread, if there's more to debate here.
> 
> Yeah, how about we spin it off into another thread (done here)
>  
> >> For the case you suggested where it would be useful to keep the range
> >> list for the CU in the .o file, I think .debug_aranges is what you're
> >> looking for.
> >
> > aranges has been off by default in LLVM for a while - it adds a lot of
> > overhead (doesn't have all the nice rnglist encodings for instance -
> > nor can it use debug_addr, and if it did it'd still be duplicate with
> > the CU ranges wherever they were).
> 
> Did you want to file an issue to improve how .debug_aranges works?
> 
> I don't currently understand the value it provides, and I at least don't have 
> a use case for it, so I'm not sure I'd be the best person to advocate/drive 
> that work.
> 
> Complaining that it duplicates CU ranges is missing the point, though;
> it's an index, like .debug_names, of course it duplicates other info.
> If you want to suggest an improved index, like we did with .debug_names,
> that would be great too.
> 
> .debug_names is quite different though - it collects information from across 
> the DIE tree - information that is expensive to otherwise gather (walking the 
> whole DIE tree).
> 
> .debug_aranges is not like that for most producers (producers that do include 
> the address ranges on the CU DIE) - the data is readily available immediately 
> on the CU. That does involve reading some of .debug_abbrev, and interpreting 
> a handful of attributes - but at least for the use cases I'm aware of, that 
> overhead isn't worth the size increase.
> 
> Do you have numbers on the benefits of .debug_aranges compared to parsing the 
> ranges from CU DIEs?
> 
> (one possible issue: the CU doesn't /have/ to contain low/high/ranges if its 
> children DIEs 

Re: [Dwarf-Discuss] Retrieving variables, function address using dwarf

[Greg Clayton via Dwarf-Discuss]

Most local variables have locations that do require registers. 
DW_OP_call_frame_cfa says it needs to push the value that defines the call 
frame address which is typically based on the SP or FP depending on how things 
were compiled, so you would need registers for this. DW_OP_fbreg is another 
common opcode for local variables which relies on you being able to evaluated 
the DW_TAG_subprogram's DW_AT_frame_base attribute, which is a location 
expression, that often is something like "SP + " or "FP + ", so you 
will need registers for that too.

> On Mar 10, 2021, at 10:38 PM, Archana Deshmukh via Dwarf-Discuss 
>  wrote:
> 
> Thanks Michael for the response. Actually, I have only this much information. 
>  
> 
> I need to get information related to 
> 
>  For global variables , I read the address "55b51afea000" from > 
> /proc//maps file. I use DW_OP_addr parameter to retrieve the address.
> 
>  55b51afea000 + DW_OP_addr gives me the address of global variables.  
> 
> For function, I read the address "55b51afea000" from > /proc//maps 
> file.. I use DW_AT_low_pc parameter to retrieve the function starting address
> 
> Now, I need to read the local variables address. As I am not executing the 
> process, I cannot use registers. I need to use DW_OP_call_frame_cfa. I am not 
> able to understand how to retrieve addresses using  DW_OP_call_frame_cfa.
> 
> Any pointer or suggestion are most welcome.
> 
> Best Regards,
> Archana 
> 
> 
> 
>   
> 
> On Tue, Mar 9, 2021 at 11:07 PM Michael Eager  > wrote:
> It's difficult to offer advice with such a spare description.
> 
> You might read the executable and relocate the .debug_info and
> other debug sections using the process map.  If you have the
> process image, this probably would not be necessary.
> 
> On 3/8/21 1:49 AM, Archana Deshmukh via Dwarf-Discuss wrote:
> > Hello,
> > 
> > I have a pinatrace.out and process map of a file.
> > With this input, I need to build a symbol table.
> > 
> > Best Regards,
> > Archana Deshmukh
> > 
> > On Sun, Mar 7, 2021 at 10:29 AM Archana Deshmukh 
> > mailto:desharchan...@gmail.com> 
> > >> wrote:
> > 
> > 
> > 
> > -- Forwarded message -
> > From: *Michael Eager* mailto:ea...@eagercon.com> 
> > >>
> > Date: Sat, Mar 6, 2021 at 10:53 PM
> > Subject: Re: [Dwarf-Discuss] Retrieving variables, function address
> > using dwarf
> > To: Archana Deshmukh  > 
> > >>, 
> > mailto:dwarf-discuss@lists.dwarfstd.org>
> >  > >>
> > 
> > 
> > On 3/5/21 8:28 PM, Archana Deshmukh via Dwarf-Discuss wrote:
> >  > I need to read the address of local variable, global variable,
> > function
> >  > name and function arguments from the process.
> >  >
> >  > For global variables , I read the address "55b51afea000" from
> >  > /proc//maps file. I use DW_OP_addr parameter to retrieve the
> > address.
> >  > 55b51afea000 + DW_OP_addr gives me the address of global variable.
> >  >
> >  > I need to read the stack segment, heap. Is there any way to read
> >  > segments? DW_AT_segment parameter seems to be for 16 bit.
> >  >
> >  > I need to read the following process map using dwarf.
> >  >
> >  > Any suggestion, pointers are welcome.
> >  >
> >  > 55b51afea000-55b51afeb000 r-xp  fd:00 5902563
> > 
> > Can you explain what you are trying to do?
> > 
> > Usually a DWARF consumer (a debugger) does not need to read the
> > process memory map.  All of the information you mention is in
> > the DWARF data.  You may need to relocate addresses in the DWARF
> > debug data.
> > 
> > DWARF does not contain information about the process memory
> > layout, such as the location of the heap or the start of the
> > stack.
> > 
> > -- 
> > Michael Eager
> > 
> > 
> > ___
> > Dwarf-Discuss mailing list
> > Dwarf-Discuss@lists.dwarfstd.org 
> > http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org 
> > 
> > 
> 
> -- 
> Michael Eager
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] implicit_value vs stack_value

[Greg Clayton via Dwarf-Discuss]

DW_OP_implicit_value, as you said, allows encoding a constant that can be 
larger than the stack value size, so it will always be needed for these kinds 
of constant values.


> On Jan 4, 2021, at 11:30 AM, Paul Robinson via Dwarf-Discuss 
>  wrote:
> 
> Happy New Year, everybody!
> 
> A colleague just had a question for me about DW_OP_implicit_value
> which led me to wonder why we have both that and DW_OP_stack_value.
> Looking at http://dwarfstd.org/ShowIssue.php?issue=071227.1 which
> introduced the latter, it says in part:
> 
>  (This operator is similar to DW_OP_implicit_value, issue "070426.1".  The
>  latter only permits the description of values known to be literals at
>  compile-time; this proposal permits the description of values which
>  have known expressions at compile-time.  It is more general; since 
>  a sequence of DW_OP_constxx, DW_OP_value can replace every instance
>  of DW_OP_implicit_value, this proposal also includes the removal of
>  DW_OP_implicit value from the standard.)
> 
> ...although I don't think "the removal of DW_OP_implicit_value" actually
> happened.
> 
> I have come up with only two advantages that DW_OP_implicit_value would
> have, compared to DW_OP_stack_value:
> 1) it can express a value larger than one expression-stack element in
>   a single operation;
> 2) it's faintly possible that it's simpler for a producer to produce.
> 
> But, DW_OP_stack_value would frequently result in a more compact 
> representation, for example:
>  DW_OP_implicit_value 4 0 0 0 0 => 6 bytes
>  DW_OP_lit0 DW_stack_value => 2 bytes
> 

DW_OP_stack_value can be used after a complex expression to just say that "the 
value on the stack if the value of the expression result and we are done". 

DWARF, prior to version 5, required expression evaluation to just know what 
kind of value something was on the expression stack and "do the right thing". 
DWARF5 cleared this up with more clearly defined rules and DW_OP_stack_value 
was one of those things. DW_OP_addr has a value that is a virtual address in a 
file and the expression evaluation needs to do the right thing. If we wanted to 
make an expression that got the address of a global variable, you could use:

DW_OP_addr(0x1000) DW_OP_stack_value

If you have a running process, you would convert address 0x1000 to the address 
that this this .data was loaded in the process and push it on the stack, and 
then the DW_OP_stack_value would let you know that the result _is_ the value 
even though it started off as a file address and got converted to a load 
address. So in a running process with a .data segment whose file address was 
0x1000 and is loaded at 0x2001000 we would have an expression stack like:

DW_OP_addr(0x1000)
stack[0] = 0x2001000 (type = load address)

DW_OP_stack_value
stack[0] = 0x2001000 (type = stack value)

Where as in a debugger that wasn't running we will have:

DW_OP_addr(0x1000)
stack[0] = 0x1000 (type = file address)

DW_OP_stack_value
stack[0] = 0x1000 (type = stack value)


So the DW_OP_stack_value allows expressions to clearly state that the 
expression result is on the stack and isn't a file or load address in case you 
end up using multiple different opcodes (DW_OP_addr, DW_OP_breg, DW_OP_reg, 
DW_OP_const, etc) that might muddy the waters was to what the resulting value 
is. 

> I don't see much value  in actually deprecating or removing
> DW_OP_implicit_value, now that it's out in the world; is it worth
> adding a non-normative note that DW_OP_stack_value is actually more
> general?

They are two different opcodes. DW_OP_implicit_value makes constant values that 
do not need to fit into a 32 or 64 bit value and that doesn't push anything 
onto the stack, and DW_OP_stack_value allows you to change what the type of the 
top stack value is, no matter how it was created.

> 
> Thanks,
> --paulr
> 
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] .debug_addr entry plus offset

[Greg Clayton via Dwarf-Discuss]

One simple approach would be to be able to represent a DW_AT_low_pc with a 
DW_FORM_data encoding just like the DW_AT_high_pc does when it is an offset 
from the DW_AT_low_pc. The value of the DW_AT_low_pc would be an offset from 
either:
1 - the parent DIE's DW_AT_low_pc (which itself might need to be resolved by 
looking at the parent scope). If the parent DIE's range is a DW_AT_ranges, then 
use the lowest address out of all of them.
2 - the first parent DIE with a DW_AT_low_pc that has a DW_FORM_addrXXX 
encoding.

Solution #1 is nice because it keeps the offset in the DW_FORM_data encoding 
small since it is always relative to the first parent scope's DW_AT_low_pc. So 
this could save a lot of space in the DWARF if we use the smallest possible 
DW_FORM_data encoding all the time.
Solution #2 could be easier as you would traverse parent scopes looking for an 
address encoding as the DW_FORM.

This would allow DW_TAG_subprogram DIEs to have a single relocation on the 
DW_AT_low_pc.

Greg Clayton


> On Sep 15, 2020, at 10:12 AM, Robinson, Paul via Dwarf-Discuss 
>  wrote:
> 
> David Blaikie has brought this up with me (or in conversations that
> I observed) a couple of times:
> 
> It's common to want to refer to a particular address plus an offset,
> for example for DW_AT_low_pc or DW_AT_ranges to describe a lexical
> block or inlined subprogram within another subprogram.  Generally
> the only symbolic address available is the entry point of the
> containing subprogram.  Back when addresses were held directly in 
> the .debug_info section, the attributes would have relocations, the
> offset would be encoded into the relocation and the linker would
> just do the right thing.
> 
> With DWARF v5, we now have the .debug_addr section, which contains
> the addresses to be fixed up by the linker.  But, we don't have a
> way to specify an offset to add to an entry in the .debug_addr
> section; instead, each unique addr+offset requires its own entry
> in the .debug_addr table.  This consumes additional space, these
> entries are generally not reusable, and it doesn't reduce the
> overall number of relocations that the linker must process.
> 
> It's not feasible to define a new attribute for address+offset,
> because an attribute has only one value, and the attribute would
> have to specify both the .debug_addr index and the offset to add.
> But, we could define an "indirect" entry in .debug_addr, and then
> reference it with an attribute in the same way that we reference
> any other .debug_addr entry.
> 
> An indirect entry would be the same size as all other entries in 
> .debug_addr (i.e., the size of an address on the target).  The
> upper half would be another index into .debug_addr and the lower
> half would be the addend.  The consumer adds the addend to the
> value from the entry specified by the "another index."
> 
> This solution doesn't save space in .debug_addr, but it does
> reduce the number of relocations.  Ideally .debug_addr would
> require only one relocation per function.
> 
> We can debate whether the addend should be signed or unsigned,
> and whether the indirect entries should be a separate subtable,
> but I wanted to float the idea here before I wrote it up as a
> proposal.
> 
> Alternatively, the indirect sub-table could be encoded with
> ULEB/SLEB pairs, but that makes it hard to find them by index.
> They could be found by a direct reference, but that requires a
> relocation from .debug_info to .debug_addr, so we haven't saved
> any relocations that way.
> 
> If there are obvious flaws I can't see, or someone is inspired
> to come up with another solution, please let me know!  Otherwise
> I'll write it up as a formal proposal probably later this week.
> 
> Thanks,
> --paulr
> 
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] Stack tracing and return addresses off by 1

[Greg Clayton via Dwarf-Discuss]

You are correct. For the first frame you don't adjust the PC when looking up 
the unwind row. For the second frame on up you can decrement the PC value by 1 
before when doing the row lookup and that is usually enough to get you to the 
correct unwind row. The issue is that the return address points to the next 
instruction after the instruction that called the function. This also fixes 
issues with tail calls. Now you might ask why decrementing by 1 works when 
instructions can often be larger that 1 byte. We just need to get to the 
previous row in the unwind table, and decrementing by 1 is usually enough to 
get us there because unwind rows start at valid instruction opcode addresses.

The other tricky thing to watch out for is that the unwind information isn't 
always valid for the first frame for all values of the PC in a function. Why? 
Most unwind information is only valid at places that can throw exceptions. This 
means that when unwinding the first frame, you really can't trust the unwind 
info unless you know you are at a location that can throw an exception which is 
hard to detect by just looking at disassembly. Compilers have the ability to 
enable asynchronous unwinding with a compiler option, but even if we do enable 
this, there is no way to look at the unwind information at run time and tell 
the difference between synchronous unwind info (not valid everywhere in the 
function, only at places that can throw exceptions) and asynchronous unwind 
info (valid for any PC value in the function). To make things worse, the 
information that is put into .debug_frame often is just the same unwind info 
that is put into .eh_frame (with a few syntactic differences in encoding), so 
just know that .debug_frame is often only valid at exception call sites but 
there is not way to tell unless your compiler emits it compiler invocation 
flags in the DWARF in the DW_TAG_compile_unit as an attribute.

The LLDB debugger will use unwind info from the binary for all frames except 
the first frame. For the first frame, we actually decode assembly and create 
our own unwind information that is valid everywhere in the function. 

Now, the good news is, if you are making a backtrace for a thread that has 
crashed, it should be at a valid address and allow you to use the unwind 
information. If you also have other threads that were stopped when one thread 
crashes, they won't be at valid locations for unwind for the first frames.


> On Jul 31, 2020, at 5:37 AM, Jayvee Neumann via Dwarf-Discuss 
>  wrote:
> 
> Hello together!
> 
> I am running into a problem while performing a stack trace of x86 code. The 
> assembly code I am running has been generated by mingw from C++ code and 
> looks like this:
> 
> 6c9c1210 <__ZN7my_class9my_methodEs>:
> 6c9c1210: sub$0x1c,%esp
> 6c9c1213: mov0x20(%esp),%edx
> 6c9c1217: mov%edx,%eax
> 6c9c1219: test   %dx,%dx
> 6c9c121c: jle6c9c1221 <__ZN7my_class9my_methodEs+0x11>
> 6c9c121e: lea0x1(%edx),%eax
> 6c9c1221: cwtl   
> 6c9c1222: mov%eax,(%esp)
> 6c9c1225: call   6c9c1190 <__Z12my_dummy_functions>
> 6c9c122a: add$0x1c,%esp
> 6c9c122d: ret$0x4
> 
> 6c9c1230 <__ZN8my_struct9my_methodEv>:
> 6c9c1230: sub$0x1c,%esp
> 6c9c1233: movzwl 0x4(%ecx),%eax
> 6c9c1237: test   %ax,%ax
> 6c9c123a: jle6c9c1241 <__ZN8my_struct9my_methodEv+0x11>
> 6c9c123c: add$0x1,%eax
> 6c9c123f: jmp6c9c1246 <__ZN8my_struct9my_methodEv+0x16>
> 6c9c1241: mov$0x0,%eax
> 6c9c1246: cwtl   
> 6c9c1247: add$0x8,%ecx
> 6c9c124a: mov%eax,(%esp)
> 6c9c124d: call   6c9c1210 <__ZN7my_class9my_methodEs>
> 6c9c1252: sub$0x4,%esp
> 6c9c1255: add$0x1c,%esp
> 6c9c1258: ret
> 6c9c1259: nop
> 6c9c125a: lea0x0(%esi),%esi
> 
> The problem manifests itself, when the instruction pointer is inside " 
> __ZN7my_class9my_methodEs" (called at 0x6c9c124d).
> 
> In order to perform the stack trace, I use the DWARF frame information for 
> calculating the previous instruction pointer. This is done by assuming the 
> return address is the instruction pointer of the previous frame. This is 
> obviously not entirely correct, since the return address points to a location 
> AFTER the previous call. Nevertheless, this assumption seems to be standard 
> for other stack tracers.
> 
> I am having a problem with this though:
> The address where I start is 0x6c9c121e. Frame information tells me the 
> following:
> 
> 0144 001c  FDE cie= pc=6c9c1210...6c9c1230
>   DW_CFA_advance_loc4: 3
>   DW_CFA_def_cfa_offset: +32
>   DW_CFA_advance_loc4: 26
>   DW_CFA_def_cfa_offset: +4
>   DW_CFA_nop:
>   DW_CFA_nop:
> 
> So the CFA offset is 32. There I find the next return address 0x6c9c124d. 
> Frame information tells me the following:
> 
> 0164 0028  FDE cie= pc=6c9c1230...6c9c1259
>   DW_CFA_advance_loc4: 3
>   DW_CFA_def_cfa_offset: +32
>   DW_CFA_advance_loc4: 31
>   DW_CFA_def_cfa_offset: +28
>   DW_CFA_advance_loc4: 3
>   

Re: [Dwarf-Discuss] Question about DW_TAG_inlined_subroutine tags

[Greg Clayton via Dwarf-Discuss]

> On Jul 30, 2020, at 12:21 PM, David Blaikie  wrote:
> 
> On Thu, Jul 30, 2020 at 12:00 PM Greg Clayton via Dwarf-Discuss
>  wrote:
>> 
>> The LTO in clang creates some really interesting DWARF... One of the latest 
>> things I discovered is DW_TAG_inlined_subroutine tags that are not contained 
>> within a DW_TAG_subprogram. I am guessing the compiler/linker wanted to 
>> outline an inlined function and tried its best to move the DWARF and didn't 
>> end up changing the tag from DW_TAG_inlined_subroutine to DW_TAG_subprogram.
> 
> (you've mentioned a couple of quirky LTO situations that I don't think
> I've seen with LLVM's LTO - do you have examples of these (this one
> and the other one decl file/line one discussed on llvm-commits)?

We have this some private company code, but I will see if I can reproduce this 
by building some LLVM or clang tools and I will file some bugs if I can 
reproduce.
> 
>> I was thinking of adding code to "llvm-dwarfdump --verify" to detect this 
>> issue, but wanted to check with the DWARF list first to make sure this would 
>> be considered an error. So I am looking for an answer to:
>> 
>> Is it ok for DW_TAG_inlined_subroutine with high and low PC values to appear 
>> on their own, not enclosed in a DW_TAG_subprogram?
> 
> The only wording I can find is:
> 
> "Each inline expansion of a subroutine is represented by a debugging
> information entry with the tag DW_TAG_inlined_subroutine. Each such
> entry is a direct child of the entry that represents the scope within
> which the inlining occurs."
> 
> I guess this could still technically allow inlining into some place
> that isn't described as a subprogram or child of a subprogram in the
> DWARF (eg: inlining into a global initializer - I guess some DWARF
> producer could model that as
> DW_TAG_compile_unit{DW_TAG_inlined_subroutine}) - so I'd err on the
> side of saying DWARF doesn't categorically disallow this.
> 
> But as an LLVM maintainer, I'd be totally fine adding that as a
> verifier check to llvm-dwarfdump.

Sounds good. I will submit a patch shortly.

Greg
___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] Selectively strip CUs from .debug_info?

[Greg Clayton via Dwarf-Discuss]

Not aware of any tools that can do this, but can't you just do this in your 
build?:

- link once with full .o files and save debug info off
- strip debug info from all third party .o files and link again. You still get 
all debug info for any .o files you didn't strip




> On Apr 9, 2020, at 2:50 PM, Robinson, Paul via Dwarf-Discuss 
>  wrote:
> 
> Does anyone know of a tool that can strip debug info for specified
> CUs from an executable?  I'm not aware of a way to do this, but
> there are many things I'm not aware of. 
> 
> The use case is someone who wants to build the entire program
> (which includes a number of 3rd-party libraries) with debug info,
> so they'll have full symbols for crash dump analysis; but then
> strip the debug info for those libraries, in order to speed up
> debugger load time.  In this scenario, stripping the 3rd-party
> code before linking isn't going to satisfy the crash dump analysis
> requirement.
> 
> I've also brought up split DWARF, but I'm not sure it fits the need.
> 
> Thanks,
> --paulr
> 
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] variable locations - safe use as lvalues

[Greg Clayton via Dwarf-Discuss]

> On Jan 21, 2020, at 2:58 PM, Frank Ch. Eigler via Dwarf-Discuss 
>  wrote:
> 
> Hi -
> 
>> While dwarfdump has 'check the DWARF' options available the
>> location list overlap test is not done at all. The -kl option suggests
>> it does loclist checking, but its checking is rather superficial.
> 
> Note that one should not present these overlaps as -errors- (because
> they may be accurate), but rather as complications for lvalue usage.
> 

Indeed. Warnings might be nice if the locations expressions are the same _and_ 
the types are different.

It would be very hard to figure out in a language agnostic way if two types are 
not the same ("int" and a typedef to a "int" for example for C based 
languages). 

Maybe if we have two different DWARF types entirely, like one 
DW_TAG_structure_type and one DW_TAG_enumeration. We would also need to omit 
any values that are constants values since two different types could both have 
the same location expression in that case. There are many complexities involved.


> - FChE
> 
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] variable locations - safe use as lvalues

[Greg Clayton via Dwarf-Discuss]

> On Jan 20, 2020, at 6:21 PM, Frank Ch. Eigler via Dwarf-Discuss 
>  wrote:
> 
> Hi -
> 
>>> - ... and undoubtedly other complications exist!
>> 
>> Interesting question.
> 
> Thanks!
> 
> We have been thinking in similar directions generally as y'all.
> 
> 
>> Complication 2: The compiler reuses variable locations at the same PC. This
>> seems to be a compiler bug.  
> 
> (Actually, this could be a valid optimization, e.g.:
> 
>   int a = expression;
>   int b = expression;
>> 
>   /* use/modify a */
>   /* use/modify b */
> 
> at that point, if $expression is a pure function, a compiler
> could evaluate it once and reuse the value.  It could do this
> by temporarily storing both a and b in the same register, and
> only separating the variables afterwards.
> 
>> [...]
>> Presumably, a debugger could check that location lists do not
>> overlap.
> 
> This could nevertheless be a valid heuristic to detect the case.

the "llvm-dwarfdump" binary in llvm.org  has a "--verify" 
option that looks for issues in DWARF and reports errors. Not sure if there are 
any other tools out there that do this, but this would be a great check to add 
to any DWARF verifier.

> 
> 
>> Complication 3a: That the value of a variable has been fetched for a
>> computation before the debugger modifies it.  This is more complicated. The
>> live range of the variable is accurate, but its value has been used before
>> the current PC.  DWARF does not include descriptions of data flow or
>> indicate where variables are fetched, so there is no information that a
>> debugger can use to assure that a modified value is actually used.
> 
> Yeah, and in the absence of dataflow metadata, tools like dyninst must
> try to reverse-engineer the dataflow in order to perform binary
> rewriting.  Is this something way out of forseeable dwarf scope?
> 
> 
>> There are a lot of issues with a debugger modifying a program while it is
>> running.  A debugger can make essentially unbounded changes to the program
>> state.  Some of these may work as expected, some may not, and it is unclear
>> how a debugger would be able to know the difference.
> 
> This is the key question: how can a tool know what is safe.  While the
> trivial case of assuming every write is unsafe is not helpful :-), it
> could be okay to have fairly conservative heuristics, known-partial
> information, and rely on only clear signals to enable write
> operations.

As soon as optimization are in, the live ranges for many variables is quite 
sparse from my experience with most compilers. LLDB can detect when we are 
debugging optimized code. This is currently done by looking for the 
DW_AT_APPLE_optimized attribute on the DW_TAG_compile_unit, so this isn't 
available everywhere. But in general, if we know a program isn't optimized, 
then the variable locations are often live for much longer. But there is no 
guarantee that we can detect when things are optimized. 
> 
> 
> Anyway, it sounds like the next step is on us to analyze & prototype
> in a compiler (gcc?).  I'd also appreciate authors of other dwarf
> consumer and producer tools to mention whether they have considered
> this area, so as to collect a census.

I enjoy data mining DWARF for the information it contains, so the 
"llvm-dwarfdump --verify" is the main area I could see adding support for 
detecting when/if a compiler emits overlapping locations for variables. Let us 
know what you find out and what you end up doing. Very interesting to think 
about.
> 
> 
> - FChE
> 
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] .debug_frame and the base address

[Greg Clayton via Dwarf-Discuss]

> On Sep 24, 2018, at 8:44 AM, Robert Harris  wrote:
> 
> 
> 
>> On 24 Sep 2018, at 15:33, Greg Clayton  wrote:
>> 
>> 
>> 
>>> On Sep 24, 2018, at 7:15 AM, Robert Harris via Dwarf-Discuss 
>>>  wrote:
>>> 
>>> I'd like some clarification regarding what precisely needs to be relocated 
>>> relative to
>>> the object load address when interpreting .debug_frame.  I'm interested in 
>>> versions
>>> 2, 3 and 4.
>>> 
>>> I think the only explicit mention of relocation in DWARF 4 is § 6.4.1:
>>> 
>>> Abstractly, this mechanism describes a very large table...
>>> ...
>>> The first column indicates an address for every location that contains
>>> code in a program. (In shared objects, this is an object-relative 
>>> offset.)
>>> 
>>> Thus adjustments need to be made to only
>>> 
>>> An FDE's initial_location field
>>> DW_CFA_set_loc's operand
>>> 
>>> Have I missed anything?  I see some ambiguity in the definition of
>>> DW_CFA_def_cfa_expression and it's making me wonder if there are any
>>> other occasions when a result would require relocation.
>> 
>> Any DWARF expression can contain a DW_OP_addr, which has an address 
>> argument, and would need to be relocated.
> 
> Do you mean that DW_OP_addr's argument should be relocated in all instances
> of a DWARF expression, i.e. in other .debug_ sections, or just in the context 
> of
> .debug_frame?

I was specifically talking about DWARF expressions in any .debug_frame info. 
But in general, yes, any DWARF expression that contains a DW_OP_addr might need 
a relocation for its address argument.

Greg___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

Re: [Dwarf-Discuss] .debug_frame and the base address

[Greg Clayton via Dwarf-Discuss]

> On Sep 24, 2018, at 7:15 AM, Robert Harris via Dwarf-Discuss 
>  wrote:
> 
> I'd like some clarification regarding what precisely needs to be relocated 
> relative to
> the object load address when interpreting .debug_frame.  I'm interested in 
> versions
> 2, 3 and 4.
> 
> I think the only explicit mention of relocation in DWARF 4 is § 6.4.1:
> 
>   Abstractly, this mechanism describes a very large table...
>   ...
>   The first column indicates an address for every location that contains
>   code in a program. (In shared objects, this is an object-relative 
> offset.)
> 
> Thus adjustments need to be made to only
> 
>   An FDE's initial_location field
>   DW_CFA_set_loc's operand
> 
> Have I missed anything?  I see some ambiguity in the definition of
> DW_CFA_def_cfa_expression and it's making me wonder if there are any
> other occasions when a result would require relocation.

Any DWARF expression can contain a DW_OP_addr, which has an address argument, 
and would need to be relocated. 

> 
> Robert Harris
>   
> ___
> Dwarf-Discuss mailing list
> Dwarf-Discuss@lists.dwarfstd.org
> http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org

___
Dwarf-Discuss mailing list
Dwarf-Discuss@lists.dwarfstd.org
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org