Re: [lldb-dev] RFC for DWZ = DW_TAG_imported_unit + DWARF-5 supplementary files

2017-08-24 Thread Greg Clayton via lldb-dev 

> On Aug 24, 2017, at 8:38 AM, Jan Kratochvil  wrote:
> 
> On Wed, 23 Aug 2017 23:55:00 +0200, Greg Clayton wrote:
>>> On Aug 23, 2017, at 2:06 PM, Jan Kratochvil via lldb-dev 
>>>  wrote:
>>> Currently it always has to as on non-OSX platforms it is using
>>> DWARFCompileUnit::Index(). But as I plan to implement DWARF-5 .debug_names
>>> index (like __apple_* index) maybe LLDB then no longer needs to populate
>>> m_die_array and so just expanding all DW_TAG_partial_unit into a single
>>> m_die_array for each DW_TAG_compile_unit is fine?
>> 
>> So I glossed over the documentation and I gathered that DWARF type info
>> might be stored in other DWARF files and references from the current file.
> 
> Yes. Arbitrary DIEs, not just type-defining DIEs. DW_TAG_imported_unit can
> happen anywhere according to the DWARF standard but for the DWZ/Fedora case it
> is a bit simpler one can limit the support for only DW_TAG_imported_unit with
> parent of DW_TAG_compile_unit (or DW_TAG_partial_unit for nested imports).
> DWZ ever uses DW_TAG_imported_unit only at the CU/PU top level.
> 
> 
>> SymbolFileDWARFDebugMap is an example of how we do things on MacOS. We have
>> one clang::ASTContext in the SymbolFileDWARFDebugMap, and multiple external
>> .o files (where each ins a SymbolFileDWARF instance) that contain unlinked
>> DWARF. Each SymbolFileDWARF instance will have:
>> 
>>  void SymbolFileDWARF::SetDebugMapModule(const lldb::ModuleSP _sp);
>> 
>> called to indicate it is actually part of the SymbolFileDWARFDebugMap.
>> Then there are functions that check the debug map file and return the
>> UniqueDWARFASTTypeMap or the TypeSystem from the SymbolFileDWARFDebugMap if
>> we have one:
> 
> Therefore IIUC you make a single context for all types from a compiled
> program?  Primitive (non-class) types can be different across CUs:
>   CU1: typedef int  foo_t;
>   CU2: typedef long foo_t;

We can and we do allow this. We will have two foo_t definitions in the AST 
context at the same level. Each variable has the correct type as it directly 
points to the right "foo_t". When these variables are used in expressions, we 
will only copy 1 of the "foo_t" definitions over into the expression AST 
context so things normally work out. If you tried to cast something to a 
"foo_t" you would run into problems though since the type would be ambiguous.
> 
> I have a problem that one DW_TAG_partial_unit can be included by multiple
> DW_TAG_compile_unit.  Therefore DWARFCompileUnit for DW_TAG_partial_unit
> itself cannot map itself by SymbolFileDWARFDebugMap to its parent
> DWARFCompileUnit for DW_TAG_compile_unit (as it has multiple parents).
> 
> I expect you cannot link a single MacOS object file into two diferent
> programs/libraries which are debugged at once by LLDB.

You can. We always open a new instance of the .o file, one for each 
SymbolFileDWARFDebugMap or unique executable that refers to the .o file.
> 
> 
> 
>> One other idea is to keep all DWARF files separate and stand alone. Your
>> main DWARF file with one or more DW_TAG_imported_unit and all
>> DW_TAG_imported_unit referenced files, each as its own SymbolFileDWARF. Any
>> reference to a DW_FORM_ref_alt would turn into a forward declaration in the
>> current SymbolFileDWARF, so the ASTContext in each SymbolFileDWARF wouldn't
>> know anything about the types,
> 
> Is it applicable even if DW_TAG_imported_unit points to DW_TAG_partial_unit's
> containing DW_TAG_variable, DW_TAG_subprogram and arbitrary other DIEs, not
> just types?

This approach will not work if that is the case. Scratch the above idea...
> 
> 
>> The other approach I might suggest is to write a DWARF linker, maybe using
>> LLVM's DWARF classes (see llvm-dsymutil sources) that takes the top level
>> DWARF and all DW_TAG_imported_unit files and combines them all back into one
>> large DWARF file.
> 
> That would defeat the advantage of DWZ-optimized debug info files.  DWZ
> reduces their size by approx. 30%.  If the relinking was made on-demand then
> it defeats the LLDB performance advantage over GDB - GDB can already read DWZ
> natively (GDB does duplicates the internal representation when reading DWZ
> CUs/PUs).  In such case I could already code such reconstruction of non-DWZ
> debug info when reading m_die_array - but that also seems needlessly slow to
> me.  DWZ should bring even performance advantage of parsing the DWZ common
> file (=imported file) only once.
> 
> 
>> One other idea is to let each DWARF file be separate, and when you need
>> a type from a DW_TAG_imported_unit you log that file as stand alone and copy
>> the type from its clang::ASTContext into the main SymbolFileDWARF's AST
>> context. We copy types all the time in expressions as each on has its own
>> AST context.
> 
> Does this work even for non-type DIEs? 

No it does not. Only types are fine, but anything else will fall down.
> 
> 
>> So there are many solutions. I would

Re: [lldb-dev] RFC for DWZ = DW_TAG_imported_unit + DWARF-5 supplementary files

2017-08-24 Thread Jan Kratochvil via lldb-dev 
On Wed, 23 Aug 2017 23:55:00 +0200, Greg Clayton wrote:
> > On Aug 23, 2017, at 2:06 PM, Jan Kratochvil via lldb-dev 
> >  wrote:
> >  Currently it always has to as on non-OSX platforms it is using
> > DWARFCompileUnit::Index(). But as I plan to implement DWARF-5 .debug_names
> > index (like __apple_* index) maybe LLDB then no longer needs to populate
> > m_die_array and so just expanding all DW_TAG_partial_unit into a single
> > m_die_array for each DW_TAG_compile_unit is fine?
> 
> So I glossed over the documentation and I gathered that DWARF type info
> might be stored in other DWARF files and references from the current file.

Yes. Arbitrary DIEs, not just type-defining DIEs. DW_TAG_imported_unit can
happen anywhere according to the DWARF standard but for the DWZ/Fedora case it
is a bit simpler one can limit the support for only DW_TAG_imported_unit with
parent of DW_TAG_compile_unit (or DW_TAG_partial_unit for nested imports).
DWZ ever uses DW_TAG_imported_unit only at the CU/PU top level.


> SymbolFileDWARFDebugMap is an example of how we do things on MacOS. We have
> one clang::ASTContext in the SymbolFileDWARFDebugMap, and multiple external
> .o files (where each ins a SymbolFileDWARF instance) that contain unlinked
> DWARF. Each SymbolFileDWARF instance will have:
> 
>   void SymbolFileDWARF::SetDebugMapModule(const lldb::ModuleSP _sp);
> 
> called to indicate it is actually part of the SymbolFileDWARFDebugMap.
> Then there are functions that check the debug map file and return the
> UniqueDWARFASTTypeMap or the TypeSystem from the SymbolFileDWARFDebugMap if
> we have one:

Therefore IIUC you make a single context for all types from a compiled
program?  Primitive (non-class) types can be different across CUs:
CU1: typedef int  foo_t;
CU2: typedef long foo_t;

I have a problem that one DW_TAG_partial_unit can be included by multiple
DW_TAG_compile_unit.  Therefore DWARFCompileUnit for DW_TAG_partial_unit
itself cannot map itself by SymbolFileDWARFDebugMap to its parent
DWARFCompileUnit for DW_TAG_compile_unit (as it has multiple parents).

I expect you cannot link a single MacOS object file into two diferent
programs/libraries which are debugged at once by LLDB.


> One other idea is to keep all DWARF files separate and stand alone. Your
> main DWARF file with one or more DW_TAG_imported_unit and all
> DW_TAG_imported_unit referenced files, each as its own SymbolFileDWARF. Any
> reference to a DW_FORM_ref_alt would turn into a forward declaration in the
> current SymbolFileDWARF, so the ASTContext in each SymbolFileDWARF wouldn't
> know anything about the types,

Is it applicable even if DW_TAG_imported_unit points to DW_TAG_partial_unit's
containing DW_TAG_variable, DW_TAG_subprogram and arbitrary other DIEs, not
just types?


> The other approach I might suggest is to write a DWARF linker, maybe using
> LLVM's DWARF classes (see llvm-dsymutil sources) that takes the top level
> DWARF and all DW_TAG_imported_unit files and combines them all back into one
> large DWARF file.

That would defeat the advantage of DWZ-optimized debug info files.  DWZ
reduces their size by approx. 30%.  If the relinking was made on-demand then
it defeats the LLDB performance advantage over GDB - GDB can already read DWZ
natively (GDB does duplicates the internal representation when reading DWZ
CUs/PUs).  In such case I could already code such reconstruction of non-DWZ
debug info when reading m_die_array - but that also seems needlessly slow to
me.  DWZ should bring even performance advantage of parsing the DWZ common
file (=imported file) only once.


> One other idea is to let each DWARF file be separate, and when you need
> a type from a DW_TAG_imported_unit you log that file as stand alone and copy
> the type from its clang::ASTContext into the main SymbolFileDWARF's AST
> context. We copy types all the time in expressions as each on has its own
> AST context.

Does this work even for non-type DIEs?


> So there are many solutions. I would vote for linking the DWARF into
> a single file much like we do with llvm-dsymutil on Mac, but that really
> depends if the type uniquing is desired within a single DWARF file and not
> across many shared libraries that all reference common DW_TAG_imported_unit
> files.

I agree the DWZ optimization does primarily what -fdebug-types-section does.

I do not see why to really do relinking files on disk, debugger should not
need that to read the debug info.

I do not fully understand why you do the llvm-dsymutil relinking when you
already have SymbolFileDWARFDebugMap in LLDB.  But I do not know OSX/Mac.


Thanks,
Jan
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC for DWZ = DW_TAG_imported_unit + DWARF-5 supplementary files

2017-08-23 Thread Greg Clayton via lldb-dev 

> On Aug 23, 2017, at 2:06 PM, Jan Kratochvil via lldb-dev 
>  wrote:
> 
> Hello,
> 
> at least Fedora Linux distribution uses DWZ to reduce DWARF debug info size:
>   https://fedoraproject.org/wiki/Features/DwarfCompressor
> 
> It is DWARF optimization - not really a compression.  One can find it by
> DW_TAG_partial_unit/DW_TAG_imported_unit:
> <0>: Abbrev Number: 103 (DW_TAG_partial_unit)
>   DW_AT_stmt_list   : 0x0
><10>   DW_AT_comp_dir: (alt indirect string, offset: 0xe61c)
> <1><14>: Abbrev Number: 45 (DW_TAG_imported_unit)
><15>   DW_AT_import  : 
> ...
> 
> I have already made some attempt for its implementation:
>   https://people.redhat.com/jkratoch/lldb-2017-08-13.patch
>   https://people.redhat.com/jkratoch/lldb-2017-08-13imp.patch
> 
> But I found that approach as a dead-end because LLDB expects all the DIEs from
> a CU really belong to the same DWARFCompileUnit.  Contrary to LLDB
> expectations the patch above creates different DWARFCompileUnit for each
> DW_TAG_partial_unit.  This the patch solves for cross-DIE references but then
> it ends up with:
>   tools/clang/lib/AST/DeclBase.cpp:75:
>   Assertion `!Parent || >getParentASTContext() == ' failed.
> as GetCompUnitForDWARFCompUnit() is returning different clang context for DIEs
> from DWZ supplementary files (different SymbolFileDWARF) vs. base file CUs.
> I tried to generate alternative user_id_t to always refer to originating CU in
> the base file but it is more and more complicated.
> 
> Therefore I would like a new approach to keep all the DIEs from
> a DW_TAG_compile_unit incl. all its imported DW_TAG_partial_unit in the same
> DWARFCompileUnit.  So far I wanted to prevent expansion/copy of all
> DW_TAG_partial_unit m_die_array data into each of its parent
> DW_TAG_compile_unit as it may be a performance hit.
> 
> But then I am not sure whether it is worth it - when LLDB does fully populate
> m_die_array?

It expands them in:

size_t DWARFCompileUnit::ExtractDIEsIfNeeded(bool cu_die_only);

It will either only make the top level CU DIE, or it will parse all DIEs.

>  Currently it always has to as on non-OSX platforms it is using
> DWARFCompileUnit::Index(). But as I plan to implement DWARF-5 .debug_names
> index (like __apple_* index) maybe LLDB then no longer needs to populate
> m_die_array and so just expanding all DW_TAG_partial_unit into a single
> m_die_array for each DW_TAG_compile_unit is fine?

So I glossed over the documentation and I gathered that DWARF type info might 
be stored in other DWARF files and references from the current file.

SymbolFileDWARFDebugMap is an example of how we do things on MacOS. We have one 
clang::ASTContext in the SymbolFileDWARFDebugMap, and multiple external .o 
files (where each ins a SymbolFileDWARF instance) that contain unlinked DWARF. 
Each SymbolFileDWARF instance will have:

  void SymbolFileDWARF::SetDebugMapModule(const lldb::ModuleSP _sp);

called to indicate it is actually part of the SymbolFileDWARFDebugMap. Then 
there are functions that check the debug map file and return the 
UniqueDWARFASTTypeMap or the TypeSystem from the SymbolFileDWARFDebugMap if we 
have one:


UniqueDWARFASTTypeMap ::GetUniqueDWARFASTTypeMap() {
  SymbolFileDWARFDebugMap *debug_map_symfile = GetDebugMapSymfile();
  if (debug_map_symfile)
return debug_map_symfile->GetUniqueDWARFASTTypeMap();
  else
return m_unique_ast_type_map;
}

TypeSystem *SymbolFileDWARF::GetTypeSystemForLanguage(LanguageType language) {
  SymbolFileDWARFDebugMap *debug_map_symfile = GetDebugMapSymfile();
  TypeSystem *type_system;
  if (debug_map_symfile) {
type_system = debug_map_symfile->GetTypeSystemForLanguage(language);
  } else {
type_system = m_obj_file->GetModule()->GetTypeSystemForLanguage(language);
if (type_system)
  type_system->SetSymbolFile(this);
  }
  return type_system;
}

This allows one master DWARF file to use a bunch of other DWARF files to make 
one cohesive debug info. That is one approach you could use, but you would need 
to not affect the SymbolFileDWARFDebugMap with any changes you made.

One other idea is to keep all DWARF files separate and stand alone. Your main 
DWARF file with one or more DW_TAG_imported_unit and all DW_TAG_imported_unit 
referenced files, each as its own SymbolFileDWARF. Any reference to a 
DW_FORM_ref_alt would turn into a forward declaration in the current 
SymbolFileDWARF, so the ASTContext in each SymbolFileDWARF wouldn't know 
anything about the types, but we would need to add the DW_TAG_imported_unit 
object files to the target, and _they_ would know about any types they own. 
This way you could have multiple libraries that are the main top level DWARF 
files refer to a bunch of common DW_TAG_imported_unit files with type info and 
and those files would only be loaded once. We would rely on LLDB being able to 
track down the forward declared types later when the variables need to get