Re: [lldb-dev] FileSpec and normalization questions

[Greg Clayton via lldb-dev]

> On Apr 19, 2018, at 11:33 AM, Jim Ingham  wrote:
> 
> I can't see any reason why we benefit from having these differently spelled 
> equivalent paths floating around.  You want to be careful to preserve the 
> path syntax, since it would be weird to be cross debugging to a Windows 
> machine and have to type Posix paths.  But other than that, removing all the 
> extraneous cruft seems goodness to me.

We translate all windows slashes all the time in any FileSpec, so it doesn't 
really matter what the user types.

> Were you proposing just that you get the DWARF parser to do this, or were you 
> proposing that that become a requirement for SymbolFile parsers, so that we 
> can drop all the code that normalizes paths from debug information before 
> comparisons?  We still have to normalize paths from users, but it would be 
> nice not to have to do it if we know the source is debug info.
> 

I was proposing to fix FileSpec::SetFile(), when false is passed for the 
"resolve" parameter, to always normalize all paths. Or I could just fix this 
inside of SymbolFileDWARF. I would rather do this at the "FileSpec" level since 
it will normalize our comparisons and just remove a ton of issue we can run 
into. So I vote for fixing FileSpec to "do the right thing".

Greg

> Jim
> 
> 
>> On Apr 19, 2018, at 11:14 AM, Greg Clayton via lldb-dev 
>>  wrote:
>> 
>> We currently have DWARF that has a DW_AT_comp_dir that is set to "./" 
>> followed by any number of extra '/' characters. I would like to have this 
>> path normalized as we parse the DWARF so we don't end up with line tables 
>> with a variety of ".//+" prefixes on each source file.
>> 
>> While looking to solve this issue, I took a look at the functionality that 
>> is in FileSpec right now. In:
>> 
>> void FileSpec::SetFile(llvm::StringRef pathname, bool resolve, PathSyntax 
>> syntax);
>> 
>> 
>> This function always calls a cheaper normalize function:
>> 
>> namespace {
>>  void Normalize(llvm::SmallVectorImpl , FileSpec::PathSyntax 
>> syntax);
>> }
>> 
>> This function does nothing for posix paths, but switches backward slashes to 
>> forward slashes. 
>> 
>> We have a FileSpec function named FileSpec::GetNormalizedPath() which will 
>> do much more path normalization on a path by removing redundant "." and ".." 
>> and "//". 
>> 
>> I can fix my DWARF issue in a few ways:
>> 1 - fix FileSpec::SetFile() to still call ::Normalize() but have it do more 
>> work and have it normalize out the and redundant or relative path info
>> 2 - call FileSpec::GetNormalizedPath() on each comp dir before using it to 
>> actually normalize it
>> 
>> The main question is: do we want paths floating around LLDB that aren't 
>> normalized? It seems like having a mixture of theses path will lead to 
>> issues in LLDB so I would vote for solution #1.
>> 
>> Also, looking at the tests for normalizing paths I found the following pairs 
>> of pre-normalized and post-normalization paths for posix:
>> 
>>  {"//", "//"},
>>  {"//net", "//net"},
>> 
>> Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we 
>> reduce "//net" to "/net"?
>> 
>> Also I found:
>> 
>>  {"./foo", "foo"},
>> 
>> Do we prefer to not have "./foo" to stay as "./foo"? Seems like if users 
>> know that their debug info has line table entries that are "./foo/foo.c" 
>> that they might type this in:
>> 
>> (lldb) b ./foo/foo.c:12
>> 
>> But this will fail since it might not match the "foo/foo.c:12" that might 
>> result from path normalization. We don't normalize right now so it doesn't 
>> matter and things would match, but part of my fix is normalizing a path in 
>> the DWARF that is currently ".foo/foo.c" down to either 
>> "./foo/foo.c" or "foo/foo.c" so it will matter depending on what we decide 
>> here. 
>> 
>> Any input would be appreciated.
>> 
>> Greg Clayton
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Ted Woodward via lldb-dev]

Our Windows buildbots use msys for gnuisms. The makefiles in the test suite run 
fine with minimal modifications (just the object delete hack Zach put in to use 
del instead of rm; msys make doesn't accept cmd syntax while Cygwin make does). 
Now, that's using clang to build Hexagon binaries, but teaching the makefile to 
use cl syntax shouldn't be too hard. I've seen it done before; same makefile 
for windows and various unix derivatives, detect what OS you were running on 
and set CFLAGS/CXXFLAGS/LDFLAGS accordingly.

Ted

--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux 
Foundation Collaborative Project

> -Original Message-
> From: lldb-dev [mailto:lldb-dev-boun...@lists.llvm.org] On Behalf Of Pavel
> Labath via lldb-dev
> Sent: Thursday, April 19, 2018 12:45 PM
> To: Leonard Mosescu 
> Cc: aaron.lee.sm...@gmail.com; LLDB 
> Subject: Re: [lldb-dev] Proposal: Using LLD in tests
> 
> On Thu, 19 Apr 2018 at 18:19, Leonard Mosescu 
> wrote:
> 
> >>the PDB tests under lit/SymbolFile/PDB need a linker to produce
> >> the
> program database
> 
> 
> > With this proposal, would we preserve any coverage for MSVC produced
> debug information?
> 
> 
> Well.. the question there is what are you trying to test? Is it the fact your
> debugger works with a particular compiler+linker combination (note that those
> tests already compile with clang-cl), or that your pdb-parsing code is sane.
> (integration vs. regression test).
> 
> Historically we've only had the former kind of tests (dotest), and we've had 
> the
> ability (and used it) to run those tests against different kinds of 
> compilers. This
> is all nice, but it means that a specific test will be testing a different 
> thing for
> every person who runs it. That's why I would like to build up a suite of more
> regression-like tests (*). I would say that the tests under lit/*** should be
> regression tests and our goal should be to remove as many system
> dependencies as possible, and leave the job of testing integration with a
> specific toolchain to "dotest" tests (**).
> 
> Technically, the answer to your question is "no", because currently dotest 
> tests
> don't know how to work with cl+link. Making that work would be an interesting
> project (although a bit annoying as the Makefiles are full of gcc-isms).
> However, I don't think that should stop us here.
> 
> (*) Ideally I would like to leave even the compiler out of the equation for 
> these
> tests, and make it so that the tests always run on the exact same set of 
> bytes. I
> am hoping I will be able to write at least some tests using .s files. 
> However, I
> don't think I will do that for all of them, because these files can be
> long/verbose/tedious to write.
> 
> (**) However, even "dotest" tests should have a "default" mode which is as
> hermetic as possible.
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Zachary Turner via lldb-dev]

We actually use it in some places, but it's limited.  Before I did that was
when I added the PathSyntax to FileSpec which essentially servers the same
purpose.  We could in theory drop PathSyntax now that LLVM supports all of
the same functionality.  It's a pretty invasive refactor though which I
never had time to do.

I think I might have tried to replace some of the low level functions in
FileSpec with the LLVM equivalents and gotten a few test failures, but I
didn't have time to investigate.  It would be a worthwhile experiment for
someone to try again if they have some cycles.

On Thu, Apr 19, 2018 at 11:28 AM Jim Ingham  wrote:

>
> > On Apr 19, 2018, at 11:21 AM, Zachary Turner  wrote:
> >
> >
> >
> > On Thu, Apr 19, 2018 at 11:19 AM Jim Ingham via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > The last time I looked at the llvm functions they only support the path
> syntax of the llvm host, which won't do for lldb.  But maybe they have
> gotten more general recently?
> >
> >
> > Yes in fact I was the one who updated them to make them more general.
> You can now specify an enumeration parameter which will cause the algorithm
> to treat paths as either windows, posix, or whatever the host is.
>
> If I were guessing, I would have guessed that!
>
> Since you probably know how the llvm functions work better than most, were
> there technical reasons why, having done the work on the llvm side, you
> didn't adopt them in lldb?  Or was it just a matter of time?
>
> Jim
>
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Jim Ingham via lldb-dev]

I can't see any reason why we benefit from having these differently spelled 
equivalent paths floating around.  You want to be careful to preserve the path 
syntax, since it would be weird to be cross debugging to a Windows machine and 
have to type Posix paths.  But other than that, removing all the extraneous 
cruft seems goodness to me.

Were you proposing just that you get the DWARF parser to do this, or were you 
proposing that that become a requirement for SymbolFile parsers, so that we can 
drop all the code that normalizes paths from debug information before 
comparisons?  We still have to normalize paths from users, but it would be nice 
not to have to do it if we know the source is debug info.

Jim


> On Apr 19, 2018, at 11:14 AM, Greg Clayton via lldb-dev 
>  wrote:
> 
> We currently have DWARF that has a DW_AT_comp_dir that is set to "./" 
> followed by any number of extra '/' characters. I would like to have this 
> path normalized as we parse the DWARF so we don't end up with line tables 
> with a variety of ".//+" prefixes on each source file.
> 
> While looking to solve this issue, I took a look at the functionality that is 
> in FileSpec right now. In:
> 
> void FileSpec::SetFile(llvm::StringRef pathname, bool resolve, PathSyntax 
> syntax);
> 
> 
> This function always calls a cheaper normalize function:
> 
> namespace {
>   void Normalize(llvm::SmallVectorImpl , FileSpec::PathSyntax 
> syntax);
> }
> 
> This function does nothing for posix paths, but switches backward slashes to 
> forward slashes. 
> 
> We have a FileSpec function named FileSpec::GetNormalizedPath() which will do 
> much more path normalization on a path by removing redundant "." and ".." and 
> "//". 
> 
> I can fix my DWARF issue in a few ways:
> 1 - fix FileSpec::SetFile() to still call ::Normalize() but have it do more 
> work and have it normalize out the and redundant or relative path info
> 2 - call FileSpec::GetNormalizedPath() on each comp dir before using it to 
> actually normalize it
> 
> The main question is: do we want paths floating around LLDB that aren't 
> normalized? It seems like having a mixture of theses path will lead to issues 
> in LLDB so I would vote for solution #1.
> 
> Also, looking at the tests for normalizing paths I found the following pairs 
> of pre-normalized and post-normalization paths for posix:
> 
>   {"//", "//"},
>   {"//net", "//net"},
> 
> Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we reduce 
> "//net" to "/net"?
> 
> Also I found:
> 
>   {"./foo", "foo"},
> 
> Do we prefer to not have "./foo" to stay as "./foo"? Seems like if users know 
> that their debug info has line table entries that are "./foo/foo.c" that they 
> might type this in:
> 
> (lldb) b ./foo/foo.c:12
> 
> But this will fail since it might not match the "foo/foo.c:12" that might 
> result from path normalization. We don't normalize right now so it doesn't 
> matter and things would match, but part of my fix is normalizing a path in 
> the DWARF that is currently ".foo/foo.c" down to either "./foo/foo.c" 
> or "foo/foo.c" so it will matter depending on what we decide here. 
> 
> Any input would be appreciated.
> 
> Greg Clayton
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Advice on architectures with multiple address spaces

[Ted Woodward via lldb-dev]

Hexagon has a single address space, so we don't need to do anything like this.

When I worked on Motorola 56xxx DSPs we had memory spaces, but we didn't use 
RSP. We had our own protocol that used a struct for addreses, with the space 
(an enum, defined per supported core) and a uint32_t (later 2 of them) for the 
address.

Ted

--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux 
Foundation Collaborative Project

> -Original Message-
> From: lldb-dev [mailto:lldb-dev-boun...@lists.llvm.org] On Behalf Of Greg
> Clayton via lldb-dev
> Sent: Thursday, April 19, 2018 11:45 AM
> To: Zdenek Prikryl 
> Cc: lldb-dev@lists.llvm.org
> Subject: Re: [lldb-dev] Advice on architectures with multiple address spaces
You might ask
> the Hexagon folks if they have done anything in case they already support this
> is some shape or form.
> 
> Greg Clayton
> 



___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Jim Ingham via lldb-dev]

> On Apr 19, 2018, at 11:21 AM, Zachary Turner  wrote:
> 
> 
> 
> On Thu, Apr 19, 2018 at 11:19 AM Jim Ingham via lldb-dev 
>  wrote:
> The last time I looked at the llvm functions they only support the path 
> syntax of the llvm host, which won't do for lldb.  But maybe they have gotten 
> more general recently?
> 
> 
> Yes in fact I was the one who updated them to make them more general.  You 
> can now specify an enumeration parameter which will cause the algorithm to 
> treat paths as either windows, posix, or whatever the host is. 

If I were guessing, I would have guessed that!

Since you probably know how the llvm functions work better than most, were 
there technical reasons why, having done the work on the llvm side, you didn't 
adopt them in lldb?  Or was it just a matter of time?

Jim


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Advice on architectures with multiple address spaces

[Jim Ingham via lldb-dev]

> On Apr 19, 2018, at 10:54 AM, Greg Clayton  wrote:
> 
> 
> 
>> On Apr 19, 2018, at 10:35 AM, Jim Ingham  wrote:
>> 
>> 
>> 
>>> On Apr 19, 2018, at 9:44 AM, Greg Clayton via lldb-dev 
>>>  wrote:
>>> 
>>> 
>>> 
 On Apr 19, 2018, at 6:51 AM, Zdenek Prikryl via lldb-dev 
  wrote:
 
 Hi lldb developers,
 
 I've been researching using lldb + gdbserver stub that is based on Harvard 
 architecture with multiple address spaces (one program, multiple data). 
 The commonly adopted approach is that everything is mapped to a single 
 "virtual" address space. The stub reads/writes from/to the right memory 
 based on the "virtual" addresses. But I'd like to use real addresses with 
 address space id instead. So, I've started looking at what has to be 
 changed.
 
 I've enhanced read/write commands (e.g. memory read --as  ...) and RSP 
 protocol (new packet) so that the stub can read/write properly. That 
 wasn't that complicated.
>>> 
>>> It might be nice to add a new RSP protocol packet that asks for the address 
>>> space names/values:
>>> 
>>> qGetAddressSpaces
>>> 
>>> which would return something like:
>>> 
>>> 1:text;2:data1,3:data2
>>> 
>>> or it would return not supported. If we get a valid return value from 
>>> qGetAddressSpaces, then it enables the use of the new packet you added 
>>> above. Else it defaults to using the old memory read functions.
>>> 
>>> 
 
 Now I've hit an issue with expressions (LLVMUserExpression.cpp) and local 
 variables (DWARFExpressions.cpp). There is a lot of memory read/write 
 functions that take just an address argument. Is the only way to go to 
 patch all these calls? Has anybody solved it differently?
>>> 
>>> My quick take is that any APIs that take just a lldb::addr_t would need to 
>>> take something like:
>>> 
>>> struct SpaceAddress {
>>> static constexpr uint32_t kNoSpace = 0;
>>> lldb::addr_t addr;
>>> uint32_t space;
>>> };
>>> 
>> 
>> I'm curious why you are suggesting another kind of address, rather than 
>> adding this functionality to Address?  When you actually go to resolve an 
>> Address in a target with a process you should have everything you need to 
>> know to give it the proper space.  Then fixing the expression evaluator (and 
>> anything else that needs fixing) would be a matter of consistently using 
>> Address rather than lldb::addr_t.  That seems general goodness, since 
>> converting to an lldb::addr_t loses information.
> 
> If we accept lldb_private::Address in all APIs that take a lldb::addr_t 
> currently, then we need to always be able to get to the target in case we 
> need to add code to resolve the address everywhere. I am thinking of 
> SpaceAddress as an augmented lldb::addr_t instead of a section + offset style 
> address. Also, there will be addresses in the code and data that do not exist 
> in actual sections. Not saying that you couldn't use lldb_private::Address. I 
> am open to suggestions though. So your though it remove all API that take 
> lldb::addr_t and use lldb_private::Address everywhere all the time?

It has always bugged me that we have these two ways of specifying addresses.  
Are there many/any places that have to resolve an Address to a real address in 
a process that don't have a Target readily available?  That would surprise me.  
I would much rather centralize on one way than adding a third.

Jim


>> 
>> Jim
>> 
>> 
>>> We would need a default value for "space" (feel free to rename) that 
>>> indicates the default address space as most of our architectures would not 
>>> need this support. If we added a constructor like:
>>> 
>>> SpaceAddress(lldb::addr_t a) : addr(a), space(kNoSpace) {}
>>> 
>>> Then all usages of the APIs that used to take just a "lldb::addr_t" would 
>>> implicitly call this constructor and continue to act as needed. Then we 
>>> would need to allow lldb_private::Address objects to resolve to a 
>>> SpaceAddress:
>>> 
>>> SpaceAddress lldb_private::Address::GetSpaceAddress(Target *target) const;
>>> 
>>> Since each lldb_private::Address has a section and each section knows its 
>>> address space. Then the tricky part is finding all locations in the 
>>> expression parser and converting those to track and use SpaceAddress. We 
>>> would probably need to modify the allocate memory packets in the RSP 
>>> protocol to be able to allocate memory in any address space as well.
>>> 
>>> I didn't spend much time think about correct names above, so feel free to 
>>> suggest alternate naming. 
>>> 
>>> Best advice:
>>> - make things "just work" to keep changes to a minimum and allowing 
>>> lldb::addr_t to implicitly convert to a SpaceAddress easily
>>> - when modifying RSP, make sure to check for existence of new feature 
>>> before enabling it
>>> - query for address space names so when we dump SpaceAddress we can show 
>>> 

Re: [lldb-dev] FileSpec and normalization questions

[Zachary Turner via lldb-dev]

On Thu, Apr 19, 2018 at 11:19 AM Jim Ingham via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> The last time I looked at the llvm functions they only support the path
> syntax of the llvm host, which won't do for lldb.  But maybe they have
> gotten more general recently?
>
>
Yes in fact I was the one who updated them to make them more general.  You
can now specify an enumeration parameter which will cause the algorithm to
treat paths as either windows, posix, or whatever the host is.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Zachary Turner via lldb-dev]

On Thu, Apr 19, 2018 at 11:14 AM Greg Clayton via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

>
> Also, looking at the tests for normalizing paths I found the following
> pairs of pre-normalized and post-normalization paths for posix:
>
>   {"//", "//"},
>   {"//net", "//net"},
>
> Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we
> reduce "//net" to "/net"?
>
>
I don't know what the author of this test had in mind, but from the POSIX
spec:

http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_11


> A pathname that begins with two successive slashes may be interpreted in
an implementation-defined manner, although more than two leading slashes
shall be treated as a single slash.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Jim Ingham via lldb-dev]

The last time I looked at the llvm functions they only support the path syntax 
of the llvm host, which won't do for lldb.  But maybe they have gotten more 
general recently?

Jim


> On Apr 19, 2018, at 11:16 AM, Davide Italiano via lldb-dev 
>  wrote:
> 
> IIRC We have path normalization functions in llvm, have you looked at them?
> 
> Thanks,
> 
> --
> Davide
> 
> On Thu, Apr 19, 2018 at 11:14 AM, Greg Clayton via lldb-dev
>  wrote:
>> We currently have DWARF that has a DW_AT_comp_dir that is set to "./"
>> followed by any number of extra '/' characters. I would like to have this
>> path normalized as we parse the DWARF so we don't end up with line tables
>> with a variety of ".//+" prefixes on each source file.
>> 
>> While looking to solve this issue, I took a look at the functionality that
>> is in FileSpec right now. In:
>> 
>> void FileSpec::SetFile(llvm::StringRef pathname, bool resolve, PathSyntax
>> syntax);
>> 
>> 
>> This function always calls a cheaper normalize function:
>> 
>> namespace {
>>  void Normalize(llvm::SmallVectorImpl , FileSpec::PathSyntax
>> syntax);
>> }
>> 
>> This function does nothing for posix paths, but switches backward slashes to
>> forward slashes.
>> 
>> We have a FileSpec function named FileSpec::GetNormalizedPath() which will
>> do much more path normalization on a path by removing redundant "." and ".."
>> and "//".
>> 
>> I can fix my DWARF issue in a few ways:
>> 1 - fix FileSpec::SetFile() to still call ::Normalize() but have it do more
>> work and have it normalize out the and redundant or relative path info
>> 2 - call FileSpec::GetNormalizedPath() on each comp dir before using it to
>> actually normalize it
>> 
>> The main question is: do we want paths floating around LLDB that aren't
>> normalized? It seems like having a mixture of theses path will lead to
>> issues in LLDB so I would vote for solution #1.
>> 
>> Also, looking at the tests for normalizing paths I found the following pairs
>> of pre-normalized and post-normalization paths for posix:
>> 
>>  {"//", "//"},
>>  {"//net", "//net"},
>> 
>> Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we
>> reduce "//net" to "/net"?
>> 
>> Also I found:
>> 
>>  {"./foo", "foo"},
>> 
>> Do we prefer to not have "./foo" to stay as "./foo"? Seems like if users
>> know that their debug info has line table entries that are "./foo/foo.c"
>> that they might type this in:
>> 
>> (lldb) b ./foo/foo.c:12
>> 
>> But this will fail since it might not match the "foo/foo.c:12" that might
>> result from path normalization. We don't normalize right now so it doesn't
>> matter and things would match, but part of my fix is normalizing a path in
>> the DWARF that is currently ".foo/foo.c" down to either
>> "./foo/foo.c" or "foo/foo.c" so it will matter depending on what we decide
>> here.
>> 
>> Any input would be appreciated.
>> 
>> Greg Clayton
>> 
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] FileSpec and normalization questions

[Davide Italiano via lldb-dev]

IIRC We have path normalization functions in llvm, have you looked at them?

Thanks,

--
Davide

On Thu, Apr 19, 2018 at 11:14 AM, Greg Clayton via lldb-dev
 wrote:
> We currently have DWARF that has a DW_AT_comp_dir that is set to "./"
> followed by any number of extra '/' characters. I would like to have this
> path normalized as we parse the DWARF so we don't end up with line tables
> with a variety of ".//+" prefixes on each source file.
>
> While looking to solve this issue, I took a look at the functionality that
> is in FileSpec right now. In:
>
> void FileSpec::SetFile(llvm::StringRef pathname, bool resolve, PathSyntax
> syntax);
>
>
> This function always calls a cheaper normalize function:
>
> namespace {
>   void Normalize(llvm::SmallVectorImpl , FileSpec::PathSyntax
> syntax);
> }
>
> This function does nothing for posix paths, but switches backward slashes to
> forward slashes.
>
> We have a FileSpec function named FileSpec::GetNormalizedPath() which will
> do much more path normalization on a path by removing redundant "." and ".."
> and "//".
>
> I can fix my DWARF issue in a few ways:
> 1 - fix FileSpec::SetFile() to still call ::Normalize() but have it do more
> work and have it normalize out the and redundant or relative path info
> 2 - call FileSpec::GetNormalizedPath() on each comp dir before using it to
> actually normalize it
>
> The main question is: do we want paths floating around LLDB that aren't
> normalized? It seems like having a mixture of theses path will lead to
> issues in LLDB so I would vote for solution #1.
>
> Also, looking at the tests for normalizing paths I found the following pairs
> of pre-normalized and post-normalization paths for posix:
>
>   {"//", "//"},
>   {"//net", "//net"},
>
> Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we
> reduce "//net" to "/net"?
>
> Also I found:
>
>   {"./foo", "foo"},
>
> Do we prefer to not have "./foo" to stay as "./foo"? Seems like if users
> know that their debug info has line table entries that are "./foo/foo.c"
> that they might type this in:
>
> (lldb) b ./foo/foo.c:12
>
> But this will fail since it might not match the "foo/foo.c:12" that might
> result from path normalization. We don't normalize right now so it doesn't
> matter and things would match, but part of my fix is normalizing a path in
> the DWARF that is currently ".foo/foo.c" down to either
> "./foo/foo.c" or "foo/foo.c" so it will matter depending on what we decide
> here.
>
> Any input would be appreciated.
>
> Greg Clayton
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] FileSpec and normalization questions

[Greg Clayton via lldb-dev]

We currently have DWARF that has a DW_AT_comp_dir that is set to "./" followed 
by any number of extra '/' characters. I would like to have this path 
normalized as we parse the DWARF so we don't end up with line tables with a 
variety of ".//+" prefixes on each source file.

While looking to solve this issue, I took a look at the functionality that is 
in FileSpec right now. In:

void FileSpec::SetFile(llvm::StringRef pathname, bool resolve, PathSyntax 
syntax);


This function always calls a cheaper normalize function:

namespace {
  void Normalize(llvm::SmallVectorImpl , FileSpec::PathSyntax 
syntax);
}

This function does nothing for posix paths, but switches backward slashes to 
forward slashes. 

We have a FileSpec function named FileSpec::GetNormalizedPath() which will do 
much more path normalization on a path by removing redundant "." and ".." and 
"//". 

I can fix my DWARF issue in a few ways:
1 - fix FileSpec::SetFile() to still call ::Normalize() but have it do more 
work and have it normalize out the and redundant or relative path info
2 - call FileSpec::GetNormalizedPath() on each comp dir before using it to 
actually normalize it

The main question is: do we want paths floating around LLDB that aren't 
normalized? It seems like having a mixture of theses path will lead to issues 
in LLDB so I would vote for solution #1.

Also, looking at the tests for normalizing paths I found the following pairs of 
pre-normalized and post-normalization paths for posix:

  {"//", "//"},
  {"//net", "//net"},

Why wouldn't we reduce "//" to just "/" for posix? And why wouldn't we reduce 
"//net" to "/net"?

Also I found:

  {"./foo", "foo"},

Do we prefer to not have "./foo" to stay as "./foo"? Seems like if users know 
that their debug info has line table entries that are "./foo/foo.c" that they 
might type this in:

(lldb) b ./foo/foo.c:12

But this will fail since it might not match the "foo/foo.c:12" that might 
result from path normalization. We don't normalize right now so it doesn't 
matter and things would match, but part of my fix is normalizing a path in the 
DWARF that is currently ".foo/foo.c" down to either "./foo/foo.c" or 
"foo/foo.c" so it will matter depending on what we decide here. 

Any input would be appreciated.

Greg Clayton

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Leonard Mosescu via lldb-dev]

>
> we have some good coverage there that our PDBs are "as good as" Microsoft
> PDBs, and in the future we have plans to have a debug info test suite that
> tests LLD-generated PDBs with Microsoft debuggers.
>

Thanks Zach. What I was asking is exactly the other half of this equation.
Testing LLDB with MSVC produced PDBs should be complementary and I doubt we
can get full coverage by just testing from the other direction (it's
plausible that even with equivalent semantic information, certain patterns
may only occur in MSVC produced debug information)

IMO reliable support for MSVC generated binaries and debug information is
critical since even with a LLVM/LLD toolchain you'll still have system/3rd
party modules, right?

Anyway, I was curious if any of this is in scope for Pavel's proposal and
the answer seems to be no, thanks everyone.

On Thu, Apr 19, 2018 at 10:44 AM, Pavel Labath  wrote:

> On Thu, 19 Apr 2018 at 18:19, Leonard Mosescu  wrote:
>
> >>the PDB tests under lit/SymbolFile/PDB need a linker to produce the
> program database
>
>
> > With this proposal, would we preserve any coverage for MSVC produced
> debug information?
>
>
> Well.. the question there is what are you trying to test? Is it the fact
> your debugger works with a particular compiler+linker
> combination (note that those tests already compile with clang-cl), or that
> your pdb-parsing code is sane. (integration vs. regression test).
>
> Historically we've only had the former kind of tests (dotest), and we've
> had the ability (and used it) to run those tests against different kinds of
> compilers. This is all nice, but it means that a specific test will be
> testing a different thing for every person who runs it. That's why I would
> like to build up a suite of more regression-like tests (*). I would say
> that the tests under lit/*** should be regression tests and our goal should
> be to remove as many system dependencies as possible, and leave the job of
> testing integration with a specific toolchain to "dotest" tests (**).
>
> Technically, the answer to your question is "no", because currently dotest
> tests don't know how to work with cl+link. Making that work would be an
> interesting project (although a bit annoying as the Makefiles are full of
> gcc-isms). However, I don't think that should stop us here.
>
> (*) Ideally I would like to leave even the compiler out of the equation for
> these tests, and make it so that the tests always run on the exact same set
> of bytes. I am hoping I will be able to write at least some tests using .s
> files. However, I don't think I will do that for all of them, because these
> files can be long/verbose/tedious to write.
>
> (**) However, even "dotest" tests should have a "default" mode which is as
> hermetic as possible.
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Advice on architectures with multiple address spaces

[Greg Clayton via lldb-dev]

> On Apr 19, 2018, at 10:35 AM, Jim Ingham  wrote:
> 
> 
> 
>> On Apr 19, 2018, at 9:44 AM, Greg Clayton via lldb-dev 
>>  wrote:
>> 
>> 
>> 
>>> On Apr 19, 2018, at 6:51 AM, Zdenek Prikryl via lldb-dev 
>>>  wrote:
>>> 
>>> Hi lldb developers,
>>> 
>>> I've been researching using lldb + gdbserver stub that is based on Harvard 
>>> architecture with multiple address spaces (one program, multiple data). The 
>>> commonly adopted approach is that everything is mapped to a single 
>>> "virtual" address space. The stub reads/writes from/to the right memory 
>>> based on the "virtual" addresses. But I'd like to use real addresses with 
>>> address space id instead. So, I've started looking at what has to be 
>>> changed.
>>> 
>>> I've enhanced read/write commands (e.g. memory read --as  ...) and RSP 
>>> protocol (new packet) so that the stub can read/write properly. That wasn't 
>>> that complicated.
>> 
>> It might be nice to add a new RSP protocol packet that asks for the address 
>> space names/values:
>> 
>> qGetAddressSpaces
>> 
>> which would return something like:
>> 
>> 1:text;2:data1,3:data2
>> 
>> or it would return not supported. If we get a valid return value from 
>> qGetAddressSpaces, then it enables the use of the new packet you added 
>> above. Else it defaults to using the old memory read functions.
>> 
>> 
>>> 
>>> Now I've hit an issue with expressions (LLVMUserExpression.cpp) and local 
>>> variables (DWARFExpressions.cpp). There is a lot of memory read/write 
>>> functions that take just an address argument. Is the only way to go to 
>>> patch all these calls? Has anybody solved it differently?
>> 
>> My quick take is that any APIs that take just a lldb::addr_t would need to 
>> take something like:
>> 
>> struct SpaceAddress {
>> static constexpr uint32_t kNoSpace = 0;
>> lldb::addr_t addr;
>> uint32_t space;
>> };
>> 
> 
> I'm curious why you are suggesting another kind of address, rather than 
> adding this functionality to Address?  When you actually go to resolve an 
> Address in a target with a process you should have everything you need to 
> know to give it the proper space.  Then fixing the expression evaluator (and 
> anything else that needs fixing) would be a matter of consistently using 
> Address rather than lldb::addr_t.  That seems general goodness, since 
> converting to an lldb::addr_t loses information.

If we accept lldb_private::Address in all APIs that take a lldb::addr_t 
currently, then we need to always be able to get to the target in case we need 
to add code to resolve the address everywhere. I am thinking of SpaceAddress as 
an augmented lldb::addr_t instead of a section + offset style address. Also, 
there will be addresses in the code and data that do not exist in actual 
sections. Not saying that you couldn't use lldb_private::Address. I am open to 
suggestions though. So your though it remove all API that take lldb::addr_t and 
use lldb_private::Address everywhere all the time?
> 
> Jim
> 
> 
>> We would need a default value for "space" (feel free to rename) that 
>> indicates the default address space as most of our architectures would not 
>> need this support. If we added a constructor like:
>> 
>> SpaceAddress(lldb::addr_t a) : addr(a), space(kNoSpace) {}
>> 
>> Then all usages of the APIs that used to take just a "lldb::addr_t" would 
>> implicitly call this constructor and continue to act as needed. Then we 
>> would need to allow lldb_private::Address objects to resolve to a 
>> SpaceAddress:
>> 
>> SpaceAddress lldb_private::Address::GetSpaceAddress(Target *target) const;
>> 
>> Since each lldb_private::Address has a section and each section knows its 
>> address space. Then the tricky part is finding all locations in the 
>> expression parser and converting those to track and use SpaceAddress. We 
>> would probably need to modify the allocate memory packets in the RSP 
>> protocol to be able to allocate memory in any address space as well.
>> 
>> I didn't spend much time think about correct names above, so feel free to 
>> suggest alternate naming. 
>> 
>> Best advice:
>> - make things "just work" to keep changes to a minimum and allowing 
>> lldb::addr_t to implicitly convert to a SpaceAddress easily
>> - when modifying RSP, make sure to check for existence of new feature before 
>> enabling it
>> - query for address space names so when we dump SpaceAddress we can show 
>> something that means something to the user. This means we would need to 
>> query the address space names from the current lldb_private::Process for 
>> display.
>> 
>> Submitting might go easier if we break it down into chunks:
>> 1 - add SpaceAddress and modify all needed APIs to use it
>> 2 - add ProcessGDBRemote changes that enable this support
>> 
>> It will be great to support this as a first class citizen within LLDB. You 
>> might ask the Hexagon folks if they have done anything in case they already 

Re: [lldb-dev] Proposal: Using LLD in tests

[Pavel Labath via lldb-dev]

On Thu, 19 Apr 2018 at 18:19, Leonard Mosescu  wrote:

>>the PDB tests under lit/SymbolFile/PDB need a linker to produce the
program database


> With this proposal, would we preserve any coverage for MSVC produced
debug information?


Well.. the question there is what are you trying to test? Is it the fact
your debugger works with a particular compiler+linker
combination (note that those tests already compile with clang-cl), or that
your pdb-parsing code is sane. (integration vs. regression test).

Historically we've only had the former kind of tests (dotest), and we've
had the ability (and used it) to run those tests against different kinds of
compilers. This is all nice, but it means that a specific test will be
testing a different thing for every person who runs it. That's why I would
like to build up a suite of more regression-like tests (*). I would say
that the tests under lit/*** should be regression tests and our goal should
be to remove as many system dependencies as possible, and leave the job of
testing integration with a specific toolchain to "dotest" tests (**).

Technically, the answer to your question is "no", because currently dotest
tests don't know how to work with cl+link. Making that work would be an
interesting project (although a bit annoying as the Makefiles are full of
gcc-isms). However, I don't think that should stop us here.

(*) Ideally I would like to leave even the compiler out of the equation for
these tests, and make it so that the tests always run on the exact same set
of bytes. I am hoping I will be able to write at least some tests using .s
files. However, I don't think I will do that for all of them, because these
files can be long/verbose/tedious to write.

(**) However, even "dotest" tests should have a "default" mode which is as
hermetic as possible.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Zachary Turner via lldb-dev]

We already don't really have any.

All of the makefiles that build the inferiors use a gcc style command line
syntax, and the way we made that work on Windows is to use clang for the
compiler.  So MSVC is already out of the equation.

There may be one or two instances where we check in a PDB file, but that
doesn't really scale.

I've been working a lot on the PDB support in LLVM and we have some good
coverage there that our PDBs are "as good as" Microsoft PDBs, and in the
future we have plans to have a debug info test suite that tests
LLD-generated PDBs with Microsoft debuggers.  If we have coverage of that,
I think that should be sufficient and maybe not worth complicating the LLDB
test suite matrix over trying to test debugging with MSVC generated PDBs.

On Thu, Apr 19, 2018 at 10:19 AM Leonard Mosescu via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

>  the PDB tests under lit/SymbolFile/PDB need a linker to produce the program
>> database
>
>
> With this proposal, would we preserve any coverage for MSVC produced debug
> information?
>
> On Thu, Apr 19, 2018 at 9:47 AM, Greg Clayton via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>>
>>
>> > On Apr 19, 2018, at 9:39 AM, Pavel Labath  wrote:
>> >
>> > Yes, I considered the lld+Mach-O situation. I understand that it does
>> not
>> > work very well, but I don't know what exactly that means.
>> >
>> > However, I am not sure that we even need a linker for Mach-O. As I
>> > understand it, in a darwin world, the linker does not even touch the
>> debug
>> > info, and lldb is already capable of reading the debug info from .o
>> files.
>> > So at least for non dsym-case it seems to me we should be good. It may
>> be
>> > possible we need a linker for the dSYM (i don't know exactly how that
>> > works), in which case we may just not be able to test dsym this way, but
>> > even that will be better than what we have now.
>>
>> The problem is the linker must produce a debug map in the symbol table
>> that contains the addresses of all linked items. So using LLD won't work
>> unless that is fully supported as LLDB won't be able to like the debug info
>> on the fly, nor will dsymutil have a debug map to use in order to make a
>> dSYM file.
>>
>> >
>> > PS: I am not proposing to do anything to the existing dotest tests (they
>> > need access to a running process anyway, so playing with the linker will
>> > not help). I am just trying to make new debug-info specific tests
>> > universally available.
>>
>> What I was trying to say is this is fine if we always compile/link as an
>> ELF file with some triple that uses ELF for these tests. They just probably
>> shouldn't be mach-o.
>>
>> Greg
>>
>> >
>> >
>> > On Thu, 19 Apr 2018 at 17:24, Greg Clayton  wrote:
>> >
>> >> The last I knew LLD doesn't work on mach-o very well, so be sure to not
>> > require LLD for linking any Darwin executables.
>> >
>> >>> On Apr 19, 2018, at 6:42 AM, Pavel Labath via lldb-dev <
>> > lldb-dev@lists.llvm.org> wrote:
>> >>>
>> >>> Hello all,
>> >>>
>> >>> currently we have a couple of tests, in-tree or under review, which
>> are
>> >>> very close to being host-independent. The only part they are missing
>> is
>> > the
>> >>> ability to link a intermediate object file:
>> >>> - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
>> >>> resolve relocations in the debug info (*)
>> >>> - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
>> >>> program database.
>> >>>
>> >>> I think it would be great if everyone were able to run these tests and
>> >>> verify they don't regress them before they actually push a patch.
>> >>>
>> >>> Apart from that, I have started looking at writing some non-execution
>> > debug
>> >>> info (**) tests as a part of adding DWARF v5 accelerator table support
>> > to
>> >>> lldb (both to test the new implementation, and to make sure I don't
>> > regress
>> >>> existing ones). Ideally I'd like to make sure that everyone is able to
>> > run
>> >>> them, regardless of their primary (or only) development platform. For
>> > this,
>> >>> I also need a linker capable of running everywhere (*)
>> >>>
>> >>> To achieve these goals, I'd like to propose that we add LLD as a
>> > (optional,
>> >>> but strongly recommended) dependency for running tests and start using
>> > it
>> >>> in the tests I mention. Doing this would optional in the sense that
>> the
>> >>> tests would be marked "REQUIRED: lld", and simply skipped if lld is
>> not
>> >>> available (so the tests would still be green). I say "strongly
>> > recommended"
>> >>> because not having lld checked out should not be an excuse for
>> breaking
>> > the
>> >>> test, and the patch author should pro-actively revert a patch which
>> > breaks
>> >>> such tests and investigate.
>> >>>
>> >>> I hope this proposal is not too controversial. LLD is already required
>> > on
>> >>> windows to run dotest tests. Also, all monorepo users likely have it
>> >>> 

Re: [lldb-dev] Advice on architectures with multiple address spaces

[Jim Ingham via lldb-dev]

> On Apr 19, 2018, at 9:44 AM, Greg Clayton via lldb-dev 
>  wrote:
> 
> 
> 
>> On Apr 19, 2018, at 6:51 AM, Zdenek Prikryl via lldb-dev 
>>  wrote:
>> 
>> Hi lldb developers,
>> 
>> I've been researching using lldb + gdbserver stub that is based on Harvard 
>> architecture with multiple address spaces (one program, multiple data). The 
>> commonly adopted approach is that everything is mapped to a single "virtual" 
>> address space. The stub reads/writes from/to the right memory based on the 
>> "virtual" addresses. But I'd like to use real addresses with address space 
>> id instead. So, I've started looking at what has to be changed.
>> 
>> I've enhanced read/write commands (e.g. memory read --as  ...) and RSP 
>> protocol (new packet) so that the stub can read/write properly. That wasn't 
>> that complicated.
> 
> It might be nice to add a new RSP protocol packet that asks for the address 
> space names/values:
> 
> qGetAddressSpaces
> 
> which would return something like:
> 
> 1:text;2:data1,3:data2
> 
> or it would return not supported. If we get a valid return value from 
> qGetAddressSpaces, then it enables the use of the new packet you added above. 
> Else it defaults to using the old memory read functions.
> 
> 
>> 
>> Now I've hit an issue with expressions (LLVMUserExpression.cpp) and local 
>> variables (DWARFExpressions.cpp). There is a lot of memory read/write 
>> functions that take just an address argument. Is the only way to go to patch 
>> all these calls? Has anybody solved it differently?
> 
> My quick take is that any APIs that take just a lldb::addr_t would need to 
> take something like:
> 
> struct SpaceAddress {
>  static constexpr uint32_t kNoSpace = 0;
>  lldb::addr_t addr;
>  uint32_t space;
> };
> 

I'm curious why you are suggesting another kind of address, rather than adding 
this functionality to Address?  When you actually go to resolve an Address in a 
target with a process you should have everything you need to know to give it 
the proper space.  Then fixing the expression evaluator (and anything else that 
needs fixing) would be a matter of consistently using Address rather than 
lldb::addr_t.  That seems general goodness, since converting to an lldb::addr_t 
loses information.

Jim


> We would need a default value for "space" (feel free to rename) that 
> indicates the default address space as most of our architectures would not 
> need this support. If we added a constructor like:
> 
> SpaceAddress(lldb::addr_t a) : addr(a), space(kNoSpace) {}
> 
> Then all usages of the APIs that used to take just a "lldb::addr_t" would 
> implicitly call this constructor and continue to act as needed. Then we would 
> need to allow lldb_private::Address objects to resolve to a SpaceAddress:
> 
>  SpaceAddress lldb_private::Address::GetSpaceAddress(Target *target) const;
> 
> Since each lldb_private::Address has a section and each section knows its 
> address space. Then the tricky part is finding all locations in the 
> expression parser and converting those to track and use SpaceAddress. We 
> would probably need to modify the allocate memory packets in the RSP protocol 
> to be able to allocate memory in any address space as well.
> 
> I didn't spend much time think about correct names above, so feel free to 
> suggest alternate naming. 
> 
> Best advice:
> - make things "just work" to keep changes to a minimum and allowing 
> lldb::addr_t to implicitly convert to a SpaceAddress easily
> - when modifying RSP, make sure to check for existence of new feature before 
> enabling it
> - query for address space names so when we dump SpaceAddress we can show 
> something that means something to the user. This means we would need to query 
> the address space names from the current lldb_private::Process for display.
> 
> Submitting might go easier if we break it down into chunks:
> 1 - add SpaceAddress and modify all needed APIs to use it
> 2 - add ProcessGDBRemote changes that enable this support
> 
> It will be great to support this as a first class citizen within LLDB. You 
> might ask the Hexagon folks if they have done anything in case they already 
> support this is some shape or form.
> 
> Greg Clayton
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Leonard Mosescu via lldb-dev]

>
>  the PDB tests under lit/SymbolFile/PDB need a linker to produce the program
> database


With this proposal, would we preserve any coverage for MSVC produced debug
information?

On Thu, Apr 19, 2018 at 9:47 AM, Greg Clayton via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

>
>
> > On Apr 19, 2018, at 9:39 AM, Pavel Labath  wrote:
> >
> > Yes, I considered the lld+Mach-O situation. I understand that it does not
> > work very well, but I don't know what exactly that means.
> >
> > However, I am not sure that we even need a linker for Mach-O. As I
> > understand it, in a darwin world, the linker does not even touch the
> debug
> > info, and lldb is already capable of reading the debug info from .o
> files.
> > So at least for non dsym-case it seems to me we should be good. It may be
> > possible we need a linker for the dSYM (i don't know exactly how that
> > works), in which case we may just not be able to test dsym this way, but
> > even that will be better than what we have now.
>
> The problem is the linker must produce a debug map in the symbol table
> that contains the addresses of all linked items. So using LLD won't work
> unless that is fully supported as LLDB won't be able to like the debug info
> on the fly, nor will dsymutil have a debug map to use in order to make a
> dSYM file.
>
> >
> > PS: I am not proposing to do anything to the existing dotest tests (they
> > need access to a running process anyway, so playing with the linker will
> > not help). I am just trying to make new debug-info specific tests
> > universally available.
>
> What I was trying to say is this is fine if we always compile/link as an
> ELF file with some triple that uses ELF for these tests. They just probably
> shouldn't be mach-o.
>
> Greg
>
> >
> >
> > On Thu, 19 Apr 2018 at 17:24, Greg Clayton  wrote:
> >
> >> The last I knew LLD doesn't work on mach-o very well, so be sure to not
> > require LLD for linking any Darwin executables.
> >
> >>> On Apr 19, 2018, at 6:42 AM, Pavel Labath via lldb-dev <
> > lldb-dev@lists.llvm.org> wrote:
> >>>
> >>> Hello all,
> >>>
> >>> currently we have a couple of tests, in-tree or under review, which are
> >>> very close to being host-independent. The only part they are missing is
> > the
> >>> ability to link a intermediate object file:
> >>> - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
> >>> resolve relocations in the debug info (*)
> >>> - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
> >>> program database.
> >>>
> >>> I think it would be great if everyone were able to run these tests and
> >>> verify they don't regress them before they actually push a patch.
> >>>
> >>> Apart from that, I have started looking at writing some non-execution
> > debug
> >>> info (**) tests as a part of adding DWARF v5 accelerator table support
> > to
> >>> lldb (both to test the new implementation, and to make sure I don't
> > regress
> >>> existing ones). Ideally I'd like to make sure that everyone is able to
> > run
> >>> them, regardless of their primary (or only) development platform. For
> > this,
> >>> I also need a linker capable of running everywhere (*)
> >>>
> >>> To achieve these goals, I'd like to propose that we add LLD as a
> > (optional,
> >>> but strongly recommended) dependency for running tests and start using
> > it
> >>> in the tests I mention. Doing this would optional in the sense that the
> >>> tests would be marked "REQUIRED: lld", and simply skipped if lld is not
> >>> available (so the tests would still be green). I say "strongly
> > recommended"
> >>> because not having lld checked out should not be an excuse for breaking
> > the
> >>> test, and the patch author should pro-actively revert a patch which
> > breaks
> >>> such tests and investigate.
> >>>
> >>> I hope this proposal is not too controversial. LLD is already required
> > on
> >>> windows to run dotest tests. Also, all monorepo users likely have it
> >>> already, or it is very easy for them to enable it. For non-monorepo
> > users
> >>> it should be a matter of checking out one extra repository. Please Let
> > me
> >>> know what you think.
> >>>
> >>> pavel
> >>>
> >>> (*) our ELF parser has very limited support for applying debug info
> >>> relocations -- it only works for x86, and only a couple of relocations
> > are
> >>> currently implemented. It would be possible to remove the linker
> > dependency
> >>> by implementing these (essentially, doing the link ourselves -- this is
> >>> what llvm does), but given the large number of architectures and
> > relocation
> >>> types, combined with the long term goal of reusing the llvm's ELF
> > parser,
> >>> this does not seem like a worthwhile goal. Also, it does not help the
> >>> windows situation, as in the PDB model it's the linker who produces the
> >>> pdb's.
> >>>
> >>> (**) I'll write a separate email about this, but what I'm essentially
> >>> thinking of is producing a 

Re: [lldb-dev] Proposal: Using LLD in tests

[Greg Clayton via lldb-dev]

> On Apr 19, 2018, at 9:39 AM, Pavel Labath  wrote:
> 
> Yes, I considered the lld+Mach-O situation. I understand that it does not
> work very well, but I don't know what exactly that means.
> 
> However, I am not sure that we even need a linker for Mach-O. As I
> understand it, in a darwin world, the linker does not even touch the debug
> info, and lldb is already capable of reading the debug info from .o files.
> So at least for non dsym-case it seems to me we should be good. It may be
> possible we need a linker for the dSYM (i don't know exactly how that
> works), in which case we may just not be able to test dsym this way, but
> even that will be better than what we have now.

The problem is the linker must produce a debug map in the symbol table that 
contains the addresses of all linked items. So using LLD won't work unless that 
is fully supported as LLDB won't be able to like the debug info on the fly, nor 
will dsymutil have a debug map to use in order to make a dSYM file.

> 
> PS: I am not proposing to do anything to the existing dotest tests (they
> need access to a running process anyway, so playing with the linker will
> not help). I am just trying to make new debug-info specific tests
> universally available.

What I was trying to say is this is fine if we always compile/link as an ELF 
file with some triple that uses ELF for these tests. They just probably 
shouldn't be mach-o.

Greg

> 
> 
> On Thu, 19 Apr 2018 at 17:24, Greg Clayton  wrote:
> 
>> The last I knew LLD doesn't work on mach-o very well, so be sure to not
> require LLD for linking any Darwin executables.
> 
>>> On Apr 19, 2018, at 6:42 AM, Pavel Labath via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>>> 
>>> Hello all,
>>> 
>>> currently we have a couple of tests, in-tree or under review, which are
>>> very close to being host-independent. The only part they are missing is
> the
>>> ability to link a intermediate object file:
>>> - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
>>> resolve relocations in the debug info (*)
>>> - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
>>> program database.
>>> 
>>> I think it would be great if everyone were able to run these tests and
>>> verify they don't regress them before they actually push a patch.
>>> 
>>> Apart from that, I have started looking at writing some non-execution
> debug
>>> info (**) tests as a part of adding DWARF v5 accelerator table support
> to
>>> lldb (both to test the new implementation, and to make sure I don't
> regress
>>> existing ones). Ideally I'd like to make sure that everyone is able to
> run
>>> them, regardless of their primary (or only) development platform. For
> this,
>>> I also need a linker capable of running everywhere (*)
>>> 
>>> To achieve these goals, I'd like to propose that we add LLD as a
> (optional,
>>> but strongly recommended) dependency for running tests and start using
> it
>>> in the tests I mention. Doing this would optional in the sense that the
>>> tests would be marked "REQUIRED: lld", and simply skipped if lld is not
>>> available (so the tests would still be green). I say "strongly
> recommended"
>>> because not having lld checked out should not be an excuse for breaking
> the
>>> test, and the patch author should pro-actively revert a patch which
> breaks
>>> such tests and investigate.
>>> 
>>> I hope this proposal is not too controversial. LLD is already required
> on
>>> windows to run dotest tests. Also, all monorepo users likely have it
>>> already, or it is very easy for them to enable it. For non-monorepo
> users
>>> it should be a matter of checking out one extra repository. Please Let
> me
>>> know what you think.
>>> 
>>> pavel
>>> 
>>> (*) our ELF parser has very limited support for applying debug info
>>> relocations -- it only works for x86, and only a couple of relocations
> are
>>> currently implemented. It would be possible to remove the linker
> dependency
>>> by implementing these (essentially, doing the link ourselves -- this is
>>> what llvm does), but given the large number of architectures and
> relocation
>>> types, combined with the long term goal of reusing the llvm's ELF
> parser,
>>> this does not seem like a worthwhile goal. Also, it does not help the
>>> windows situation, as in the PDB model it's the linker who produces the
>>> pdb's.
>>> 
>>> (**) I'll write a separate email about this, but what I'm essentially
>>> thinking of is producing a stand-alone module (either from .yaml, .s,
> .ll,
>>> or .c), hitting it with various FindXXX methods, and dumping the
> results.
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Advice on architectures with multiple address spaces

[Greg Clayton via lldb-dev]

> On Apr 19, 2018, at 6:51 AM, Zdenek Prikryl via lldb-dev 
>  wrote:
> 
> Hi lldb developers,
> 
> I've been researching using lldb + gdbserver stub that is based on Harvard 
> architecture with multiple address spaces (one program, multiple data). The 
> commonly adopted approach is that everything is mapped to a single "virtual" 
> address space. The stub reads/writes from/to the right memory based on the 
> "virtual" addresses. But I'd like to use real addresses with address space id 
> instead. So, I've started looking at what has to be changed.
> 
> I've enhanced read/write commands (e.g. memory read --as  ...) and RSP 
> protocol (new packet) so that the stub can read/write properly. That wasn't 
> that complicated.

It might be nice to add a new RSP protocol packet that asks for the address 
space names/values:

qGetAddressSpaces

which would return something like:

1:text;2:data1,3:data2

or it would return not supported. If we get a valid return value from 
qGetAddressSpaces, then it enables the use of the new packet you added above. 
Else it defaults to using the old memory read functions.


> 
> Now I've hit an issue with expressions (LLVMUserExpression.cpp) and local 
> variables (DWARFExpressions.cpp). There is a lot of memory read/write 
> functions that take just an address argument. Is the only way to go to patch 
> all these calls? Has anybody solved it differently?

My quick take is that any APIs that take just a lldb::addr_t would need to take 
something like:

struct SpaceAddress {
  static constexpr uint32_t kNoSpace = 0;
  lldb::addr_t addr;
  uint32_t space;
};

We would need a default value for "space" (feel free to rename) that indicates 
the default address space as most of our architectures would not need this 
support. If we added a constructor like:

SpaceAddress(lldb::addr_t a) : addr(a), space(kNoSpace) {}

Then all usages of the APIs that used to take just a "lldb::addr_t" would 
implicitly call this constructor and continue to act as needed. Then we would 
need to allow lldb_private::Address objects to resolve to a SpaceAddress:

  SpaceAddress lldb_private::Address::GetSpaceAddress(Target *target) const;

Since each lldb_private::Address has a section and each section knows its 
address space. Then the tricky part is finding all locations in the expression 
parser and converting those to track and use SpaceAddress. We would probably 
need to modify the allocate memory packets in the RSP protocol to be able to 
allocate memory in any address space as well.

I didn't spend much time think about correct names above, so feel free to 
suggest alternate naming. 

Best advice:
- make things "just work" to keep changes to a minimum and allowing 
lldb::addr_t to implicitly convert to a SpaceAddress easily
- when modifying RSP, make sure to check for existence of new feature before 
enabling it
- query for address space names so when we dump SpaceAddress we can show 
something that means something to the user. This means we would need to query 
the address space names from the current lldb_private::Process for display.

Submitting might go easier if we break it down into chunks:
1 - add SpaceAddress and modify all needed APIs to use it
2 - add ProcessGDBRemote changes that enable this support

It will be great to support this as a first class citizen within LLDB. You 
might ask the Hexagon folks if they have done anything in case they already 
support this is some shape or form.

Greg Clayton

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Pavel Labath via lldb-dev]

Yes, I considered the lld+Mach-O situation. I understand that it does not
work very well, but I don't know what exactly that means.

However, I am not sure that we even need a linker for Mach-O. As I
understand it, in a darwin world, the linker does not even touch the debug
info, and lldb is already capable of reading the debug info from .o files.
So at least for non dsym-case it seems to me we should be good. It may be
possible we need a linker for the dSYM (i don't know exactly how that
works), in which case we may just not be able to test dsym this way, but
even that will be better than what we have now.

PS: I am not proposing to do anything to the existing dotest tests (they
need access to a running process anyway, so playing with the linker will
not help). I am just trying to make new debug-info specific tests
universally available.


On Thu, 19 Apr 2018 at 17:24, Greg Clayton  wrote:

> The last I knew LLD doesn't work on mach-o very well, so be sure to not
require LLD for linking any Darwin executables.

> > On Apr 19, 2018, at 6:42 AM, Pavel Labath via lldb-dev <
lldb-dev@lists.llvm.org> wrote:
> >
> > Hello all,
> >
> > currently we have a couple of tests, in-tree or under review, which are
> > very close to being host-independent. The only part they are missing is
the
> > ability to link a intermediate object file:
> > - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
> > resolve relocations in the debug info (*)
> > - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
> > program database.
> >
> > I think it would be great if everyone were able to run these tests and
> > verify they don't regress them before they actually push a patch.
> >
> > Apart from that, I have started looking at writing some non-execution
debug
> > info (**) tests as a part of adding DWARF v5 accelerator table support
to
> > lldb (both to test the new implementation, and to make sure I don't
regress
> > existing ones). Ideally I'd like to make sure that everyone is able to
run
> > them, regardless of their primary (or only) development platform. For
this,
> > I also need a linker capable of running everywhere (*)
> >
> > To achieve these goals, I'd like to propose that we add LLD as a
(optional,
> > but strongly recommended) dependency for running tests and start using
it
> > in the tests I mention. Doing this would optional in the sense that the
> > tests would be marked "REQUIRED: lld", and simply skipped if lld is not
> > available (so the tests would still be green). I say "strongly
recommended"
> > because not having lld checked out should not be an excuse for breaking
the
> > test, and the patch author should pro-actively revert a patch which
breaks
> > such tests and investigate.
> >
> > I hope this proposal is not too controversial. LLD is already required
on
> > windows to run dotest tests. Also, all monorepo users likely have it
> > already, or it is very easy for them to enable it. For non-monorepo
users
> > it should be a matter of checking out one extra repository. Please Let
me
> > know what you think.
> >
> > pavel
> >
> > (*) our ELF parser has very limited support for applying debug info
> > relocations -- it only works for x86, and only a couple of relocations
are
> > currently implemented. It would be possible to remove the linker
dependency
> > by implementing these (essentially, doing the link ourselves -- this is
> > what llvm does), but given the large number of architectures and
relocation
> > types, combined with the long term goal of reusing the llvm's ELF
parser,
> > this does not seem like a worthwhile goal. Also, it does not help the
> > windows situation, as in the PDB model it's the linker who produces the
> > pdb's.
> >
> > (**) I'll write a separate email about this, but what I'm essentially
> > thinking of is producing a stand-alone module (either from .yaml, .s,
.ll,
> > or .c), hitting it with various FindXXX methods, and dumping the
results.
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Proposal: Using LLD in tests

[Greg Clayton via lldb-dev]

The last I knew LLD doesn't work on mach-o very well, so be sure to not require 
LLD for linking any Darwin executables.

> On Apr 19, 2018, at 6:42 AM, Pavel Labath via lldb-dev 
>  wrote:
> 
> Hello all,
> 
> currently we have a couple of tests, in-tree or under review, which are
> very close to being host-independent. The only part they are missing is the
> ability to link a intermediate object file:
> - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
> resolve relocations in the debug info (*)
> - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
> program database.
> 
> I think it would be great if everyone were able to run these tests and
> verify they don't regress them before they actually push a patch.
> 
> Apart from that, I have started looking at writing some non-execution debug
> info (**) tests as a part of adding DWARF v5 accelerator table support to
> lldb (both to test the new implementation, and to make sure I don't regress
> existing ones). Ideally I'd like to make sure that everyone is able to run
> them, regardless of their primary (or only) development platform. For this,
> I also need a linker capable of running everywhere (*)
> 
> To achieve these goals, I'd like to propose that we add LLD as a (optional,
> but strongly recommended) dependency for running tests and start using it
> in the tests I mention. Doing this would optional in the sense that the
> tests would be marked "REQUIRED: lld", and simply skipped if lld is not
> available (so the tests would still be green). I say "strongly recommended"
> because not having lld checked out should not be an excuse for breaking the
> test, and the patch author should pro-actively revert a patch which breaks
> such tests and investigate.
> 
> I hope this proposal is not too controversial. LLD is already required on
> windows to run dotest tests. Also, all monorepo users likely have it
> already, or it is very easy for them to enable it. For non-monorepo users
> it should be a matter of checking out one extra repository. Please Let me
> know what you think.
> 
> pavel
> 
> (*) our ELF parser has very limited support for applying debug info
> relocations -- it only works for x86, and only a couple of relocations are
> currently implemented. It would be possible to remove the linker dependency
> by implementing these (essentially, doing the link ourselves -- this is
> what llvm does), but given the large number of architectures and relocation
> types, combined with the long term goal of reusing the llvm's ELF parser,
> this does not seem like a worthwhile goal. Also, it does not help the
> windows situation, as in the PDB model it's the linker who produces the
> pdb's.
> 
> (**) I'll write a separate email about this, but what I'm essentially
> thinking of is producing a stand-alone module (either from .yaml, .s, .ll,
> or .c), hitting it with various FindXXX methods, and dumping the results.
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] Advice on architectures with multiple address spaces

[Zdenek Prikryl via lldb-dev]

Hi lldb developers,

I've been researching using lldb + gdbserver stub that is based on 
Harvard architecture with multiple address spaces (one program, multiple 
data). The commonly adopted approach is that everything is mapped to a 
single "virtual" address space. The stub reads/writes from/to the right 
memory based on the "virtual" addresses. But I'd like to use real 
addresses with address space id instead. So, I've started looking at 
what has to be changed.


I've enhanced read/write commands (e.g. memory read --as  ...) and 
RSP protocol (new packet) so that the stub can read/write properly. That 
wasn't that complicated.


Now I've hit an issue with expressions (LLVMUserExpression.cpp) and 
local variables (DWARFExpressions.cpp). There is a lot of memory 
read/write functions that take just an address argument. Is the only way 
to go to patch all these calls? Has anybody solved it differently?


Thanks for any advice.

--
Zdenek

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] Proposal: Using LLD in tests

[Pavel Labath via lldb-dev]

Hello all,

currently we have a couple of tests, in-tree or under review, which are
very close to being host-independent. The only part they are missing is the
ability to link a intermediate object file:
- the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
resolve relocations in the debug info (*)
- the PDB tests under lit/SymbolFile/PDB need a linker to produce the
program database.

I think it would be great if everyone were able to run these tests and
verify they don't regress them before they actually push a patch.

Apart from that, I have started looking at writing some non-execution debug
info (**) tests as a part of adding DWARF v5 accelerator table support to
lldb (both to test the new implementation, and to make sure I don't regress
existing ones). Ideally I'd like to make sure that everyone is able to run
them, regardless of their primary (or only) development platform. For this,
I also need a linker capable of running everywhere (*)

To achieve these goals, I'd like to propose that we add LLD as a (optional,
but strongly recommended) dependency for running tests and start using it
in the tests I mention. Doing this would optional in the sense that the
tests would be marked "REQUIRED: lld", and simply skipped if lld is not
available (so the tests would still be green). I say "strongly recommended"
because not having lld checked out should not be an excuse for breaking the
test, and the patch author should pro-actively revert a patch which breaks
such tests and investigate.

I hope this proposal is not too controversial. LLD is already required on
windows to run dotest tests. Also, all monorepo users likely have it
already, or it is very easy for them to enable it. For non-monorepo users
it should be a matter of checking out one extra repository. Please Let me
know what you think.

pavel

(*) our ELF parser has very limited support for applying debug info
relocations -- it only works for x86, and only a couple of relocations are
currently implemented. It would be possible to remove the linker dependency
by implementing these (essentially, doing the link ourselves -- this is
what llvm does), but given the large number of architectures and relocation
types, combined with the long term goal of reusing the llvm's ELF parser,
this does not seem like a worthwhile goal. Also, it does not help the
windows situation, as in the PDB model it's the linker who produces the
pdb's.

(**) I'll write a separate email about this, but what I'm essentially
thinking of is producing a stand-alone module (either from .yaml, .s, .ll,
or .c), hitting it with various FindXXX methods, and dumping the results.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev