Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Adrian Prantl via lldb-dev


> On Feb 25, 2019, at 2:21 PM, Davide Italiano  wrote:
> 
> On Mon, Feb 25, 2019 at 1:57 PM Adrian Prantl  wrote:
>> 
>> 
>> 
>>> On Feb 25, 2019, at 1:40 PM, Davide Italiano  wrote:
>>> 
>>> On Mon, Feb 25, 2019 at 1:35 PM Adrian Prantl  wrote:
 
 
 
> On Feb 25, 2019, at 1:15 PM, Davide Italiano  
> wrote:
> 
> On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
>> 
>> On 21/02/2019 19:48, Ted Woodward wrote:
>>> 
>>> 
 -Original Message-
 From: lldb-dev  On Behalf Of Pavel 
 Labath
 via lldb-dev
 Sent: Thursday, February 21, 2019 8:35 AM
 To: Davide Italiano 
 Cc: LLDB 
 Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
 
 On 21/02/2019 00:03, Davide Italiano wrote:
> I found out that there are tests that effectively require
> interactivity. Some of the lldb-mi ones are an example.
> A common use-case is that of sending SIGTERM in a loop to make sure
> `lldb-mi` doesn't crash and handle the signal correctly.
> 
> This functionality is really hard to replicate in lit_as is_.
> Any ideas on how we could handle this case?
 
 How hard is it to import a new version of pexpect which supports 
 python3 and
 stuff?
 
 I'm not sure how the situation is on darwin, but I'd expect (:P) that 
 most linux
 systems either already have it installed, or have an easy way to do 
 so. So we
 may not even be able to get away with just using the system one and 
 skipping
 tests when it's not present.
 
 BTW, for lldb-mi I would actually argue that it should *not* use 
 pexpect :D.
 Interactivity is one thing, and I'm very much in favour of keeping 
 that ability,
 but pexpect is not a prerequisite for that. For me, the main advantage 
 of
 pexpect is that it emulates a real terminal. However, lldb-mi does not 
 need
 that stuff. It doesn't have any command line editing capabilities or 
 similar. It's
 expecting to communicate with an IDE over a pipe, and that's it.
 
 Given that, it should be fairly easy to rewrite the lldb-mi tests to 
 work on top
 of the standard python "subprocess" library. While we're doing that, 
 we might
 actually fix some of the issues that have been bugging everyone in the 
 lldb-mi
 tests. At least for me, the most annoying thing was that when lldb-mi 
 fails to
 produce the expected output, the test does not fail immediately, but 
 instead
 the implementation of self.expect("^whatever") waits until the timeout
 expires, optimistically hoping that it will find some output that 
 match the
 pattern.
 
> 
> Pavel, I think yours is a really nice idea.
> I'm no python expert, but I found out making the conversion is
> relatively simple.
> I propose a proof-of-concept API and implementation here:
> 
> https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68
> 
> Comments appreciated! Once we agree on how this should look like, I do
> recommend to have a new lldbMITest base class and incrementally start
> moving the tests to it.
> Once we're done, we can delete the old class.
> 
> Does this sound reasonable?
 
 What you are saying is that we would write the tests as Python tests in a 
 way similar to how lldbtest.expect() look in the dotest.py tests, banking 
 on synchronous mode taking care of all the synchronization? Are you 
 thinking of doing this for all the remaining tests or only the ones where 
 a command input depends on the result of a previous command?
 
>>> 
>>> I'm thinking to do this for all the remaining tests. Do you have any
>>> concerns about this? (I'm aware your GSoC student introduced the `lit
>>> lldb-mi` tests for a reason, I just don't know exactly what the reason
>>> was).
>> 
>> I think the reason was that for tests that don't need synchronization and 
>> have a static command input, writing a lit/FileCheck test is straightforward 
>> and easy. At the time I thought we could just rewrite *all* lldb-mi tests as 
>> FileCheck tests. If we do need a python mechanism anyway, however, I don't 
>> actually think that having two ways of writing tests is better; I'd rather 
>> have all tests in one place. Thankfully there shouldn't be that many tests, 
>> so we should be able to just convert all of them over to whatever format we 
>> eventually settle on.
>> -- Adrian
>> 
> 
> Sounds good to me. Do you have a preference? I wanted to move away
> from pexpect, so my preference was that of using `lit`.
> I then realized that there are some functionalities that are not
> really that easy to 

Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Davide Italiano via lldb-dev
On Mon, Feb 25, 2019 at 1:57 PM Adrian Prantl  wrote:
>
>
>
> > On Feb 25, 2019, at 1:40 PM, Davide Italiano  wrote:
> >
> > On Mon, Feb 25, 2019 at 1:35 PM Adrian Prantl  wrote:
> >>
> >>
> >>
> >>> On Feb 25, 2019, at 1:15 PM, Davide Italiano  
> >>> wrote:
> >>>
> >>> On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
> 
>  On 21/02/2019 19:48, Ted Woodward wrote:
> >
> >
> >> -Original Message-
> >> From: lldb-dev  On Behalf Of Pavel 
> >> Labath
> >> via lldb-dev
> >> Sent: Thursday, February 21, 2019 8:35 AM
> >> To: Davide Italiano 
> >> Cc: LLDB 
> >> Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
> >>
> >> On 21/02/2019 00:03, Davide Italiano wrote:
> >>> I found out that there are tests that effectively require
> >>> interactivity. Some of the lldb-mi ones are an example.
> >>> A common use-case is that of sending SIGTERM in a loop to make sure
> >>> `lldb-mi` doesn't crash and handle the signal correctly.
> >>>
> >>> This functionality is really hard to replicate in lit_as is_.
> >>> Any ideas on how we could handle this case?
> >>
> >> How hard is it to import a new version of pexpect which supports 
> >> python3 and
> >> stuff?
> >>
> >> I'm not sure how the situation is on darwin, but I'd expect (:P) that 
> >> most linux
> >> systems either already have it installed, or have an easy way to do 
> >> so. So we
> >> may not even be able to get away with just using the system one and 
> >> skipping
> >> tests when it's not present.
> >>
> >> BTW, for lldb-mi I would actually argue that it should *not* use 
> >> pexpect :D.
> >> Interactivity is one thing, and I'm very much in favour of keeping 
> >> that ability,
> >> but pexpect is not a prerequisite for that. For me, the main advantage 
> >> of
> >> pexpect is that it emulates a real terminal. However, lldb-mi does not 
> >> need
> >> that stuff. It doesn't have any command line editing capabilities or 
> >> similar. It's
> >> expecting to communicate with an IDE over a pipe, and that's it.
> >>
> >> Given that, it should be fairly easy to rewrite the lldb-mi tests to 
> >> work on top
> >> of the standard python "subprocess" library. While we're doing that, 
> >> we might
> >> actually fix some of the issues that have been bugging everyone in the 
> >> lldb-mi
> >> tests. At least for me, the most annoying thing was that when lldb-mi 
> >> fails to
> >> produce the expected output, the test does not fail immediately, but 
> >> instead
> >> the implementation of self.expect("^whatever") waits until the timeout
> >> expires, optimistically hoping that it will find some output that 
> >> match the
> >> pattern.
> >>
> >>>
> >>> Pavel, I think yours is a really nice idea.
> >>> I'm no python expert, but I found out making the conversion is
> >>> relatively simple.
> >>> I propose a proof-of-concept API and implementation here:
> >>>
> >>> https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68
> >>>
> >>> Comments appreciated! Once we agree on how this should look like, I do
> >>> recommend to have a new lldbMITest base class and incrementally start
> >>> moving the tests to it.
> >>> Once we're done, we can delete the old class.
> >>>
> >>> Does this sound reasonable?
> >>
> >> What you are saying is that we would write the tests as Python tests in a 
> >> way similar to how lldbtest.expect() look in the dotest.py tests, banking 
> >> on synchronous mode taking care of all the synchronization? Are you 
> >> thinking of doing this for all the remaining tests or only the ones where 
> >> a command input depends on the result of a previous command?
> >>
> >
> > I'm thinking to do this for all the remaining tests. Do you have any
> > concerns about this? (I'm aware your GSoC student introduced the `lit
> > lldb-mi` tests for a reason, I just don't know exactly what the reason
> > was).
>
> I think the reason was that for tests that don't need synchronization and 
> have a static command input, writing a lit/FileCheck test is straightforward 
> and easy. At the time I thought we could just rewrite *all* lldb-mi tests as 
> FileCheck tests. If we do need a python mechanism anyway, however, I don't 
> actually think that having two ways of writing tests is better; I'd rather 
> have all tests in one place. Thankfully there shouldn't be that many tests, 
> so we should be able to just convert all of them over to whatever format we 
> eventually settle on.
> -- Adrian
>

Sounds good to me. Do you have a preference? I wanted to move away
from pexpect, so my preference was that of using `lit`.
I then realized that there are some functionalities that are not
really that easy to replicate in lit (e.g. sending signals, see my
previous mail), hence my vote for Python.subprocess (as Pavel

Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Davide Italiano via lldb-dev
On Mon, Feb 25, 2019 at 1:35 PM Adrian Prantl  wrote:
>
>
>
> > On Feb 25, 2019, at 1:15 PM, Davide Italiano  wrote:
> >
> > On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
> >>
> >> On 21/02/2019 19:48, Ted Woodward wrote:
> >>>
> >>>
>  -Original Message-
>  From: lldb-dev  On Behalf Of Pavel 
>  Labath
>  via lldb-dev
>  Sent: Thursday, February 21, 2019 8:35 AM
>  To: Davide Italiano 
>  Cc: LLDB 
>  Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
> 
>  On 21/02/2019 00:03, Davide Italiano wrote:
> > I found out that there are tests that effectively require
> > interactivity. Some of the lldb-mi ones are an example.
> > A common use-case is that of sending SIGTERM in a loop to make sure
> > `lldb-mi` doesn't crash and handle the signal correctly.
> >
> > This functionality is really hard to replicate in lit_as is_.
> > Any ideas on how we could handle this case?
> 
>  How hard is it to import a new version of pexpect which supports python3 
>  and
>  stuff?
> 
>  I'm not sure how the situation is on darwin, but I'd expect (:P) that 
>  most linux
>  systems either already have it installed, or have an easy way to do so. 
>  So we
>  may not even be able to get away with just using the system one and 
>  skipping
>  tests when it's not present.
> 
>  BTW, for lldb-mi I would actually argue that it should *not* use pexpect 
>  :D.
>  Interactivity is one thing, and I'm very much in favour of keeping that 
>  ability,
>  but pexpect is not a prerequisite for that. For me, the main advantage of
>  pexpect is that it emulates a real terminal. However, lldb-mi does not 
>  need
>  that stuff. It doesn't have any command line editing capabilities or 
>  similar. It's
>  expecting to communicate with an IDE over a pipe, and that's it.
> 
>  Given that, it should be fairly easy to rewrite the lldb-mi tests to 
>  work on top
>  of the standard python "subprocess" library. While we're doing that, we 
>  might
>  actually fix some of the issues that have been bugging everyone in the 
>  lldb-mi
>  tests. At least for me, the most annoying thing was that when lldb-mi 
>  fails to
>  produce the expected output, the test does not fail immediately, but 
>  instead
>  the implementation of self.expect("^whatever") waits until the timeout
>  expires, optimistically hoping that it will find some output that match 
>  the
>  pattern.
> 
> >
> > Pavel, I think yours is a really nice idea.
> > I'm no python expert, but I found out making the conversion is
> > relatively simple.
> > I propose a proof-of-concept API and implementation here:
> >
> > https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68
> >
> > Comments appreciated! Once we agree on how this should look like, I do
> > recommend to have a new lldbMITest base class and incrementally start
> > moving the tests to it.
> > Once we're done, we can delete the old class.
> >
> > Does this sound reasonable?
>
> What you are saying is that we would write the tests as Python tests in a way 
> similar to how lldbtest.expect() look in the dotest.py tests, banking on 
> synchronous mode taking care of all the synchronization? Are you thinking of 
> doing this for all the remaining tests or only the ones where a command input 
> depends on the result of a previous command?
>

I'm thinking to do this for all the remaining tests. Do you have any
concerns about this? (I'm aware your GSoC student introduced the `lit
lldb-mi` tests for a reason, I just don't know exactly what the reason
was).
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Adrian Prantl via lldb-dev


> On Feb 25, 2019, at 1:40 PM, Davide Italiano  wrote:
> 
> On Mon, Feb 25, 2019 at 1:35 PM Adrian Prantl  wrote:
>> 
>> 
>> 
>>> On Feb 25, 2019, at 1:15 PM, Davide Italiano  wrote:
>>> 
>>> On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
 
 On 21/02/2019 19:48, Ted Woodward wrote:
> 
> 
>> -Original Message-
>> From: lldb-dev  On Behalf Of Pavel 
>> Labath
>> via lldb-dev
>> Sent: Thursday, February 21, 2019 8:35 AM
>> To: Davide Italiano 
>> Cc: LLDB 
>> Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
>> 
>> On 21/02/2019 00:03, Davide Italiano wrote:
>>> I found out that there are tests that effectively require
>>> interactivity. Some of the lldb-mi ones are an example.
>>> A common use-case is that of sending SIGTERM in a loop to make sure
>>> `lldb-mi` doesn't crash and handle the signal correctly.
>>> 
>>> This functionality is really hard to replicate in lit_as is_.
>>> Any ideas on how we could handle this case?
>> 
>> How hard is it to import a new version of pexpect which supports python3 
>> and
>> stuff?
>> 
>> I'm not sure how the situation is on darwin, but I'd expect (:P) that 
>> most linux
>> systems either already have it installed, or have an easy way to do so. 
>> So we
>> may not even be able to get away with just using the system one and 
>> skipping
>> tests when it's not present.
>> 
>> BTW, for lldb-mi I would actually argue that it should *not* use pexpect 
>> :D.
>> Interactivity is one thing, and I'm very much in favour of keeping that 
>> ability,
>> but pexpect is not a prerequisite for that. For me, the main advantage of
>> pexpect is that it emulates a real terminal. However, lldb-mi does not 
>> need
>> that stuff. It doesn't have any command line editing capabilities or 
>> similar. It's
>> expecting to communicate with an IDE over a pipe, and that's it.
>> 
>> Given that, it should be fairly easy to rewrite the lldb-mi tests to 
>> work on top
>> of the standard python "subprocess" library. While we're doing that, we 
>> might
>> actually fix some of the issues that have been bugging everyone in the 
>> lldb-mi
>> tests. At least for me, the most annoying thing was that when lldb-mi 
>> fails to
>> produce the expected output, the test does not fail immediately, but 
>> instead
>> the implementation of self.expect("^whatever") waits until the timeout
>> expires, optimistically hoping that it will find some output that match 
>> the
>> pattern.
>> 
>>> 
>>> Pavel, I think yours is a really nice idea.
>>> I'm no python expert, but I found out making the conversion is
>>> relatively simple.
>>> I propose a proof-of-concept API and implementation here:
>>> 
>>> https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68
>>> 
>>> Comments appreciated! Once we agree on how this should look like, I do
>>> recommend to have a new lldbMITest base class and incrementally start
>>> moving the tests to it.
>>> Once we're done, we can delete the old class.
>>> 
>>> Does this sound reasonable?
>> 
>> What you are saying is that we would write the tests as Python tests in a 
>> way similar to how lldbtest.expect() look in the dotest.py tests, banking on 
>> synchronous mode taking care of all the synchronization? Are you thinking of 
>> doing this for all the remaining tests or only the ones where a command 
>> input depends on the result of a previous command?
>> 
> 
> I'm thinking to do this for all the remaining tests. Do you have any
> concerns about this? (I'm aware your GSoC student introduced the `lit
> lldb-mi` tests for a reason, I just don't know exactly what the reason
> was).

I think the reason was that for tests that don't need synchronization and have 
a static command input, writing a lit/FileCheck test is straightforward and 
easy. At the time I thought we could just rewrite *all* lldb-mi tests as 
FileCheck tests. If we do need a python mechanism anyway, however, I don't 
actually think that having two ways of writing tests is better; I'd rather have 
all tests in one place. Thankfully there shouldn't be that many tests, so we 
should be able to just convert all of them over to whatever format we 
eventually settle on.
-- adrian


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Adrian Prantl via lldb-dev


> On Feb 25, 2019, at 1:15 PM, Davide Italiano  wrote:
> 
> On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
>> 
>> On 21/02/2019 19:48, Ted Woodward wrote:
>>> 
>>> 
 -Original Message-
 From: lldb-dev  On Behalf Of Pavel Labath
 via lldb-dev
 Sent: Thursday, February 21, 2019 8:35 AM
 To: Davide Italiano 
 Cc: LLDB 
 Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
 
 On 21/02/2019 00:03, Davide Italiano wrote:
> I found out that there are tests that effectively require
> interactivity. Some of the lldb-mi ones are an example.
> A common use-case is that of sending SIGTERM in a loop to make sure
> `lldb-mi` doesn't crash and handle the signal correctly.
> 
> This functionality is really hard to replicate in lit_as is_.
> Any ideas on how we could handle this case?
 
 How hard is it to import a new version of pexpect which supports python3 
 and
 stuff?
 
 I'm not sure how the situation is on darwin, but I'd expect (:P) that most 
 linux
 systems either already have it installed, or have an easy way to do so. So 
 we
 may not even be able to get away with just using the system one and 
 skipping
 tests when it's not present.
 
 BTW, for lldb-mi I would actually argue that it should *not* use pexpect 
 :D.
 Interactivity is one thing, and I'm very much in favour of keeping that 
 ability,
 but pexpect is not a prerequisite for that. For me, the main advantage of
 pexpect is that it emulates a real terminal. However, lldb-mi does not need
 that stuff. It doesn't have any command line editing capabilities or 
 similar. It's
 expecting to communicate with an IDE over a pipe, and that's it.
 
 Given that, it should be fairly easy to rewrite the lldb-mi tests to work 
 on top
 of the standard python "subprocess" library. While we're doing that, we 
 might
 actually fix some of the issues that have been bugging everyone in the 
 lldb-mi
 tests. At least for me, the most annoying thing was that when lldb-mi 
 fails to
 produce the expected output, the test does not fail immediately, but 
 instead
 the implementation of self.expect("^whatever") waits until the timeout
 expires, optimistically hoping that it will find some output that match the
 pattern.
 
> 
> Pavel, I think yours is a really nice idea.
> I'm no python expert, but I found out making the conversion is
> relatively simple.
> I propose a proof-of-concept API and implementation here:
> 
> https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68
> 
> Comments appreciated! Once we agree on how this should look like, I do
> recommend to have a new lldbMITest base class and incrementally start
> moving the tests to it.
> Once we're done, we can delete the old class.
> 
> Does this sound reasonable?

What you are saying is that we would write the tests as Python tests in a way 
similar to how lldbtest.expect() look in the dotest.py tests, banking on 
synchronous mode taking care of all the synchronization? Are you thinking of 
doing this for all the remaining tests or only the ones where a command input 
depends on the result of a previous command?

-- adrian

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


Re: [lldb-dev] [RFC]The future of pexpect

2019-02-25 Thread Davide Italiano via lldb-dev
On Fri, Feb 22, 2019 at 6:32 AM Pavel Labath  wrote:
>
> On 21/02/2019 19:48, Ted Woodward wrote:
> >
> >
> >> -Original Message-
> >> From: lldb-dev  On Behalf Of Pavel Labath
> >> via lldb-dev
> >> Sent: Thursday, February 21, 2019 8:35 AM
> >> To: Davide Italiano 
> >> Cc: LLDB 
> >> Subject: [EXT] Re: [lldb-dev] [RFC]The future of pexpect
> >>
> >> On 21/02/2019 00:03, Davide Italiano wrote:
> >>> I found out that there are tests that effectively require
> >>> interactivity. Some of the lldb-mi ones are an example.
> >>> A common use-case is that of sending SIGTERM in a loop to make sure
> >>> `lldb-mi` doesn't crash and handle the signal correctly.
> >>>
> >>> This functionality is really hard to replicate in lit_as is_.
> >>> Any ideas on how we could handle this case?
> >>
> >> How hard is it to import a new version of pexpect which supports python3 
> >> and
> >> stuff?
> >>
> >> I'm not sure how the situation is on darwin, but I'd expect (:P) that most 
> >> linux
> >> systems either already have it installed, or have an easy way to do so. So 
> >> we
> >> may not even be able to get away with just using the system one and 
> >> skipping
> >> tests when it's not present.
> >>
> >> BTW, for lldb-mi I would actually argue that it should *not* use pexpect 
> >> :D.
> >> Interactivity is one thing, and I'm very much in favour of keeping that 
> >> ability,
> >> but pexpect is not a prerequisite for that. For me, the main advantage of
> >> pexpect is that it emulates a real terminal. However, lldb-mi does not need
> >> that stuff. It doesn't have any command line editing capabilities or 
> >> similar. It's
> >> expecting to communicate with an IDE over a pipe, and that's it.
> >>
> >> Given that, it should be fairly easy to rewrite the lldb-mi tests to work 
> >> on top
> >> of the standard python "subprocess" library. While we're doing that, we 
> >> might
> >> actually fix some of the issues that have been bugging everyone in the 
> >> lldb-mi
> >> tests. At least for me, the most annoying thing was that when lldb-mi 
> >> fails to
> >> produce the expected output, the test does not fail immediately, but 
> >> instead
> >> the implementation of self.expect("^whatever") waits until the timeout
> >> expires, optimistically hoping that it will find some output that match the
> >> pattern.
> >>

Pavel, I think yours is a really nice idea.
I'm no python expert, but I found out making the conversion is
relatively simple.
I propose a proof-of-concept API and implementation here:

https://gist.github.com/dcci/94a4936a227d9c7627b91ae9575b7b68

Comments appreciated! Once we agree on how this should look like, I do
recommend to have a new lldbMITest base class and incrementally start
moving the tests to it.
Once we're done, we can delete the old class.

Does this sound reasonable?

--
Davide
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


[lldb-dev] RFC: Moving debug info parsing out of process

2019-02-25 Thread Zachary Turner via lldb-dev
Hi all,

We've got some internal efforts in progress, and one of those would benefit
from debug info parsing being out of process (independently of whether or
not the rest of LLDB is out of process).

There's a couple of advantages to this, which I'll enumerate here:

   - It improves one source of instability in LLDB which has been known to
   be problematic -- specifically, that debug info can be bad and handling
   this can often be difficult and bring down the entire debug session.  While
   other efforts have been made to address stability by moving things out of
   process, they have not been upstreamed, and even if they had I think we
   would still want this anyway, for reasons that follow.
   - It becomes theoretically possible to move debug info parsing not just
   to another process, but to another machine entirely.  In a broader sense,
   this decouples the physical debug info location (and for that matter,
   representation) from the debugger host.
   - It becomes testable as an independent component, because you can just
   send requests to it and dump the results and see if they make sense.
   Currently there is almost zero test coverage of this aspect of LLDB apart
   from what you can get after going through many levels of indirection via
   spinning up a full debug session and doing things that indirectly result in
   symbol queries.

The big win here, at least from my point of view, is the second one.
Traditional symbol servers operate by copying entire symbol files (DSYM,
DWP, PDB) from some machine to the debugger host.  These can be very large
-- we've seen 12+ GB in some cases -- which ranges from "slow bandwidth
hog" to "complete non-starter" depending on the debugger host and network.
In this kind of scenario, one could theoretically run the debug info
process on the same NAS, cloud, or whatever as the symbol server.  Then,
rather than copying over an entire symbol file, it responds only to the
query you issued -- if you asked for a type, it just returns a packet
describing the type you requested.

The API itself would be stateless (so that you could make queries for
multiple targets in any order) as well as asynchronous (so that responses
might arrive out of order).  Blocking could be implemented in LLDB, but
having the server be asynchronous means multiple clients could connect to
the same server instance.  This raises interesting possibilities.  For
example, one can imagine thousands of developers connecting to an internal
symbol server on the network and being able to debug remote processes or
core dumps over slow network connections or on machines with very little
storage (e.g. chromebooks).


On the LLDB side, all of this is hidden behind the SymbolFile interface, so
most of LLDB doesn't have to change at all.   While this is in development,
we could have SymbolFileRemote and keep the existing local codepath the
default, until such time that it's robust and complete enough that we can
switch the default.

Thoughts?
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev


Re: [lldb-dev] SB API is not working properly with OSPython plugin

2019-02-25 Thread Tatyana Krasnukha via lldb-dev
This is ThreadList::GetSelectedThread, it calls 
OperatingSystemPython::UpdateThreadList indirectly.

However, I cannot reproduce the error Alexander gets, due to a deadlock.

-Original Message-
From: jing...@apple.com  
Sent: Friday, February 22, 2019 2:14 AM
To: Tatyana Krasnukha 
Cc: Alexander Polyakov ; LLDB 
Subject: Re: [lldb-dev] SB API is not working properly with OSPython plugin

What's triggering one of the OS Plugin methods to get run on this separate 
thread?  I would expect SetPrivateState would just cause the private stop event 
to get broadcast to the private state thread, and then that would wake up and 
then it would be the one to call the OS Plugin to do it's job.  That's how the 
GDBRemote plugin works, for instance.

Jim


> On Feb 21, 2019, at 2:54 PM, Tatyana Krasnukha 
>  wrote:
> 
> That clarified things, thanks!
> 
> I think, this is the reason:
> 
> ProcessRunLock ::GetRunLock() {  if 
> (m_private_state_thread.EqualsThread(Host::GetCurrentThread()))
>return m_private_run_lock;
>  else
>return m_public_run_lock;
> }
> 
> In our case the current thread is not m_private_state_thread. I create a 
> separate thread for waiting a processor to halt and then SetPrivateState. It 
> seems, that was an inappropriate approach.
> 
> -Original Message-
> From: jing...@apple.com 
> Sent: Thursday, February 21, 2019 11:58 PM
> To: Tatyana Krasnukha 
> Cc: Alexander Polyakov ; LLDB 
> 
> Subject: Re: [lldb-dev] SB API is not working properly with OSPython 
> plugin
> 
> 
> 
>> On Feb 21, 2019, at 11:22 AM, Tatyana Krasnukha via lldb-dev 
>>  wrote:
>> 
>>> lldb Process::SetPrivateState (stopped) stop_id = 2
>>> error: error: process must be stopped.
>> 
>> These two lines are printed from different threads, you cannot be sure the 
>> real order of execution is the same.
>> 
>> The plugin should subscribe on public state change events and wait until one 
>> comes (correct me if I’m wrong about that).
> 
> That's not right.  When the process stops, but before lldb broadcasts a 
> public stop event, it will query the OS Plugin directly to build up the 
> thread list.  It needs to do that before it declares a public stop because it 
> uses the thread list to reason about whether to declare a public stop or not. 
>  So the OS Plugin (and BTW the Thread Step Plan plugin is in the same boat) 
> have to run after a private stop but before public one.  There isn't a 
> principled way to do that.  The best we have at present is a hack that says 
> "if you are running on the private state thread, then you get to do things 
> between private & public stop as if there were a public stop".  Cleaning that 
> up is item 5 on the lldb Projects page if anyone is interested...
> 
> Jim
> 
>> 
>> From: lldb-dev  On Behalf Of 
>> Alexander Polyakov via lldb-dev
>> Sent: Thursday, February 21, 2019 9:54 PM
>> To: Jim Ingham 
>> Cc: LLDB 
>> Subject: Re: [lldb-dev] SB API is not working properly with OSPython 
>> plugin
>> 
>> It seems that the process plugin uses the Process::SetPrivateState at the 
>> right time. If you look at the log, you will see that the process is already 
>> in the 'private stopped' state when the OS plugin is invoked.
>> 
>> (lldb) c
>> lldb Process::Resume -- locking run lock
>> lldb Process::PrivateResume() m_stop_id = 1, public state: 
>> stopped private state: stopped
>> lldb Process::SetPrivateState (running)
>> intern-state Process::ShouldBroadcastEvent (0x1abea90) => new state: 
>> running, last broadcast state: running - YES
>> intern-state Process::HandlePrivateEvent (pid = 1) broadcasting new 
>> state running (old state stopped) to public
>> intern-state Process::PushProcessIOHandler pushing IO handler
>> intern-state Process::HandlePrivateEvent updated m_iohandler_sync to 1
>> lldb Process thinks the process has resumed.
>> intern-state timeout = , event_sp)...
>> lldb waited from m_iohandler_sync to change from 0. New value is 
>> 1.
>> dbg.evt-handler  Process::SetPublicState (state = running, restarted 
>> =
>> 0) Process 1 resuming
>> lldb Process::SetPrivateState (stopped)
>> lldb Process::SetPrivateState (stopped) stop_id = 2
>> error: error: process must be stopped.
>> intern-state Process::ShouldBroadcastEvent (0x7f3e9c007450) => new 
>> state: stopped, last broadcast state: stopped - YES
>> intern-state Process::HandlePrivateEvent (pid = 1) broadcasting new 
>> state stopped (old state running) to public
>> intern-state timeout = , event_sp)...
>> dbg.evt-handler  Process::SetPublicState (state = stopped, restarted 
>> =
>> 0) dbg.evt-handler  Process::SetPublicState (stopped) -- unlocking 
>> run lock
>> 
>> 
>> On Fri, Feb 15, 2019 at 1:38 AM Jim Ingham  wrote:
>> Your plugin should have set the private state to stopped when it figures out 
>> however it does that the process has stopped.  The API is 
>>