labath added a comment.

In, @aprantl wrote:

> >   The advantage of the second one is that we will have the ability to 
> > inject commands which depend on the results of previous commands (something 
> > that I think we will need, sooner or later).
> That is worth considering. To write good tests that depend on previous 
> results, we probably want to have SBAPI-like Python scripting for lldb-mi 
> available. To make that work we would need to introduce an 
> lldbmi.HandleCommand("-mycmd...") interface that runs one command, blocks on 
> it, and returns the output in a string. And then we would need to export that 
> to Python. Since the interface is string-in-string-out, we could probably do 
> it via C without even involving SWIG. As an escape hatch to access features 
> outside of the mi command set, we could implement the -interpreter-exec mi 
> command that passes its input to SBAPI's HandleCommand() if need be.
>  Alternatively we could use gtest to write unittest-style lldb-mi tests 
> directly in C++, if going via Python is too messy. Again, we would do that in 
> a blocking or synchronized I/O mode.

I agree with your analysis. I don't want to give an opinion on the direction, 
a) it's not clear to me which way to go, each possibility has different 
tradeoffs involved
b) I don't see myself doing anything related to lldb-mi any time soon

What I can offer as an additional data point is the lldb-server test suite. 
lldb-server very similar to lldb-mi in a way, as it has an almost-text-based 
protocol as its main form of interaction with the world. We also have two ways 
of testing it: a python one (`test/testcases/tools/lldb-server` and a gtest one 
(`unittests/tools/lldb-server`). the python one is older, but a bit messy 
(IMO). The gtest one was started because we had a person interested in 
improving lldb-server testing situation. I supported starting a fresh sub-test 
suite, because I saw it as a way to solve a couple some fundamental issues in 
the old one:

- code duplication: the python test suite needs a complete reimplementation of 
the gdb-remote protocol. the c++ test can just hook it to the existing 
functionality at an appropriate layer.
- remote testing mess: a very convoluted setup is required to run tests on a 
remote (android) device. It's weird that you have to build a host lldb and run 
the tests from there, even though the lldb-server and the executables it debugs 
run on a different host/arch. The python part of the test needs to run on the 
host (there's no easily available python for android), so there's a lot of 
flakyness and complexity coming from the port forwarding/network communication).

The c++ based approach could solve these, as the gtest executable could run on 
the same device as the lldb-server under test, and it would be possible to run 
the tests via check-lldb(-server?) from the cross-build tree instead of copying 
things around between two build trees.

Sadly, the person who was supposed to do this left the team. I have tried to 
carry this idea on slowly, but haven't yet gotten around to implementing the 
most important part (running tests remotely), so I couldn't start migrating the 
tests. In retrospect, I am less excited about this idea than I was a year ago, 
though I still think it would offer a better testing story for lldb-server.

Overall, I am not sure how much of this relates to lldb-mi (for example, you 
probably aren't interested in running it in such a constrained environment), 
but I thought it might be interesting for you to look at these before deciding 
on a direction. I also don't want to hold up Alexander's internship over this, 
as I'm not sure this is what he signed up for. I wanted to bring up the testing 
story, because I think there are some issues there, but I'll leave it up to you 
(for some definition of "you") to decide on what to do about that.


lldb-commits mailing list

Reply via email to