I have been thinking about something similar. Ildb-mi specifically I have
serious concerns about because the code itself is a bit of an abomination.
It has its own entire test suite, which also doesn't work very well.
Using a tool like lldb mi is very similar in spirit to how llvm already
uses lit though. There is a collection of very specific tools that exist
just for testing, and the output of those is run through lit)
Lldb-mi was developed independently, had no review, and was essentially a
large code drop. and it also was not developed with testing in mind. I
agree we could probably make it work, but I've been in that code a few
times and I have to say, I don't want to go back.
In any case, i think we agree in principle that from a high level, we could
get a lot of benefits from this style of test.
I'll have more specifics on my thoughts in a few weeks
On Fri, Sep 16, 2016 at 4:07 PM Jason Molenda <jmole...@apple.com> wrote:
> I thought about this more overnight and I'm more convinced that lit and
> lldb-mi make a great pair. lldb-mi is a programmatic text format that
> isn't subject to the whims of command-line UI design over the years; well
> tests written in terms of MI will be resilient and stable. lldb-mi MUCH
> more closely matches a non-lldb developer's view of how a debugger works.
> The syntax is different from what they use, but it isn't hard to figure
> out. I haven't touched MI in four or five years and it only took me a
> couple minutes to figure out how to do run to a breakpoint and print a
> -file-exec-and-symbols "/tmp/a.out"
> -break-insert a.c:10
> -var-create var1 * c
> (I omitted the responses from all the command except -break-insert and
> -var-create to make it easier to read, but go run lldb-mi and play with it
> yourself or read the gdb MI documentation on the web. It's a really simple
> format to work in.)
> Several of the proposal have been special commands in the lldb command
> line driver that have non-changing output styles or the like. Why
> re-invent something that already exists? This makes a ton more sense on
> every front. Yes, it means you can't copy and paste your lldb debug
> session into a test case --- but we've all agreed that that's not something
> we want anyone doing anyway.
> I'm in favor of lit + lldb-mi and think this would add a lot of value as
> an additional way for people to write test cases.
> > On Sep 15, 2016, at 7:40 PM, Zachary Turner <ztur...@google.com> wrote:
> > One thing I wonder about. It seems like everyone is mostly on the same
> page about command line output .
> > What about input? Would anyone have objections to a test which ran a few
> commands to get the debugger into a particular state before doing something
> to verify the output? Let's assume I'm waving my hands about this last step
> but that it doesn't involve scraping the output of a debugger command
> > Maybe this will never even be an issue, but I just want to make sure
> everyone is on the same page and that the objections are:
> > a) specific to command OUTPUT, not input (i.e. it's ok to have a test
> run "break set -n main")
> > b) specific to *non trivial* output (checking that "p 5" displays 5 is
> > c) specific to the the output of the *user* command line interface, so
> that some hypothetical other command line interface (which again I'm being
> hand wavy about) would not be subject to the same objections.
> > On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda <jmole...@apple.com>
> > > On Sep 15, 2016, at 8:02 AM, Zachary Turner <ztur...@google.com>
> > >
> > >
> > > It sounds like your goal is also "tests have to use the SB API and no
> other API", which if so I think that's counterproductive. More
> productive, IMO, would be being open to any alternative that addresses the
> concerns you have with command-line tests. There are more than 2 ways to
> skin a cat, so to speak.
> > Thinking about this a bit, another approach would be to do lit tests on
> top of lldb-mi. MI is a structured format for the debugger and a UI to
> communicate back and forth with a simple text markup language (it would be
> JSON if it were being done today, but it was added to gdb eighteen years
> ago, so it's not). The commands correspond to the commands a debugger user
> would think to use -- no need to understand the structure of how lldb is
> implemented, like with the SB API. The format is a little unusual for a
> human to type, but back when we supported gdb at Apple we worked in MI all
> the time (it was used to communicate between Xcode, our IDE, and gdb) by
> hand when testing and it was fine. "-exec-run" instead of run, that kind of
> thing. I think there are four dozens different commands.
> > lldb-mi itself uses SB API. And the concerns about hardcoding command
> line UI don't apply, it's a key-value format intended for computers, no one
> is going to add an extra space character to anything -- the most it changes
> is that new key-value pairs are added to responses.
> > I agree there are many acceptable ways to write lit tests that don't
> involve lldb command line scraping, and I'm all in favor of using lit with
> tests that don't do that. Of course the patch we're discussing has lit
> test examples that contradict our own policy on writing tests. Once lit is
> supported in lldb, are we going to reject any testsuite additions that
> depend on the text output from the command line lldb? If we're all on the
> same page here, then I have no reservations.
> > Just to say out loud the future I can easily see: We add lit, then we
> have people helpfully write a few dozen tests in lit that depend on the
> command line debugger output. Those patches have to be rejected.
> > J
lldb-commits mailing list