I have been thinking about something similar. Ildb-mi specifically I have serious concerns about because the code itself is a bit of an abomination. It has its own entire test suite, which also doesn't work very well.
Using a tool like lldb mi is very similar in spirit to how llvm already uses lit though. There is a collection of very specific tools that exist just for testing, and the output of those is run through lit) Lldb-mi was developed independently, had no review, and was essentially a large code drop. and it also was not developed with testing in mind. I agree we could probably make it work, but I've been in that code a few times and I have to say, I don't want to go back. In any case, i think we agree in principle that from a high level, we could get a lot of benefits from this style of test. I'll have more specifics on my thoughts in a few weeks On Fri, Sep 16, 2016 at 4:07 PM Jason Molenda <jmole...@apple.com> wrote: > I thought about this more overnight and I'm more convinced that lit and > lldb-mi make a great pair. lldb-mi is a programmatic text format that > isn't subject to the whims of command-line UI design over the years; well > tests written in terms of MI will be resilient and stable. lldb-mi MUCH > more closely matches a non-lldb developer's view of how a debugger works. > The syntax is different from what they use, but it isn't hard to figure > out. I haven't touched MI in four or five years and it only took me a > couple minutes to figure out how to do run to a breakpoint and print a > variable: > > -file-exec-and-symbols "/tmp/a.out" > > -break-insert a.c:10 > > =breakpoint-modified,bkpt={number="6",type="breakpoint",disp="keep",enabled="y",addr="0x0000000100000f70",func="main",file="a.c",fullname="/tmp/a.c",line="10",times="0",original-location="a.c:10"} > > -exec-run > > -thread-info > > -exec-next > > -var-create var1 * c > > ^done,name="var1",numchild="0",value="12",type="int",thread-id="1",has_more="0" > > > (I omitted the responses from all the command except -break-insert and > -var-create to make it easier to read, but go run lldb-mi and play with it > yourself or read the gdb MI documentation on the web. It's a really simple > format to work in.) > > > Several of the proposal have been special commands in the lldb command > line driver that have non-changing output styles or the like. Why > re-invent something that already exists? This makes a ton more sense on > every front. Yes, it means you can't copy and paste your lldb debug > session into a test case --- but we've all agreed that that's not something > we want anyone doing anyway. > > I'm in favor of lit + lldb-mi and think this would add a lot of value as > an additional way for people to write test cases. > > > J > > > > > On Sep 15, 2016, at 7:40 PM, Zachary Turner <ztur...@google.com> wrote: > > > > One thing I wonder about. It seems like everyone is mostly on the same > page about command line output . > > > > What about input? Would anyone have objections to a test which ran a few > commands to get the debugger into a particular state before doing something > to verify the output? Let's assume I'm waving my hands about this last step > but that it doesn't involve scraping the output of a debugger command > > > > Maybe this will never even be an issue, but I just want to make sure > everyone is on the same page and that the objections are: > > > > a) specific to command OUTPUT, not input (i.e. it's ok to have a test > run "break set -n main") > > b) specific to *non trivial* output (checking that "p 5" displays 5 is > ok) > > c) specific to the the output of the *user* command line interface, so > that some hypothetical other command line interface (which again I'm being > hand wavy about) would not be subject to the same objections. > > > > > > On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda <jmole...@apple.com> > wrote: > > > > > On Sep 15, 2016, at 8:02 AM, Zachary Turner <ztur...@google.com> > wrote: > > > > > > > > > It sounds like your goal is also "tests have to use the SB API and no > other API", which if so I think that's counterproductive. More > productive, IMO, would be being open to any alternative that addresses the > concerns you have with command-line tests. There are more than 2 ways to > skin a cat, so to speak. > > > > Thinking about this a bit, another approach would be to do lit tests on > top of lldb-mi. MI is a structured format for the debugger and a UI to > communicate back and forth with a simple text markup language (it would be > JSON if it were being done today, but it was added to gdb eighteen years > ago, so it's not). The commands correspond to the commands a debugger user > would think to use -- no need to understand the structure of how lldb is > implemented, like with the SB API. The format is a little unusual for a > human to type, but back when we supported gdb at Apple we worked in MI all > the time (it was used to communicate between Xcode, our IDE, and gdb) by > hand when testing and it was fine. "-exec-run" instead of run, that kind of > thing. I think there are four dozens different commands. > > > > lldb-mi itself uses SB API. And the concerns about hardcoding command > line UI don't apply, it's a key-value format intended for computers, no one > is going to add an extra space character to anything -- the most it changes > is that new key-value pairs are added to responses. > > > > > > I agree there are many acceptable ways to write lit tests that don't > involve lldb command line scraping, and I'm all in favor of using lit with > tests that don't do that. Of course the patch we're discussing has lit > test examples that contradict our own policy on writing tests. Once lit is > supported in lldb, are we going to reject any testsuite additions that > depend on the text output from the command line lldb? If we're all on the > same page here, then I have no reservations. > > > > Just to say out loud the future I can easily see: We add lit, then we > have people helpfully write a few dozen tests in lit that depend on the > command line debugger output. Those patches have to be rejected. > > > > J > >
_______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits