> On Sep 19, 2016, at 2:11 PM, Zachary Turner via lldb-dev
> <firstname.lastname@example.org> wrote:
> On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote:
> > • Separate testing tools
> > • One question that remains open is how to represent
> > the complicated needs of a debugger in lit tests. Part a) above covers the
> > trivial cases, but what about the difficult cases? In
> > https://reviews.llvm.org/D24591 a number of ideas were discussed. We
> > started getting to this idea towards the end, about a separate tool which
> > has an interface independent of the command line interface and which can be
> > used to test. lldb-mi was mentioned. While I have serious concerns about
> > lldb-mi due to its poorly written and tested codebase, I do agree in
> > principle with the methodology. In fact, this is the entire philosophy
> > behind lit as used with LLVM, clang, lld, etc.
> We have a public API... Why are we going to go and spend _any_ time on doing
> anything else? I just don't get it. What a waste of time. We have a public
> API. Lets use it. Not simple enough for people? Tough. Testing a debugger
> should be done using the public API except when we are actually trying to
> test the LLDB command line in pexpect like tests. Those and only those are
> fine to covert over to using LIT and use text scraping. But 95% of our
> testing should be done using the API that our IDEs use. Using MI? Please no.
> FWIW, I agree with you about mi.
> That said, the public api is problematic. Here you've got an API which, once
> you do something to, is set in stone forever. And yet in order to test
> something, you have to add it to the API. So now, in order to test
> something, you have to make a permanent change to a public API that can never
> be undone.
> It's also a public facing API, when a lot of the stuff we need to be able to
> look at and inspect is not really suitable for public consumption.
> If something is a public API that can never be changed once it's added, then
> there should be an extremely strict review process in order to add something
> to it. It should be VERY HARD to modify (it's currently not, which is a
> debate for another day, but anyway...). And yet that's fundamentally
> incompatible with having tests be easy to write. When it's hard to add the
> thing you need to add in order to write the test, then it's hard to write the
> Furthermore, with a system such as the one I proposed, you could even run
> tests with LLDB_DISABLE_PYTHON. IDK about you, but that would be *amazing*
> from my point of view. Even though I spent months making Python 3 work, I
> would be happy to burn it all with fire and go back to just Python 2 if we
> had a viable testing strategy.
It would not be amazing. One of the primary benefits of lldb IS the Python
API. If you don't think that's wonderful in a debugger, go look at what effort
it took to make that happen in gdb once lldb started providing it and you'll
see that it was really highly motivated. So switching our testing away from
one of our very strong points would be a very bad choice IMHO.
On the "should I add it to the SB API to test it or not question." Most of the
time, when I find I need to add an API in order write a test for something, the
thing I'm trying to get my hands on is something that other people might be
interested in as well. If we go to the mode of adding affordances only for
ourselves when writing for testing, we remove one of the times you should be
asking yourself "should I add this to the SB API so other people can use it."
I'm fine with having an lldb-testing module with API's based on the same
methodology as the SB API's. If you think something you are adding is too
esoteric to be part of the SB API put it there. But by default put it into the
> lldb-dev mailing list
lldb-dev mailing list