> On Sep 19, 2016, at 2:11 PM, Zachary Turner <ztur...@google.com> wrote:
> On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote:
> > • Separate testing tools
> > • One question that remains open is how to represent
> > the complicated needs of a debugger in lit tests. Part a) above covers the
> > trivial cases, but what about the difficult cases? In
> > https://reviews.llvm.org/D24591 a number of ideas were discussed. We
> > started getting to this idea towards the end, about a separate tool which
> > has an interface independent of the command line interface and which can be
> > used to test. lldb-mi was mentioned. While I have serious concerns about
> > lldb-mi due to its poorly written and tested codebase, I do agree in
> > principle with the methodology. In fact, this is the entire philosophy
> > behind lit as used with LLVM, clang, lld, etc.
> We have a public API... Why are we going to go and spend _any_ time on doing
> anything else? I just don't get it. What a waste of time. We have a public
> API. Lets use it. Not simple enough for people? Tough. Testing a debugger
> should be done using the public API except when we are actually trying to
> test the LLDB command line in pexpect like tests. Those and only those are
> fine to covert over to using LIT and use text scraping. But 95% of our
> testing should be done using the API that our IDEs use. Using MI? Please no.
> FWIW, I agree with you about mi.
> That said, the public api is problematic. Here you've got an API which, once
> you do something to, is set in stone forever. And yet in order to test
> something, you have to add it to the API. So now, in order to test
> something, you have to make a permanent change to a public API that can never
> be undone.
So let me clarify: I don't want people adding to the public API for the sole
reason of testing.. That falls into the "test another way" category. So I
should have said "add it to the public API if it would be a valid addition to
the public API.
> It's also a public facing API, when a lot of the stuff we need to be able to
> look at and inspect is not really suitable for public consumption.
Again, this falls into the "test another way" category as is perfect for gtest
> If something is a public API that can never be changed once it's added, then
> there should be an extremely strict review process in order to add something
> to it. It should be VERY HARD to modify (it's currently not, which is a
> debate for another day, but anyway...). And yet that's fundamentally
> incompatible with having tests be easy to write. When it's hard to add the
> thing you need to add in order to write the test, then it's hard to write the
Yes yes yes. Sorry about the confusion. My first sentence should clear this
> Furthermore, with a system such as the one I proposed, you could even run
> tests with LLDB_DISABLE_PYTHON. IDK about you, but that would be *amazing*
> from my point of view. Even though I spent months making Python 3 work, I
> would be happy to burn it all with fire and go back to just Python 2 if we
> had a viable testing strategy.
I am not opposed to that. That being say we could start having most API tests
actually use the C++ public API and have the tests written in C++. We would
need a .a file with some canned functionality in it, but anything that is
testing the public API could just be C++.
So I definitely don't want people adding to the public API just for testing.
The rules should be:
- Does my feature belong in the public API?
- Yes, great, add it and test it there
- No, test it with gtest or lit
lldb-dev mailing list