> On Sep 19, 2016, at 2:11 PM, Zachary Turner <ztur...@google.com> wrote:
> On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote:
> >               • Separate testing tools
> >                       • One question that remains open is how to represent 
> > the complicated needs of a debugger in lit tests.  Part a) above covers the 
> > trivial cases, but what about the difficult cases?  In 
> > https://reviews.llvm.org/D24591 a number of ideas were discussed.  We 
> > started getting to this idea towards the end, about a separate tool which 
> > has an interface independent of the command line interface and which can be 
> > used to test.  lldb-mi was mentioned.  While I have serious concerns about 
> > lldb-mi due to its poorly written and tested codebase, I do agree in 
> > principle with the methodology.  In fact, this is the entire philosophy 
> > behind lit as used with LLVM, clang, lld, etc.
> We have a public API... Why are we going to go and spend _any_ time on doing 
> anything else? I just don't get it. What a waste of time. We have a public 
> API. Lets use it. Not simple enough for people? Tough. Testing a debugger 
> should be done using the public API except when we are actually trying to 
> test the LLDB command line in pexpect like tests. Those and only those are 
> fine to covert over to using LIT and use text scraping. But 95% of our 
> testing should be done using the API that our IDEs use. Using MI? Please no.
> FWIW, I agree with you about mi.
> That said, the public api is problematic.  Here you've got an API which, once 
> you do something to, is set in stone forever.  And yet in order to test 
> something, you have to add it to the API.  So now, in order to test 
> something, you have to make a permanent change to a public API that can never 
> be undone.

So let me clarify: I don't want people adding to the public API for the sole 
reason of testing.. That falls into the "test another way" category. So I 
should have said "add it to the public API if it would be a valid addition to 
the public API.
> It's also a public facing API, when a lot of the stuff  we need to be able to 
> look at and inspect is not really suitable for public consumption.

Again, this falls into the "test another way" category as is perfect for gtest 
or lit.
> If something is a public API that can never be changed once it's added, then 
> there should be an extremely strict review process in order to add something 
> to it.  It should be VERY HARD to modify (it's currently not, which is a 
> debate for another day, but anyway...).  And yet that's fundamentally 
> incompatible with having tests be easy to write.  When it's hard to add the 
> thing you need to add in order to write the test, then it's hard to write the 
> test.

Yes yes yes. Sorry about the confusion. My first sentence should clear this 

> Furthermore, with a system such as the one I proposed, you could even run 
> tests with LLDB_DISABLE_PYTHON.  IDK about you, but that would be *amazing* 
> from my point of view.  Even though I spent months making Python 3 work, I 
> would be happy to burn it all with fire and go back to just Python 2 if we 
> had a viable testing strategy.

I am not opposed to that. That being say we could start having most API tests 
actually use the C++ public API and have the tests written in C++. We would 
need a .a file with some canned functionality in it, but anything that is 
testing the public API could just be C++. 

So I definitely don't want people adding to the public API just for testing. 
The rules should be:
- Does my feature belong in the public API?
  - Yes, great, add it and test it there
  - No, test it with gtest or lit


lldb-dev mailing list

Reply via email to