> On Sep 19, 2016, at 2:11 PM, Zachary Turner <ztur...@google.com> wrote: > > > > On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote: > > > • Separate testing tools > > • One question that remains open is how to represent > > the complicated needs of a debugger in lit tests. Part a) above covers the > > trivial cases, but what about the difficult cases? In > > https://reviews.llvm.org/D24591 a number of ideas were discussed. We > > started getting to this idea towards the end, about a separate tool which > > has an interface independent of the command line interface and which can be > > used to test. lldb-mi was mentioned. While I have serious concerns about > > lldb-mi due to its poorly written and tested codebase, I do agree in > > principle with the methodology. In fact, this is the entire philosophy > > behind lit as used with LLVM, clang, lld, etc. > > We have a public API... Why are we going to go and spend _any_ time on doing > anything else? I just don't get it. What a waste of time. We have a public > API. Lets use it. Not simple enough for people? Tough. Testing a debugger > should be done using the public API except when we are actually trying to > test the LLDB command line in pexpect like tests. Those and only those are > fine to covert over to using LIT and use text scraping. But 95% of our > testing should be done using the API that our IDEs use. Using MI? Please no. > > FWIW, I agree with you about mi. > > That said, the public api is problematic. Here you've got an API which, once > you do something to, is set in stone forever. And yet in order to test > something, you have to add it to the API. So now, in order to test > something, you have to make a permanent change to a public API that can never > be undone.
So let me clarify: I don't want people adding to the public API for the sole reason of testing.. That falls into the "test another way" category. So I should have said "add it to the public API if it would be a valid addition to the public API. > > It's also a public facing API, when a lot of the stuff we need to be able to > look at and inspect is not really suitable for public consumption. Again, this falls into the "test another way" category as is perfect for gtest or lit. > > If something is a public API that can never be changed once it's added, then > there should be an extremely strict review process in order to add something > to it. It should be VERY HARD to modify (it's currently not, which is a > debate for another day, but anyway...). And yet that's fundamentally > incompatible with having tests be easy to write. When it's hard to add the > thing you need to add in order to write the test, then it's hard to write the > test. Yes yes yes. Sorry about the confusion. My first sentence should clear this up... > > Furthermore, with a system such as the one I proposed, you could even run > tests with LLDB_DISABLE_PYTHON. IDK about you, but that would be *amazing* > from my point of view. Even though I spent months making Python 3 work, I > would be happy to burn it all with fire and go back to just Python 2 if we > had a viable testing strategy. I am not opposed to that. That being say we could start having most API tests actually use the C++ public API and have the tests written in C++. We would need a .a file with some canned functionality in it, but anything that is testing the public API could just be C++. So I definitely don't want people adding to the public API just for testing. The rules should be: - Does my feature belong in the public API? - Yes, great, add it and test it there - No, test it with gtest or lit Greg _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
