BTW, another way to achieve this worthy goal is to require that all the lldb command-line commands be written using the C++ Version of the SB API's. It should be its own module, and the driver should link to that, not directly to the LLDB.framework at all. That would also force us to provide a way to make "real" command line commands using the SB API, rather than commands that parse their arguments and options and provide help in their own ad-hoc way, and don't do command line completion at all.
I don't think we can force Python here, partly because I don't want to favor a particular scripting language at this fundamental a level, but more because there are legit targets where providing Python is problematic. But this would test everything up to the SWIG bindings, and make sure they stay full-featured enough for real work. We didn't do it this way originally because getting the commands up and working and building on them provided a more plausible development path than having to design our public API before we could start testing it well. I am pretty sure that was the right choice at the time, but the reasons are less relevant now that we've got a working debugger and a pretty well designed public API. Jim > On Sep 19, 2016, at 3:49 PM, Jim Ingham via lldb-dev > <lldb-dev@lists.llvm.org> wrote: > >> >> On Sep 19, 2016, at 3:32 PM, Zachary Turner <ztur...@google.com> wrote: >> >> Ok, in that case I agree with you more. We should test the scripting >> interface. It's part of the software, it should be treated as such. 100% >> on board. But if we find that it is lacking (or even just inconvenient) to >> test the full capabilities of the debugger, we shouldn't force it to. > > I would make an even stronger statement. As lldb developers, if there's > something we want to do and it isn't available, we go add it to lldb_private, > and maybe wire it up to a command so we can use it. That's the path of least > resistance. If it weren't for writing the testsuite using the SB API's, we > wouldn't tend to write much code using it ourselves. Writing test cases for > the Python API in the testsuite is a nice forcing function to get US using > the SB API's to solve actual problems. So where reasonable, I think we > actually SHOULD force ourselves to use it. > > There are some things it is annoying to test from the outside, particularly > small utility functions that you need to test error cases and the like that > it's just hard to get the actual debugger to generate reliably. Those are > great candidates for white box type unit testing. But the full capabilities > of the debugger should either be available through the SB API's or they > really aren't doing anybody any good. So if the SB API's are not up to > revealing the full capabilities of the debugger, we need to fix that, not > fall back on using stuff only lldb developers can have access just so we can > get our tests written more quickly. > > Jim > > >> >> All of our tests right now are full-scale integration tests. Integration >> tests are certainly helpful, but they aren't the entire world. And they >> shouldn't even be a first line of defense, but rather a 3rd or 4th line of >> defense due to how general they are. Like the example I gave earlier about >> TestSigned and TestUnsigned on Windows, where one was failing and one was >> passing, and the reason had nothing to do with signedness. >> >> On Mon, Sep 19, 2016 at 3:24 PM Jim Ingham <jing...@apple.com> wrote: >> >>> On Sep 19, 2016, at 3:19 PM, Zachary Turner <ztur...@google.com> wrote: >>> >>> Obviously I defer to you on whether testing via the SB API is better than >>> what GDB does or used to do. But these are not the only two systems in the >>> world. Having used both LLDB and LLVM's test suite extensively, I still >>> remain unconvinced that LLDB's testing situation could not be improved. Do >>> we improve it by doing what GDB did? Obviously not. Can we improve it >>> further by doing something completely different than what GDB did *and* >>> what LLDB currently does? I remain convinced that we can. >> >> I think you misread my message. I said: >> >> Having a Python interface so that developers can program the debugger is >> very powerful. >> >> As evidence, see all the work the gdb folks did to add one (you have to know >> a little bit about how gdb works to see how hard this would be, but you can >> imagine a debugger written in C primarily to dump text to a terminal, think >> of how you would wrap that in Python and you'll get a sense.) >> >> Now forget about gdb, that was only an example of how much some folks >> thought a scripting interface was worth. >> >> Instead, focus on this good thing, the scripting interface. We should test >> this thing that we think is valuable. >> >> What's a good way to make sure we test that? >> >> USE IT FOR TESTING. >> >> Jim >> >> >>> >>> >>> On Mon, Sep 19, 2016 at 3:07 PM Jim Ingham <jing...@apple.com> wrote: >>> >>>> On Sep 19, 2016, at 2:11 PM, Zachary Turner via lldb-dev >>>> <lldb-dev@lists.llvm.org> wrote: >>>> >>>> >>>> >>>> On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote: >>>> >>>>> • Separate testing tools >>>>> • One question that remains open is how to represent >>>>> the complicated needs of a debugger in lit tests. Part a) above covers >>>>> the trivial cases, but what about the difficult cases? In >>>>> https://reviews.llvm.org/D24591 a number of ideas were discussed. We >>>>> started getting to this idea towards the end, about a separate tool which >>>>> has an interface independent of the command line interface and which can >>>>> be used to test. lldb-mi was mentioned. While I have serious concerns >>>>> about lldb-mi due to its poorly written and tested codebase, I do agree >>>>> in principle with the methodology. In fact, this is the entire >>>>> philosophy behind lit as used with LLVM, clang, lld, etc. >>>> >>>> We have a public API... Why are we going to go and spend _any_ time on >>>> doing anything else? I just don't get it. What a waste of time. We have a >>>> public API. Lets use it. Not simple enough for people? Tough. Testing a >>>> debugger should be done using the public API except when we are actually >>>> trying to test the LLDB command line in pexpect like tests. Those and only >>>> those are fine to covert over to using LIT and use text scraping. But 95% >>>> of our testing should be done using the API that our IDEs use. Using MI? >>>> Please no. >>>> >>>> FWIW, I agree with you about mi. >>>> >>>> That said, the public api is problematic. Here you've got an API which, >>>> once you do something to, is set in stone forever. And yet in order to >>>> test something, you have to add it to the API. So now, in order to test >>>> something, you have to make a permanent change to a public API that can >>>> never be undone. >>>> >>>> It's also a public facing API, when a lot of the stuff we need to be able >>>> to look at and inspect is not really suitable for public consumption. >>>> >>>> If something is a public API that can never be changed once it's added, >>>> then there should be an extremely strict review process in order to add >>>> something to it. It should be VERY HARD to modify (it's currently not, >>>> which is a debate for another day, but anyway...). And yet that's >>>> fundamentally incompatible with having tests be easy to write. When it's >>>> hard to add the thing you need to add in order to write the test, then >>>> it's hard to write the test. >>>> >>>> Furthermore, with a system such as the one I proposed, you could even run >>>> tests with LLDB_DISABLE_PYTHON. IDK about you, but that would be >>>> *amazing* from my point of view. Even though I spent months making Python >>>> 3 work, I would be happy to burn it all with fire and go back to just >>>> Python 2 if we had a viable testing strategy. >>> >>> It would not be amazing. One of the primary benefits of lldb IS the Python >>> API. If you don't think that's wonderful in a debugger, go look at what >>> effort it took to make that happen in gdb once lldb started providing it >>> and you'll see that it was really highly motivated. So switching our >>> testing away from one of our very strong points would be a very bad choice >>> IMHO. >>> >>> On the "should I add it to the SB API to test it or not question." Most of >>> the time, when I find I need to add an API in order write a test for >>> something, the thing I'm trying to get my hands on is something that other >>> people might be interested in as well. If we go to the mode of adding >>> affordances only for ourselves when writing for testing, we remove one of >>> the times you should be asking yourself "should I add this to the SB API so >>> other people can use it." I'm fine with having an lldb-testing module with >>> API's based on the same methodology as the SB API's. If you think >>> something you are adding is too esoteric to be part of the SB API put it >>> there. But by default put it into the SB API's. >>> >>> Jim >>> >>> >>>> _______________________________________________ >>>> lldb-dev mailing list >>>> lldb-dev@lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>> >> > > _______________________________________________ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
