> On Aug 15, 2018, at 10:15 AM, Jim Ingham via lldb-dev
> <lldb-dev@lists.llvm.org> wrote:
>
>
>
>> On Aug 14, 2018, at 7:48 PM, Zachary Turner via lldb-dev
>> <lldb-dev@lists.llvm.org> wrote:
>>
>>
>> On Tue, Aug 14, 2018 at 6:58 PM Jason Molenda <jmole...@apple.com> wrote:
>>
>>
>>> On Aug 14, 2018, at 6:39 PM, Zachary Turner <ztur...@google.com> wrote:
>>>
>>> Having bugs also makes the debugger harder to innovate in the future
>>> because it’s, not having tests leads to having bugs, and sb api tests leads
>>> to not having te
>>
>> Yes, lldb does not have these problems -- because we learned from our
>> decades working on gdb, and did not repeat that mistake. To be honest, lldb
>> is such a young debugger - barely a decade old, depending on how you count
>> it, that ANY testsuite approach would be fine at this point. Add a couple
>> more decades and we'd be back into the hole that gdb was in. {I have not
>> worked on gdb in over a decade, so I don't know how their testing
>> methodology may be today}
>> That doesn’t mean that the current approach is the final word. As new
>> people come onto the project, new ideas come forth and we should entertain
>> them rather than deciding that all decisions are set in stone forever.
>>
>> For example, the object model based approach I mentioned earlier would not
>> have any of the problems that you’ve described from gdb. Just because one
>> set of problems has been solved doesn’t mean we should declare victory and
>> say there’s no point in trying to solve the remaining problems too. And
>> right now, the problem is that we need to be coming up with a way to make
>> tests easier to write so that people will actually write them
>
> Just historically it is not true that we didn't have problems with command
> output scraping vrs. the testsuite. The major offender in this regard was
> breakpoint setting/checking. But the first time I had to go adjust a whole
> bunch of tests when I wanted to change the breakpoint output, I added test
> routines to do all this business, which mitigated the need for these changes.
> It is true I haven't had to fiddle with this much then - though that is
> mostly because I was careful in how I checked output, and didn't check
> inessential aspects. That's easy to do when you sit down and do it once, but
> if everybody just wings some regex every time they do a check you will end up
> with lots over-eager checks that cause problems down the road.
I definitely sympathize with this, but don't think the problems that arise in
some tests which pattern-match against input are characteristic of all such
tests. The lesson you're describing -- the need to be careful to only examine
the interesting parts of the API/driver output -- applies generally.
> I also want to reiterate another point Jason made, which is that LLDB not
> only provides a command line interface, but it also provides an API. Again,
> we've gone over this before but I'm pretty sure that at this point it's still
> true that in terms of number of users, far more people interact with lldb
> through the API (Xcode, VSCode through the gdb-mi or through Greg's new
> adaptor when that's done...) than interact with it through the command line.
> And writing SB API tests when you add a new feature is a great way to ensure
> that you have thought about exposing your new feature to the SB API's
> correctly. In that example, a GUI needs to have a way to indicate that these
> "tail call" frames are special and you shouldn't be worried that they have no
> local variables, even though we show you source for them that obviously has
> variables...
I agree with all of this. However, I don't take it to mean that there ought not
be tests for the driver output.
> So we need ways to encourage lldb developers to be aware that the SB API's
> need to have (tested) access to new features in the debugger and not just the
> command line. SB API tests are good way to encourage this. We can also do
> it as part of reviews and the like, but I think it better done as part of
> your development than after the fact in reviews... Again, that doesn't argue
> for their being the ONLY way to test things in lldb, but it does need to be
> part of an lldb developer's practice.
Totally agree.
vedant
> I like your idea of some kind of object model based output for test checking.
> I somehow had the impression that that was your intent for lldb-test once it
> was a real boy, but maybe I was just projecting... Relying on ad hoc dumping
> routines which are mostly for debugging purposes seemed okay as a boot-strap
> method, but not a great design long-term. Teaching either the SB or the
> lldb_private objects to describe themselves in a structured way and using
> that to do testing seems great.
>
> Jim
>
>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
