> On Sep 11, 2015, at 11:47 AM, Zachary Turner <ztur...@google.com> wrote: > > We'll probably rewrite tests that we find are failing specifically as a > result of issues like this, but I agree it's not worth re-writing everything > else except on an as-needed basis. > > To make the distinction explicit and enforce it kind of at an organnizational > level, would it be worth creating folders under lldb/test like > lldb/test/commands, and recommending that all HandleCommand tests to go there? > > Possibly unrelated question: But in regards to this api test vs. > HandleCommand test situation, is that what the purpose of the > @python_api_test decorator is (and is this decorator even still useful)?
I think it might have had some utility back when the SB API's were new and we were shuffling them around a lot, though I never found a use for it. I don't see any reason for it to exist any more. Jim > > On Fri, Sep 11, 2015 at 11:42 AM Jim Ingham <jing...@apple.com> wrote: > I have held from the beginning that the only tests that should be written > using HandleCommand are those that explicitly test command behavior, and if > it is possible to write a test using the SB API you should always do it that > way for the very reasons you cite. Not everybody agreed with me at first, so > we ended up with a bunch of tests that do complex things using HandleCommand > where they really ought not to. I'm not sure it is worth the time to go > rewrite all those tests, but we shouldn't write any new tests that way. > > Jim > > > > On Sep 11, 2015, at 11:33 AM, Zachary Turner via lldb-dev > > <lldb-dev@lists.llvm.org> wrote: > > > > The past few weeks I've spent a lot of time xfailing the rest of the > > failing tests on windows so we can enable tests to run on the bots. > > > > One thing I ran into more frequently than I would have liked is that the > > tests were failing not because the functionality is broken, but because the > > substrings being grepped for in the output had a slightly different format > > on Windows. The pattern for tests is frequently something like: > > > > result = runCommand(<some command>) > > self.expect(<result matches some regex>) > > > > A good example of this is that when you do a backtrace, on windows you > > might see a fully demangled function name such as a.out`void foo(int x), > > whereas on other platforms you might just see a.out`foo. > > > > I saw the reverse situation as well, where a test was passing but shouldn't > > have because it was actually broken, but due to the impreciseness of > > grepping output the grep was suceeding. Specifically, this was happening > > on a test that verified function parameters. it launched a program with 3 > > arguments, and then looked for "(int)argc=3" in the frame info. It was > > broken on Windows because argc was pointing to junk memory, so it was > > actually printing "(int)argc=3248902" in the output. Test was passing, > > even though it was broken. > > > > Rather than make the regexes more complicated, I think the right fix here > > is to stop using the command system and grepping to write tests. Just go > > through the api for everything, including verifying the result. In the > > second case, for example, you launch the process, set the breakpoint, wait > > for it to stop, find the argument named argc, and verify that it's value is > > 3. > > > > I don't want to propose going back and rewriting every single test to do > > this, but I want to see how people feel about moving toward this model > > going forward as the default method of writing tests. > > > > I do still think we need some tests that verify commands run, but I think > > those tests should focus not on doing complicated interactions with the > > debugger, and instead just verifying that things parse correctly and the > > command is configured correctly, with the underlying functionality being > > tested by the api tests. > > > > Thoughts? > > > > > > _______________________________________________ > > lldb-dev mailing list > > lldb-dev@lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
