Note that the buildbots use dosep.py. Stressing the tests as in dotest.py is useful as it also ensures that lldb is robust. We have other stress tests (i.e. running builds in parallel with test loops) that we occasionally run in private jobs on the buildbots. Just trying to strike a balance.
As long as the tests are run in the same order by dotest.py (at least by default), we'll have deterministic failures. This is a good use-case for pdb since we can break in one of hundreds of tests. Good tests should clean up after themselves, and that also helps the tests serve as examples of lldb usage. Calling these points out on lldb.org could help. What do you think? - Ashok -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Richard Mitton Sent: Tuesday, September 24, 2013 10:00 PM To: [email protected] Subject: [lldb-dev] Test suite itself creating failures Hi all, So I was looking into why TestInferiorAssert was (still) failing on my machine, and it turned out the root cause was in fact that tests are not run in isolation; dotest.py runs multiple tests using the same LLDB context for each one. So if a test doesn't clean up after itself properly, it can cause following tests to incorrectly fail. Is this really a good idea? Wouldn't it make more sense to make it so tests are always run individually, to guarantee consistent results? -- Richard Mitton [email protected] _______________________________________________ lldb-dev mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev _______________________________________________ lldb-dev mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
