Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
Seems reasonable. On Fri, Dec 11, 2015 at 11:46 AM, Zachary Turner wrote: > I think the correct fix is to just not throw the exception, and do it the > right way. unittest doesn't even use an exception for that anymore, but > has some kind of method to tag the test as xfail or skip, whereas our

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
I think the correct fix is to just not throw the exception, and do it the right way. unittest doesn't even use an exception for that anymore, but has some kind of method to tag the test as xfail or skip, whereas our unittest2 uses an exceptionf or the same purpose. Basically, we need to look at t

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
Okay. Sounds like something we can work around one way or another, either by introducing the correct exception name for unittest, or introducing our own if we need to do so. On Fri, Dec 11, 2015 at 11:22 AM, Zachary Turner wrote: > If I remember correctly it was in the way we had implemented on

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
If I remember correctly it was in the way we had implemented one of the expected fail decorators. We were manually throwing some kind of exception to indicate an xfail or a skip, and that exception doesn't exist in the upstream unittest. Basically, we were relying on an implementation detail of u

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
I think we can do this, and I'd like us to do this unless it's proven to break something we're not aware of. I think you did some research on this after we discussed last, but something (maybe in the decorators) didn't just work. Was that right? On Fri, Dec 11, 2015 at 11:18 AM, Zachary Turner

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
Also at some point I will probably want to kill unittest2 and move to the upstream unittest. AFAICT we only use unittest2 because it works on 2.6 and unittest doesn't. But now that we're ok with saying 2.6 is unsupported, we can in theory go to the upstream unittest. On Fri, Dec 11, 2015 at 11:1

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
On Fri, Dec 11, 2015 at 11:17 AM, Zachary Turner wrote: > Not sure I follow. Are you trying to test the execution engine itself > (dotest.py, lldbtest.py, etc) > This. Test the lldb highly-specialized test runner internals. > or are you trying to have another alternative to running individua

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
Not sure I follow. Are you trying to test the execution engine itself (dotest.py, lldbtest.py, etc) or are you trying to have another alternative to running individual tests? The if __name__ == "__main__": unittest.main() stuff was deleted deleted from all tests a few months ago as part of

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
The tests end up looking substantially similar to our lldb test suite tests, as they were based on unittest2, which is/was a relative of unittest that now lives in Python. The docs for unittest in python 2.x have generally been accurate for the unittest2 lib we use. At least, for the areas I use.

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
It just requires running the test file as a python script. The runner is fired off like this: if __name__ == "__main__": unittest.main() which is typically added to the bottom of all test files so you can call it directly. -Todd On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala wrote: > Unitt

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
Unittest. Comes with Python. On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner wrote: > Presumably those tests use an entirely different, hand-rolled test running > infrastructure? > > On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala wrote: > >> One thing I want to make sure we can do is have a sane

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
Presumably those tests use an entirely different, hand-rolled test running infrastructure? On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala wrote: > One thing I want to make sure we can do is have a sane way of storing and > running tests that test the test execution engine. Those are tests that >

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
One thing I want to make sure we can do is have a sane way of storing and running tests that test the test execution engine. Those are tests that should not run as part of an "lldb test run". These are tests that maintainers of the test system run to make sure we're not breaking stuff when we to

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
I like it. On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner wrote: > Yea wasn't planning on doing this today, just throwing the idea out there. > > On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala wrote: > >> I'm fine with the idea. >> >> FWIW the test events model will likely shift a bit, as it is cu

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Zachary Turner via lldb-dev
Yea wasn't planning on doing this today, just throwing the idea out there. On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala wrote: > I'm fine with the idea. > > FWIW the test events model will likely shift a bit, as it is currently a > single sink, whereas I am likely to turn it into a test event filt

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Todd Fiala via lldb-dev
I'm fine with the idea. FWIW the test events model will likely shift a bit, as it is currently a single sink, whereas I am likely to turn it into a test event filter chain shortly here. Formatters still make sense as they'll be the things at the end of the chain. Minor detail, result_formatter.p

Re: [lldb-dev] Separating test runner and tests

2015-12-11 Thread Pavel Labath via lldb-dev
Sounds like a reasonable thing to do. A couple of tiny remarks: - when you do the move, you might as well rename dotest into something else, just to avoid the "which dotest should I run" type of questions... - there is nothing that makes it obvious that "engine" is actually a "test running engine",

[lldb-dev] Separating test runner and tests

2015-12-10 Thread Zachary Turner via lldb-dev
Currently our folder structure looks like this: lldbsuite |-- test |-- dotest.py |-- dosep.py |-- lldbtest.py |-- ... |-- functionalities |-- lang |-- expression_command |-- ... etc I've been thinking about organizing it like this instead: lldbsuite |-- test |