The run.sh scripts are just a convenience for running test in every subdirectory with TAP output, and in the case of stress tests, calling test.py for every line of param-list.txt in each subdirectory.
AFAIU, the junit tests are the ones driving the traversal in your case, so you probably don't want run.sh at all. However, what would prevent a junit test from invoking e.g. ./test.py prepare 512000 [do test] ./test.py clean 512000 ? Thanks, Mathieu ----- Original Message ----- > From: "Alexandre Montplaisir" <[email protected]> > To: "Mathieu Desnoyers" <[email protected]> > Cc: "Matthew Khouzam" <[email protected]>, "lttng-dev" > <[email protected]>, "Trace Compass > Developer Discussions" <[email protected]> > Sent: Monday, November 17, 2014 12:12:25 AM > Subject: Re: Stress tests added to ctf testsuite > > We can run scripts, yeah. But, as far as I understand it, the ./run.sh > scripts require passing a CTF_READER_BIN executable, which we don't have > during the testing phase. > > Another advantage of using parameterized tests is that it creates one > JUnit test for every instance that is created at run-time (here, one per > test trace to test). This way they show up individually in test reports > [1]. And if one of them fails, at the end we know which one did, and not > just "something failed in the script". > > I see how the test files can be big, so generating *everything* may not > be desirable. Could it be per test then? Or even part of the Python > scripts themselves? (generate -> test -> clean, have a default array of > sizes to generate, user can specify other sizes?) > > Thanks, > Alexandre > > > [1] For example: > https://hudson.eclipse.org/tracecompass/job/tracecompass-master-nightly/31/testReport/org.eclipse.tracecompass.ctf.core.tests.ctftestsuite/CtfTestSuiteTests/ > > On 2014-11-16 03:49 PM, Mathieu Desnoyers wrote: > > Can you modify Junit, or use a different test system, > > to allow calling scripts instead ? I expect we will end up having > > much more automatically generated test-cases over time, and the more > > we will have, the less sense it will make to generate them all at > > test start. > > > > Thanks! > > > > Mathieu > > > > ----- Original Message ----- > >> From: "Alexandre Montplaisir" <[email protected]> > >> To: "Mathieu Desnoyers" <[email protected]>, "Matthew > >> Khouzam" <[email protected]> > >> Cc: "lttng-dev" <[email protected]>, "Trace Compass Developer > >> Discussions" <[email protected]> > >> Sent: Sunday, November 16, 2014 9:39:44 PM > >> Subject: Re: Stress tests added to ctf testsuite > >> > >> Hi Mathieu, > >> > >> Looks great! Preliminary testing shows that Trace Compass also blows up > >> in many different ways, but hey that's expected ;) > >> > >> If I may suggest, having a script, either shell or Python, to just > >> generate/clean the test traces would be very useful. > >> We don't use the run.sh scripts in CI (we're within JUnit tests, there > >> is no executable at that point), we just look at the trace directories > >> and try to run everything in there. If there was a simple way to "create > >> everything" and then "clean everything" it would make it easier to > >> integrate in our (and I guess others') build system. > >> > >> Cheers, > >> Alexandre > >> > >> > >> On 2014-11-16 11:41 AM, Mathieu Desnoyers wrote: > >>> Hi! > >>> > >>> If you look at > >>> https://github.com/efficios/ctf-testsuite > >>> > >>> I just pushed various CTF stress-tests, and did some > >>> general cleanup in the tree (e.g. renaming tests to > >>> meaningful names). > >>> > >>> Have fun! > >>> > >>> Thanks, > >>> > >>> Mathieu > >>> > >> > > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
