I don't know if it helps or not, but jtreg defines the following exit codes:
0: OK
1: No tests to run ... none specified, or no tests to run in the
specified set
2: Some tests failed ... jtreg ran the test and but the test did not pass
3: Some tests had an error ... jtreg could not run the test
4: Bad command-line args ... user error
5: Something else bad happened ... typically a configuration issue
6: Exception/crash ... oops
The intent is that they are ordered by seriousness, so that you can set
a simple numeric threshhold of what you consider an acceptable outcome.
-- Jon
On 3/2/17 1:25 PM, Magnus Ihse Bursie wrote:
On 2017-03-02 15:37, Erik Joelsson wrote:
On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
On 2017-03-02 12:19, Erik Joelsson wrote:
I don't think I like this part. It's not uncommon to expect non
zero return when tests are failing even in developer sessions. If
we are to ever convert to using this new run-test for automated
systems, which we really should, it must return non zero on failures.
While this is probably true, that's not the only thing that needs
adapting for having this run in automated systems. At this point in
time, the goal was limited to providing a good developer experience.
I hope too that we can expand this framework for using it in
distributed test systems, but that needs much more work, and will
likely be more intrusive.
I'm guessing you added this to avoid the extra failure printing
from the build system.
Well yes and no. In the old test framework, the behavior was not
consistent whether to exit on failed tests result or not. I chose
the stance that having successful make execution of the tests, even
if some tests fails, warranted a successful make execution.
In which cases were there inconsistency? JPRT certainly relies on
make failing for everything it was running. That would at least cover
the vast majority of cases actually in use. I would go as far as
saying any other cases not currently conforming should be viewed as
bugs.
For instance, if you set TREAT_EXIT_CODE_1_AS_0 then, as the variable
says, exit code 1 will be treated as exit code 0. For the langtools
test Makefile, you could either set EXIT=true to avoid (!) having it
exit on failure. Also, langtools jtreg uses a EXIT_IF_FATAL check
which compares > FATAL_JTREG_EXIT = 3. In one of our internal test
suites, the default behavior is to use exit code 0 even on test
failures, and our current make wrapper looks for the existence of a
file to trigger a non-zero exit. So it's a bit messy.
This can of course be changed to the reverse so failed tests always
lead to a make failure exit, or have the behavior selected by the user.
Surely this can be worked around differently?
Yes, but not so unobtrusively. I wanted to have this change make a
minimal impact on existing code.
My suggestion is that we keep the current functionality, and work on
getting a way to return non-zero results from test failures as part
of a further development of this framework for distributed testing.
I suggest that we change it to fail on failed tests instead of
changing the behavior of the current test mechanism.
I'm not sure I understand what you mean. My patch does not change any
behavior of the current test mechanism. I will not introduce any such
changes, not even if you wanted me to. :-) So that sounds like a false
dichotomy.
I can change the new run-test system to fail on failed test. To keep
it as readable as it is now, I would need to make a hack with the
top-level FailureHandler. This sure can be done, but increases the
risk of unintended consequences. But since you feel so strongly about
this, I assume that's the way to go.
/Magnus
There will likely be other changes needed before automated systems
can use this, but this is a very fundamental part of the API. If make
doesn't fail, any wrapping tool/script/system is unable to know if
the run was successful. All other build systems I know of that run
tests do this (gradle, maven, ant etc).
/Erik
/Magnus
