On 2017-03-02 12:19, Erik Joelsson wrote:
I don't think I like this part. It's not uncommon to expect non zero
return when tests are failing even in developer sessions. If we are to
ever convert to using this new run-test for automated systems, which
we really should, it must return non zero on failures.
While this is probably true, that's not the only thing that needs
adapting for having this run in automated systems. At this point in
time, the goal was limited to providing a good developer experience. I
hope too that we can expand this framework for using it in distributed
test systems, but that needs much more work, and will likely be more
intrusive.
I'm guessing you added this to avoid the extra failure printing from
the build system.
Well yes and no. In the old test framework, the behavior was not
consistent whether to exit on failed tests result or not. I chose the
stance that having successful make execution of the tests, even if some
tests fails, warranted a successful make execution.
This can of course be changed to the reverse so failed tests always lead
to a make failure exit, or have the behavior selected by the user.
Surely this can be worked around differently?
Yes, but not so unobtrusively. I wanted to have this change make a
minimal impact on existing code.
My suggestion is that we keep the current functionality, and work on
getting a way to return non-zero results from test failures as part of a
further development of this framework for distributed testing.
/Magnus
