On Sat, 2 Feb 2019, D. Hugh Redelmeier wrote:
| It’s a bug but an old one. It might be triggered now with some secrets file
install change
So: is this test known to fail?
For this one I am not entirely sure. For some others we do know that
they are failing because work needs to be done.
If it is, it is a waste of time and a cognative burden for each person
testing.
We have gone through a few rounds of discussions on this. There are no
good options.
- we need an announced category of "expected failures"
It won't help, because people will always ignore it and never fix it.
- we need reference logs reflecting such bugs so that when something
changes in the output we get a signal.
If we ship with known bugs, we can have the test system show them as
bugs?
- if the expected failure creates non-deterministic results, we need
to declare that too
non-deterministic is something no one wants. It just happens due to a
number of known and unknown reasons.
The alternative is to ignore any test that failed "before". That's
negligent, but it's what most of us do.
Yes, there are no good options, other than getting more resources to fix
things :P
Paul
_______________________________________________
Swan-dev mailing list
[email protected]
https://lists.libreswan.org/mailman/listinfo/swan-dev
