On 9/28/2016 9:20 PM, Denis Kononenko wrote:
There're 60+ tests related to PKCS11. Two years they have been "passed" on 3
unsupported platforms on hosts where usually no NSS libraries were installed. How can we
rely on these results?
;-) The words of "unsupported platforms" are very confusing in this mail
thread.
Let's think more about what if a test failed. What one will do if a
test failed?
1. Test fail means source code problems for developers. One cannot
submit a change-set if a test failed. He need to pay additional effort
and analysis the failure. It take one developer a lot effort to know
the root cause. I would never like this unnecessary cost.
2. In order to get the test pass, he need to install the NSS libs
although NSS has nothing to do with his changeset. It may be a very
very hard step or even impossible (for example licenses issues) step for
him.
TBH, I did not see much benefits to fail on unsupported platforms. I
agree that skip for pass is not a good idea, but fail to warn is worse.
I think the root cause if "unsupported platforms" actually are supported
platforms, but by accident the NSS libraries are not installed or not
installed properly.
If one is not interested in NSS, the test get ignored (passed). If one
is interested in NSS, he should install the NSS libs and the test get
checked.
What do you think if fix the testing infrastructure with properly
installed NSS libs?
> The problem is the tests report they passed but actually they were
> skipped. I have no objections against skipping tests but it would
> be better to give a hint somehow how many tests were skipped and why.
Agreed. Unfortunately, there are only two options, "pass" or "fail", at
present. It would be nice if there is a grey area. Any idea to make
summary of skipped tests and reasons?
Xuelei
