tstclnt exit code: do we really need to distinguish between failures during PR_Send and PR_Recv?
I am modifying tstclnt to test my patch for bug 542832. With my patch, tstclnt always succeeds or fails like it did before, but in some of the failure cases it exits with exit code 1 when the testcase expects it to exit with exit code 254, and sometimes it exits with exit code 254 when the testcase expects it to exit with exit code 1. Basically, the current rule is that tstclnt exits with an exit code of 254 if PR_Send failed, and 1 for any other error. This doesn't seem like a useful rule. Instead, a modification of the proposal by Glen Beasley in bug 402058 comment 9 [1] seems better: If the handshake failed because of a certificate validation failure, return 254; if it failed due to any other error, return 1. In the future, we could distinguish other types of errors (e.g. SSL version mismatch could be 253, TLS renegotiation support mismatch could be 252, etc.). This seems like it would be much more useful than distinguishing whether we failed during PR_Send or PR_Recv. Thoughts? Cheers, Brian [1] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c9 Additional background on error codes returned by tstclnt: [2] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c7 [3] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c5 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=86528#c9 [5] https://bugzilla.mozilla.org/show_bug.cgi?id=68869 -- dev-tech-crypto mailing list dev-tech-crypto@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-tech-crypto
Re: tstclnt exit code: do we really need to distinguish between failures during PR_Send and PR_Recv?
On 10/31/2011 01:45 AM, Brian Smith wrote: I am modifying tstclnt to test my patch for bug 542832. With my patch, tstclnt always succeeds or fails like it did before, but in some of the failure cases it exits with exit code 1 when the testcase expects it to exit with exit code 254, and sometimes it exits with exit code 254 when the testcase expects it to exit with exit code 1. Basically, the current rule is that tstclnt exits with an exit code of 254 if PR_Send failed, and 1 for any other error. This doesn't seem like a useful rule. Instead, a modification of the proposal by Glen Beasley in bug 402058 comment 9 [1] seems better: If the handshake failed because of a certificate validation failure, return 254; if it failed due to any other error, return 1. In the future, we could distinguish other types of errors (e.g. SSL version mismatch could be 253, TLS renegotiation support mismatch could be 252, etc.). This seems like it would be much more useful than distinguishing whether we failed during PR_Send or PR_Recv. Thoughts? Cheers, Brian I would look at the checks and see what kind of failures it is expecting. I think the main thing is to distinguish a normal connection failure with a client auth authorization failure. That being said, I suspect that we currently don't distinguish between these two, and wind up with a false success on expected client auth failure cases when the server is down. In practice that's not too much of a problem because several of the other tests in the suite will fail in that case. Upshot: creating a rationalized error system from tstclnt sounds fine, as long that the tests are adjusted accordingly. (longer term, we may want to set the error code based on the PORT_GetError() value). bob [1] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c9 Additional background on error codes returned by tstclnt: [2] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c7 [3] https://bugzilla.mozilla.org/show_bug.cgi?id=402058#c5 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=86528#c9 [5] https://bugzilla.mozilla.org/show_bug.cgi?id=68869 -- dev-tech-crypto mailing list dev-tech-crypto@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-tech-crypto
Re: tstclnt exit code: do we really need to distinguish between failures during PR_Send and PR_Recv?
Robert Relyea wrote: That being said, I suspect that we currently don't distinguish between these two, and wind up with a false success on expected client auth failure cases when the server is down. In practice that's not too much of a problem because several of the other tests in the suite will fail in that case. Upshot: creating a rationalized error system from tstclnt sounds fine, as long that the tests are adjusted accordingly. (longer term, we may want to set the error code based on the PORT_GetError() value). Bob, I think your suggestion makes a lot of sense. We can only return 255 distinct error codes from tstclnt but there are many more possible error codes from PORT_GetError(). How about this updated proposal: * Add an additional command-line flag -E error-id (e.g. -E SSL_ERROR_CERT_DOMAIN) to tstclnt. * If the flag is given and PR_Read() or PR_Write() fails with the given error, exit with code 2, which would mean failed with the expected error. * Otherwise, if error flag was given, and an error was encountered, exit with code 3, which would mean failed with the wrong error. * Otherwise, if the error flag was given, and no error was encountered, exit with code 4, which would mean succeeded but expected to fail. * Otherwise, return 1 on any error or 0 if there were no errors. We could then make an analogous change to selfserv (hopefully in another bug). Thoughts? Cheers, Brian -- dev-tech-crypto mailing list dev-tech-crypto@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-tech-crypto