Re: Improving X.509 certificate validation errors

2020-03-26 Thread Martin Ukrop
Hi Ben,

Yes, a reply after a few weeks is still very useful, thanks!

You are right about the point that every library has an "expired" code,
though I start to see other differences. The number of errors itself wildly
differ – OpenSSL has over 75 of certificate-related errors, while GnuTLS
has only about 18 and Botan about 40.
Hearing developer opinions helps me do it in a way useful for you. I
understand big, replied-on, open-source projects are sensitive about
changes. Though I believe at least updating the documentation should be
harmless (the details on the error message in OpenSSL docs sometimes just
restate the message). I'll open a separate issue/WiP pull request when we
have specific propositions.

PS: @Kurt, thanks for the pointer to Frankencert, it did not cross my radar


On Thu, 26 Mar 2020 at 10:28, Kurt Roeckx  wrote:

> On Wed, Mar 25, 2020 at 10:21:36PM -0700, Benjamin Kaduk wrote:
> > I tihnk it's an interesting idea.  To me, perhaps the most valuable part
> > would be to accumulate a corpus of certificates/chains that are malformed
> > or fail to validate due to a wide variety of errors, almost akin to a
> > fuzzing corpus.  I'd also be curious (though I'm not entirely sure how
> > large a practical impact it would have) to perform a clustering analysis
> > across different X.509 implementations and see if different
> implementations
> > produce different distributions of errors.  (That is, we might expect
> each
> > implementation to have an error for "not valid yet", "expired", "missing
> > required ASN.1 field", etc.; each implementation will have a different
> > error string, of course, but if we group all certificates that produce
> the
> > same error with the same implementation together, we have a bunch of
> > different clusters.  Repeating the clustering across all implementations
> > lets us compare the different distributions, and examine certificates
> that
> > end up in a different cluster in different implementations.)
> That's what frankencert did.
> Kurt

Improving X.509 certificate validation errors

2020-03-05 Thread Martin Ukrop

I’m the lead of a university project investigating (and improving) the
usability of certificate validation errors. Our goal is to simplify the
ecosystem by consolidating the errors and their documentation in one place,
providing replicable example certificates for all validation errors and by
explaining better what the individual errors mean. The project is live at

Now we are reaching out to library developers and users (you) to ask for

Currently, we base the system on OpenSSL errors (as it’s the most common).
We have example certificates for 30+ OpenSSL errors and in-progress mapping
for corresponding errors error for OpenSSL, GnuTLS, Botan and MbedTLS.
In the future, we plan the possibility of web reorganization based on the
other libraries (currently, the web is organized by OpenSSL), adding the
error frequencies based on IP-wide scans and elaborating on the
consequences of individual errors.
Ultimately, we want to propose better (ideally user-tested) errors and
their documentation. (Just recently, we made a survey among 180 developers
regarding their error documentation preference with good reception).

As developers/users of TLS libraries, what do you think of the idea?
* Which part(s) do you find the most/least useful?
* Is there anything you see missing?
* What are your thoughts on unifying the error taxonomy? (a very long-term
goal, if at all possible)

During spring, we would like to start creating pull requests improving the
documentation and error messages in some of the libraries. Would you
welcome such contributions?

For transparency: My PhD is done at Masaryk University (Czech Republic) and
I’m partially supported by Red Hat Czech.

With regards,