Re: [openssl-project] Is making tests faster a bugfix?

2018-03-29 Thread Tim Hudson
Improved testing to me is something that is a good thing - and a value
judgement.
It doesn't change libcrypto or libssl - and that to me is the way I think
of it.
Fixing tests and apps and Makefiles to me are different from adding
features to libcrypto or libssl.

On this one - the fuzz testing has been sufficiently slow to reduce its
usefulness - and this is a step in the right direction.

It is however also a bit outside of our current policy on such things - so
perhaps we need to update that.

Tim.


On Thu, Mar 29, 2018 at 11:45 PM, Richard Levitte 
wrote:

> In message  on Thu, 29
> Mar 2018 14:03:06 +0100, Matt Caswell  said:
>
> matt>
> matt>
> matt> On 29/03/18 14:00, Salz, Rich wrote:
> matt> > Please see https://github.com/openssl/openssl/pull/5788
> matt> >
> matt> > I don’t think it is, but I’d like to know what others think.
> matt>
> matt> I do think this should be applied. The tests in question are not just
> matt> slow but *really* slow to the point that I often exit them before
> they
> matt> have completed. This removes the benefits of having the tests in the
> matt> first place. From that perspective I view this as a bug fix.
>
> Something to remember is that no user will ever complain about this,
> because we don't deliver the contents of fuzz/corpora in our tarballs.
>
> In other words, this is a developer only change of our current tests,
> and you will only hear from developers who do engage in fuzz testing,
> i.e. those who do these tests as part of a release, just to pick a
> very recent example.
>
> Also, you may note that this test re-engages fuzz testing as part of
> our normal tests that are run for every PR, which means that we will
> catch errors that the fuzzers can detect much earlier.  Because the
> fuzz testing took so long time, we had them only engaged with
> [extended tests], something that's almost never used.
>
> So I would argue that faster fuzz testing means more fuzz testing, and
> hopefully better testing of stuff that's harder to catch otherwise.
>
> Cheers,
> Richard ( plus, from a very personal point of view, it's *my* time,
>   and Matt's, and whoever else's who tests for releases, that
>   gets substantially less wasted! )
>
> --
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> ___
> openssl-project mailing list
> openssl-project@openssl.org
> https://mta.openssl.org/mailman/listinfo/openssl-project
>
___
openssl-project mailing list
openssl-project@openssl.org
https://mta.openssl.org/mailman/listinfo/openssl-project

Re: [openssl-project] Is making tests faster a bugfix?

2018-03-29 Thread Matt Caswell


On 29/03/18 14:00, Salz, Rich wrote:
> Please see https://github.com/openssl/openssl/pull/5788
> 
> I don’t think it is, but I’d like to know what others think.

I do think this should be applied. The tests in question are not just
slow but *really* slow to the point that I often exit them before they
have completed. This removes the benefits of having the tests in the
first place. From that perspective I view this as a bug fix.

Matt

___
openssl-project mailing list
openssl-project@openssl.org
https://mta.openssl.org/mailman/listinfo/openssl-project