Re: Errored: openssl/openssl#31939 (master - 34b1676)

2020-02-14 Thread Dr Paul Dale
An alternative would be to only run a cut down selection of tests with msan.


Pauli
-- 
Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
Phone +61 7 3031 7217
Oracle Australia




> On 14 Feb 2020, at 11:00 pm, Matt Caswell  wrote:
> 
> 
> 
> On 14/02/2020 12:23, Nicola Tuveri wrote:
>> If ASAN is too slow to run in the CI should we restore the previous
>> homemade checks for memory leaks as an alternative to be run in regular
>> CI runs and leave ASAN builds to run-checker on the master branch only? 
> 
> To be clear the build that is timing out uses *msan* not *asan*.
> 
> As I understand it msan detects unitialised reads. whereas asan detects
> memory corruption, buffer overflows, use-after-free bugs, and memory leaks.
> 
> The previous "home-made" checks only detected memory leaks, so it is not
> comparable with the functionality offered by msan.
> 
> The msan documentation
> (https://urldefense.com/v3/__https://clang.llvm.org/docs/MemorySanitizer.html__;!!GqivPVa7Brio!JwD52xjNRP5yVXTD3K12Mn17HiC2xHM_O6YzFE7G32G1BYh-NID9mIM6xiEvnC0$
>  ) suggests that a slow
> down of 3x is typical.
> 
> It seems reasonable to me to disable msan checks in Travis entirely, and
> have them only in run-checker.
> 
>> 
>> Here is another idea that would be interesting if we restore the
>> previous checks:
>> I don't know what kind of options github offers on this, but would it be
>> possible to run triggered CI on something that is not Travis and does
>> not timeout and still have the results in the PR?
> 
> I am sure there are hooks to do this. Richard has been talking for quite
> a while about setting up a buildbot infrastructure. If that could be
> integrated into github that would be really neat.
> 
> Matt
> 
> 
>> If something like that would be possible we could move the ASAN builds
>> to extended_tests, rely on the previous memleak detection for the
>> regular CI runs, and then trigger with a script or Github Action the
>> extended_tests when the approval:done label is added. 
>> 
>> That way, by the time something is ready to be merged we should have a
>> full picture! 
>> 
>> 
>> Nicola
>> 
>> On Wed, Feb 5, 2020, 10:25 Matt Caswell > > wrote:
>> 
>>Since we fixed the Travis builds 4 out of the 8 builds on master that
>>have taken place have errored due to a timeout.
>> 
>>The msan build is consistently taking a *very* long time to run. If it
>>gets to 50 minutes then Travis cuts it off and the build fails.
>> 
>>Should we disable the msan build?
>> 
>>Matt
>> 
>> 
>> Forwarded Message 
>>Subject:Errored: openssl/openssl#31939 (master - 34b1676)
>>Date:   Wed, 05 Feb 2020 00:02:01 +
>>From:   Travis CI mailto:bui...@travis-ci.org>>
>>To: openssl-comm...@openssl.org 
>> 
>> 
>> 
>>openssl
>> 
>>/
>> 
>>openssl
>> 
>>
>> >  >
>> 
>> 
>>branch iconmaster 
>> >  >
>> 
>>build has errored
>>Build #31939 has errored
>>
>> >  >
>>arrow to build time
>>clock icon50 mins and 3 secs
>> 
>>Pauli avatarPauli
>> 
>>34b1676 CHANGESET →
>>
>> >  >
>> 
>>Make minimum size for secure memory a size_t.
>> 
>>The minimum size argument to CRYPTO_secure_malloc_init() was an int but
>>ought
>>to be a size_t since it is a size.
>> 
>>From an API perspective, this is a change. However, the minimum size is
>>verified as being a positive power of two and it will typically be a
>>small
>>constant.
>> 
>>Reviewed-by: David von Oheimb >>
>>(Merged from #11003)
>> 
>>Want to know about upcoming build environment updates?
>> 
>>Would you like to stay up-to-date with the upcoming Travis CI build
>>environment updates? We set up a mailing list for you!
>> 
>>SIGN UP HERE 
>> >  >
>> 
>>book icon
>> 
>>Documentation 
>> >  > about Travis CI
>> 
>>

Re: Deprecation

2020-02-14 Thread Benjamin Kaduk
On Fri, Feb 14, 2020 at 05:59:32PM +0100, Tomas Mraz wrote:
> On Fri, 2020-02-14 at 16:17 +, Salz, Rich wrote:
> > I think we should delay the deprecation of engine stuff to 4.0.
> > Personally I don't have sense of stability of provider API.
> >  
> > I share this concern.  This is the first release of the provider
> > infrastructure, and while it will be known to work for FIPS modules,
> > it’s hubris to think OpenSSL will get it completely right for other
> > uses.
> >  
> > All other deprecations point to existing API’s or, if they are brand
> > new, are not a lot of code/work to implement them.  The provider
> > interface is both brand new and significant work.  Deprecating and
> > saying “use providers” without at least one release cycle of 
> > providers, seems premature.
> 
> This is an argument for not removing engines in 4.0 (or earlier than
> one major release after the provider interface fully stabilizes and
> proves viable). However I do not think that it is argument for not
> deprecating it. Deprecation is declaration of intent and not a way to
> force people stop using the API immediately.
> 
> How is the provider API going to improve/stabilize, if the 3rd party
> engine suppliers will not get the the message that the engine API is
> eventually going away in future and start writing providers as
> replacement.

I have similar feelings to Tomas.

Note that the current policy's test for "is removal allowed" is a
two-pronged test, and if the next release after 3.0 is 4.0, we would not be
allowed to remove engines in 4.0, since providers (the "engine
replacement") will not have been available in a (previous) LTS release.

If we know that something will need to go away eventually, it seems like we
can use deprecation annotations to publicize that fact, even if the
eventual removal will be delayed for quite some time due to other
considerations.

-Ben


Re: Deprecation

2020-02-14 Thread Tomas Mraz
On Fri, 2020-02-14 at 16:17 +, Salz, Rich wrote:
> I think we should delay the deprecation of engine stuff to 4.0.
> Personally I don't have sense of stability of provider API.
>  
> I share this concern.  This is the first release of the provider
> infrastructure, and while it will be known to work for FIPS modules,
> it’s hubris to think OpenSSL will get it completely right for other
> uses.
>  
> All other deprecations point to existing API’s or, if they are brand
> new, are not a lot of code/work to implement them.  The provider
> interface is both brand new and significant work.  Deprecating and
> saying “use providers” without at least one release cycle of 
> providers, seems premature.

This is an argument for not removing engines in 4.0 (or earlier than
one major release after the provider interface fully stabilizes and
proves viable). However I do not think that it is argument for not
deprecating it. Deprecation is declaration of intent and not a way to
force people stop using the API immediately.

How is the provider API going to improve/stabilize, if the 3rd party
engine suppliers will not get the the message that the engine API is
eventually going away in future and start writing providers as
replacement.

-- 
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
  Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]




Re: Deprecation

2020-02-14 Thread Salz, Rich
  *   I think we should delay the deprecation of engine stuff to 4.0. 
Personally I don't have sense of stability of provider API.

I share this concern.  This is the first release of the provider 
infrastructure, and while it will be known to work for FIPS modules, it’s 
hubris to think OpenSSL will get it completely right for other uses.

All other deprecations point to existing API’s or, if they are brand new, are 
not a lot of code/work to implement them.  The provider interface is both brand 
new and significant work.  Deprecating and saying “use providers” without at 
least one release cycle of providers, seems premature.


Re: Errored: openssl/openssl#31939 (master - 34b1676)

2020-02-14 Thread Nicola Tuveri
On Fri, 14 Feb 2020 at 14:00, Matt Caswell  wrote:

>
> To be clear the build that is timing out uses *msan* not *asan*.


>
As I understand it msan detects unitialised reads. whereas asan detects
> memory corruption, buffer overflows, use-after-free bugs, and memory leaks.
>
> The previous "home-made" checks only detected memory leaks, so it is not
> comparable with the functionality offered by msan.
>
>
Thanks for the clarification! I was indeed confused!



> The msan documentation
> (https://clang.llvm.org/docs/MemorySanitizer.html) suggests that a slow
> down of 3x is typical.
>
> It seems reasonable to me to disable msan checks in Travis entirely, and
> have them only in run-checker.
>
>
I agree with you.


> > Here is another idea that would be interesting if we restore the
> > previous checks:
> > I don't know what kind of options github offers on this, but would it be
> > possible to run triggered CI on something that is not Travis and does
> > not timeout and still have the results in the PR?
>
> I am sure there are hooks to do this. Richard has been talking for quite
> a while about setting up a buildbot infrastructure. If that could be
> integrated into github that would be really neat.
>
>
It would be neat indeed, when I first heard about buildbot I tried to set
aside some time to play with it, but failed in finding the time needed!
But at least from what I read it does indeed seem like a very interesting
and useful tool for our purposes!

I have no doubt sooner or later Richard will be more successful than I was
at finding the time to work on this item as well!

Nicola


Re: Errored: openssl/openssl#31939 (master - 34b1676)

2020-02-14 Thread Matt Caswell



On 14/02/2020 12:23, Nicola Tuveri wrote:
> If ASAN is too slow to run in the CI should we restore the previous
> homemade checks for memory leaks as an alternative to be run in regular
> CI runs and leave ASAN builds to run-checker on the master branch only? 

To be clear the build that is timing out uses *msan* not *asan*.

As I understand it msan detects unitialised reads. whereas asan detects
memory corruption, buffer overflows, use-after-free bugs, and memory leaks.

The previous "home-made" checks only detected memory leaks, so it is not
comparable with the functionality offered by msan.

The msan documentation
(https://clang.llvm.org/docs/MemorySanitizer.html) suggests that a slow
down of 3x is typical.

It seems reasonable to me to disable msan checks in Travis entirely, and
have them only in run-checker.

> 
> Here is another idea that would be interesting if we restore the
> previous checks:
> I don't know what kind of options github offers on this, but would it be
> possible to run triggered CI on something that is not Travis and does
> not timeout and still have the results in the PR?

I am sure there are hooks to do this. Richard has been talking for quite
a while about setting up a buildbot infrastructure. If that could be
integrated into github that would be really neat.

Matt


> If something like that would be possible we could move the ASAN builds
> to extended_tests, rely on the previous memleak detection for the
> regular CI runs, and then trigger with a script or Github Action the
> extended_tests when the approval:done label is added. 
> 
> That way, by the time something is ready to be merged we should have a
> full picture! 
> 
> 
> Nicola
> 
> On Wed, Feb 5, 2020, 10:25 Matt Caswell  > wrote:
> 
> Since we fixed the Travis builds 4 out of the 8 builds on master that
> have taken place have errored due to a timeout.
> 
> The msan build is consistently taking a *very* long time to run. If it
> gets to 50 minutes then Travis cuts it off and the build fails.
> 
> Should we disable the msan build?
> 
> Matt
> 
> 
>  Forwarded Message 
> Subject:        Errored: openssl/openssl#31939 (master - 34b1676)
> Date:   Wed, 05 Feb 2020 00:02:01 +
> From:   Travis CI mailto:bui...@travis-ci.org>>
> To:     openssl-comm...@openssl.org 
> 
> 
> 
> openssl
> 
> /
> 
> openssl
> 
> 
> 
> 
> 
> branch iconmaster 
> 
> build has errored
> Build #31939 has errored
> 
> 
> arrow to build time
> clock icon50 mins and 3 secs
> 
> Pauli avatarPauli
> 
> 34b1676 CHANGESET →
> 
> 
> Make minimum size for secure memory a size_t.
> 
> The minimum size argument to CRYPTO_secure_malloc_init() was an int but
> ought
> to be a size_t since it is a size.
> 
> From an API perspective, this is a change. However, the minimum size is
> verified as being a positive power of two and it will typically be a
> small
> constant.
> 
> Reviewed-by: David von Oheimb  >
> (Merged from #11003)
> 
> Want to know about upcoming build environment updates?
> 
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you!
> 
> SIGN UP HERE 
> 
> book icon
> 
> Documentation  about Travis CI
> 
> Have any questions? We're here to help.
> >
> Unsubscribe
> 
> 
> from build emails from the openssl/openssl repository.
> To unsubscribe from *all* build emails, please update your settings
> 
> .
> 
> black and white travis ci logo 
> 
> Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
> Jacops | Contact: cont...@travis-ci.com
>   > |
> Amtsgericht Charlottenburg, Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß
> §27 a Umsatzsteuergesetz: DE282002648
> 


Re: Errored: openssl/openssl#31939 (master - 34b1676)

2020-02-14 Thread Nicola Tuveri
If ASAN is too slow to run in the CI should we restore the previous
homemade checks for memory leaks as an alternative to be run in regular CI
runs and leave ASAN builds to run-checker on the master branch only?

Here is another idea that would be interesting if we restore the previous
checks:
I don't know what kind of options github offers on this, but would it be
possible to run triggered CI on something that is not Travis and does not
timeout and still have the results in the PR?
If something like that would be possible we could move the ASAN builds to
extended_tests, rely on the previous memleak detection for the regular CI
runs, and then trigger with a script or Github Action the extended_tests
when the approval:done label is added.

That way, by the time something is ready to be merged we should have a full
picture!


Nicola

On Wed, Feb 5, 2020, 10:25 Matt Caswell  wrote:

> Since we fixed the Travis builds 4 out of the 8 builds on master that
> have taken place have errored due to a timeout.
>
> The msan build is consistently taking a *very* long time to run. If it
> gets to 50 minutes then Travis cuts it off and the build fails.
>
> Should we disable the msan build?
>
> Matt
>
>
>  Forwarded Message 
> Subject:Errored: openssl/openssl#31939 (master - 34b1676)
> Date:   Wed, 05 Feb 2020 00:02:01 +
> From:   Travis CI 
> To: openssl-comm...@openssl.org
>
>
>
> openssl
>
> /
>
> openssl
>
> <
> https://travis-ci.org/openssl/openssl?utm_medium=notification_source=email
> >
>
>
> branch iconmaster 
>
> build has errored
> Build #31939 has errored
> <
> https://travis-ci.org/openssl/openssl/builds/646181069?utm_medium=notification_source=email
> >
> arrow to build time
> clock icon50 mins and 3 secs
>
> Pauli avatarPauli
>
> 34b1676 CHANGESET →
> 
>
> Make minimum size for secure memory a size_t.
>
> The minimum size argument to CRYPTO_secure_malloc_init() was an int but
> ought
> to be a size_t since it is a size.
>
> From an API perspective, this is a change. However, the minimum size is
> verified as being a positive power of two and it will typically be a small
> constant.
>
> Reviewed-by: David von Oheimb 
> (Merged from #11003)
>
> Want to know about upcoming build environment updates?
>
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you!
>
> SIGN UP HERE 
>
> book icon
>
> Documentation  about Travis CI
>
> Have any questions? We're here to help. 
> Unsubscribe
> <
> https://travis-ci.org/account/preferences/unsubscribe?repository=5849220_medium=notification_source=email
> >
> from build emails from the openssl/openssl repository.
> To unsubscribe from *all* build emails, please update your settings
> <
> https://travis-ci.org/account/preferences/unsubscribe?utm_medium=notification_source=email
> >.
>
> black and white travis ci logo 
>
> Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
> Jacops | Contact: cont...@travis-ci.com  |
> Amtsgericht Charlottenburg, Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß
> §27 a Umsatzsteuergesetz: DE282002648
>
>


Re: Deprecation

2020-02-14 Thread Richard Levitte
On Fri, 14 Feb 2020 12:17:30 +0100,
Dmitry Belyavsky wrote:
> 
> 
> Dear Richard,
> 
> On Fri, Feb 14, 2020 at 1:37 PM Richard Levitte  wrote:
> 
> It should be pointed out that the engine stuff isn't as stable as you
> might think, because it shares internal structures with libcrypto.
> Even if we handle those structures via function calls, all it takes is
> loading an engine linked against - say - 1.1.0 in 1.1.1 to run a real
> risk of a spectacular KABOOM if any of the structure that are touched
> changed between those OpenSSL versions.
> 
> Does ABI compatibility cover a significant share of these cases?

Yes.  Any structure change between those OpenSSL version potentially
means an ABI incompatibility between an engine and the libcrypto that
loads it.

This is precisely the reason why we have forbidden shared opaque
structures over the libcrypto <-> provider boundary.

...

> > But some features I'm interested in imply engine model (and it will
> > be great if somebody else could look at PR 10904 to avoid it when
> > possible).
>
> Apart from a few details, that PR looks rather innocuous.  I frankly
> haven't had time to look at it too deeply (I just discovered that I
> was prompted), as I haven't dived in the CMS code yet...
> 
> Well, the problem is introducing some new control values. I don't
> feel policy here very well. I also plan to submit at least one
> TLS-related PR significantly more innocent...

We haven't formed a clear policy on those, it's a bit of play as we go
and do what makes most sense.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecation

2020-02-14 Thread Dmitry Belyavsky
Dear Richard,

On Fri, Feb 14, 2020 at 1:37 PM Richard Levitte  wrote:

> On Fri, 14 Feb 2020 10:41:05 +0100,
> Dmitry Belyavsky wrote:
> >
> >
> > Hello,
> >
> > On Fri, Feb 14, 2020 at 5:30 AM Dr Paul Dale 
> wrote:
> >
> > There is some pushback against the deprecations going on in various
> PRs.
> >
> > The plan has always been to deprecate engines in 3.0 and removing
> support for them 5+ years later.
> > Originally, the path was to have included an engine provider that
> could load engines and make them appear
> > to be a provider.  After a fair amount of investigation, this was
> deemed to be too difficult in the 3.0
> > time frame.
> >
> > Do we still want to deprecate engines in 3.0?
> > Should we defer until 4.0 instead?
> >
> > I think we should delay the deprecation of engine stuff to 4.0.
> Personally I don't have sense of stability of
> > provider API.
>
> It should be pointed out that the engine stuff isn't as stable as you
> might think, because it shares internal structures with libcrypto.
> Even if we handle those structures via function calls, all it takes is
> loading an engine linked against - say - 1.1.0 in 1.1.1 to run a real
> risk of a spectacular KABOOM if any of the structure that are touched
> changed between those OpenSSL versions.
>

Does ABI compatibility cover a significant share of these cases?


> This is a consequence of making structures opaque and feeling much
> more liberty in changing them, and we didn't quite pay attention.
>
> The whole model around providers is intentionally designed to allow
> providers to run flexibly with any OpenSSL version, even if they are
> linked with some (possibly different) libcrypto version.
>
> So the question of stability depends on what you look at.
>

Agreed.

It's true that some parts of the provider API is fluctuating a bit, as
> early designs need modifications to better meet demands.  We're still
> in development.  Come beta1 (schedule for June), I expect that it will
> have stabilized.  Come the release, it must have.
>

Sure.

> The main benefits seem to boil down to continuing to support existing
> engines vs removing the legacy code
> > paths and switching to the provider model.
> >
> > For me, both as open-source and commercial engine developer seems
> > reasonable to delay conversion from engines to providers at least
> > until 3.0.0 feature freeze happens.
>
> According to our policy, a deprecation starts at the release that has
> those deprecations, and removal of deprecated stuff can only happen 5
> years later at the earliest.  Seeing deprecations in the source now
> doesn't change that, the clock will start ticking when we release
> 3.0, so consider what you see happening on github as a pre-deprecation
> warning.
>

Yes.


> > But some features I'm interested in imply engine model (and it will
> > be great if somebody else could look at PR 10904 to avoid it when
> > possible).
>
> Apart from a few details, that PR looks rather innocuous.  I frankly
> haven't had time to look at it too deeply (I just discovered that I
> was prompted), as I haven't dived in the CMS code yet...
>

Well, the problem is introducing some new control values. I don't feel
policy here very well.
I also plan to submit at least one TLS-related PR significantly more
innocent...

-- 
SY, Dmitry Belyavsky


Re: Deprecation

2020-02-14 Thread Dmitry Belyavsky
Dear Matt,

On Fri, Feb 14, 2020 at 12:48 PM Matt Caswell  wrote:

>
>
> On 14/02/2020 09:41, Dmitry Belyavsky wrote:
> > Hello,
> >
> > On Fri, Feb 14, 2020 at 5:30 AM Dr Paul Dale  > > wrote:
> >
> > There is some pushback against the deprecations going on in various
> PRs.
> >
> > The plan has always been to deprecate engines in 3.0 and removing
> > support for them 5+ years later.  Originally, the path was to have
> > included an engine provider that could load engines and make them
> > appear to be a provider.  After a fair amount of investigation, this
> > was deemed to be too difficult in the 3.0 time frame.
> >
> > Do we still want to deprecate engines in 3.0?
> > Should we defer until 4.0 instead?
> >
> >
> > I think we should delay the deprecation of engine stuff to 4.0.
> > Personally I don't have sense of stability of provider API.
>
> Well - it isn't stable *right now*. Its in development. But moving
> forward the provider model *is* the way we intend to go. By the time of
> the 3.0 release the provider stuff had better be stable, otherwise we've
> gone wrong.
>

Totally agree. Though the conversion between engines and providers is not
as straightforward as I want, so I prefer to wait for some time.

>
> As has been pointed out many times. Deprecation does not mean removal
> (yet). Its just a signal that at some later time we will remove it.
>

Sure.

>
> And the 3.0 is the right time to signal that. We don't want to hold on
> the "legacy" codepaths for any longer than we have to. They were only
> ever originally intended to be temporary, and to be removed before 3.0
> got released. However it now looks like they will live beyond the 3.0
> release. They should not live for any longer than absolutely necessary.
>

Hmmm. Not sure here. I remember the introduction of EVP_PKEY_ameth/pmeth
currently moving to providers.

> The main benefits seem to boil down to continuing to support
> > existing engines vs removing the legacy code paths and switching to
> > the provider model.
>

It's worth explaining the benefits to engine authors :)
I understand the benefits of OpenSSL as a product and still now don't have
a firm understanding of benefits for the surrounding ecosystem. Though I
did not dive into the providers deeply.

I still have a suspicion that we will not have function signatures for
callbacks in providers.
Do we? If don't, we'll have a set of errors implementing it.

> For me, both as open-source and commercial engine developer seems
> > reasonable to delay conversion from engines to providers at least until
> > 3.0.0 feature freeze happens.
>
> That's a perfectly reasonable strategy. But this decision is independent
> of whether we deprecate the ENGINE APIs in 3.0 or not. It would be
> perfectly ok for people to continue to *use* engines in 3.0 even though
> they are deprecated.
>

Sure.


> > But some features I'm interested in imply engine model (and it will be
> > great if somebody else could look at PR 10904 to avoid it when possible).
>
> If there are gaps then we need to plug them. The provider model should
> be able to do everything that you can do now in ENGINEs.
>

Totally agree.

-- 
SY, Dmitry Belyavsky


Re: Deprecation

2020-02-14 Thread Matt Caswell



On 14/02/2020 10:46, Dr Paul Dale wrote:
> Roumen in
> https://github.com/openssl/openssl/pull/10977#issuecomment-584818517
> Dmitry
> in https://github.com/openssl/openssl/pull/11082#issuecomment-585603911

Thanks.

Having re-read the comments and the thread here, I am still of the
opinion that we should press ahead with the deprecations as planned.

Matt

> And a further one via private email.
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> 
> 
> 
>> On 14 Feb 2020, at 7:37 pm, Matt Caswell > > wrote:
>>
>>
>>
>> On 14/02/2020 02:30, Dr Paul Dale wrote:
>>> There is some pushback against the deprecations going on in various PRs.
>>
>> I've not followed all of the PRs. Can you point at some specific
>> comments you've received pushing back on this?
>>
>> Matt
>>
>>
>>>
>>> The plan has always been to deprecate engines in 3.0 and removing
>>> support for them 5+ years later.  Originally, the path was to have
>>> included an engine provider that could load engines and make them appear
>>> to be a provider.  After a fair amount of investigation, this was deemed
>>> to be too difficult in the 3.0 time frame.
>>>
>>> Do we still want to deprecate engines in 3.0?
>>> Should we defer until 4.0 instead?
>>>
>>>
>>> The main benefits seem to boil down to continuing to support existing
>>> engines vs removing the legacy code paths and switching to the provider
>>> model.
>>>
>>>
>>> Pauli
>>> -- 
>>> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
>>> Phone +61 7 3031 7217
>>> Oracle Australia
>>>
> 


Re: Deprecation

2020-02-14 Thread Dr Paul Dale
Roumen in https://github.com/openssl/openssl/pull/10977#issuecomment-584818517 

Dmitry in https://github.com/openssl/openssl/pull/11082#issuecomment-585603911 

And a further one via private email.

Pauli
-- 
Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
Phone +61 7 3031 7217
Oracle Australia




> On 14 Feb 2020, at 7:37 pm, Matt Caswell  wrote:
> 
> 
> 
> On 14/02/2020 02:30, Dr Paul Dale wrote:
>> There is some pushback against the deprecations going on in various PRs.
> 
> I've not followed all of the PRs. Can you point at some specific
> comments you've received pushing back on this?
> 
> Matt
> 
> 
>> 
>> The plan has always been to deprecate engines in 3.0 and removing
>> support for them 5+ years later.  Originally, the path was to have
>> included an engine provider that could load engines and make them appear
>> to be a provider.  After a fair amount of investigation, this was deemed
>> to be too difficult in the 3.0 time frame.
>> 
>> Do we still want to deprecate engines in 3.0?
>> Should we defer until 4.0 instead?
>> 
>> 
>> The main benefits seem to boil down to continuing to support existing
>> engines vs removing the legacy code paths and switching to the provider
>> model.
>> 
>> 
>> Pauli
>> -- 
>> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
>> Phone +61 7 3031 7217
>> Oracle Australia
>> 



Re: Deprecation

2020-02-14 Thread Richard Levitte
On Fri, 14 Feb 2020 10:41:05 +0100,
Dmitry Belyavsky wrote:
> 
> 
> Hello,
> 
> On Fri, Feb 14, 2020 at 5:30 AM Dr Paul Dale  wrote:
> 
> There is some pushback against the deprecations going on in various PRs.
>
> The plan has always been to deprecate engines in 3.0 and removing support 
> for them 5+ years later. 
> Originally, the path was to have included an engine provider that could 
> load engines and make them appear
> to be a provider.  After a fair amount of investigation, this was deemed 
> to be too difficult in the 3.0
> time frame.
>
> Do we still want to deprecate engines in 3.0?
> Should we defer until 4.0 instead?
> 
> I think we should delay the deprecation of engine stuff to 4.0. Personally I 
> don't have sense of stability of
> provider API.

It should be pointed out that the engine stuff isn't as stable as you
might think, because it shares internal structures with libcrypto.
Even if we handle those structures via function calls, all it takes is
loading an engine linked against - say - 1.1.0 in 1.1.1 to run a real
risk of a spectacular KABOOM if any of the structure that are touched
changed between those OpenSSL versions.
This is a consequence of making structures opaque and feeling much
more liberty in changing them, and we didn't quite pay attention.

The whole model around providers is intentionally designed to allow
providers to run flexibly with any OpenSSL version, even if they are
linked with some (possibly different) libcrypto version.

So the question of stability depends on what you look at.

It's true that some parts of the provider API is fluctuating a bit, as
early designs need modifications to better meet demands.  We're still
in development.  Come beta1 (schedule for June), I expect that it will
have stabilized.  Come the release, it must have.

> The main benefits seem to boil down to continuing to support existing 
> engines vs removing the legacy code
> paths and switching to the provider model.
> 
> For me, both as open-source and commercial engine developer seems
> reasonable to delay conversion from engines to providers at least
> until 3.0.0 feature freeze happens.

According to our policy, a deprecation starts at the release that has
those deprecations, and removal of deprecated stuff can only happen 5
years later at the earliest.  Seeing deprecations in the source now
doesn't change that, the clock will start ticking when we release
3.0, so consider what you see happening on github as a pre-deprecation
warning.

> But some features I'm interested in imply engine model (and it will
> be great if somebody else could look at PR 10904 to avoid it when
> possible).

Apart from a few details, that PR looks rather innocuous.  I frankly
haven't had time to look at it too deeply (I just discovered that I
was prompted), as I haven't dived in the CMS code yet...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecation

2020-02-14 Thread Dmitry Belyavsky
Hello,

On Fri, Feb 14, 2020 at 5:30 AM Dr Paul Dale  wrote:

> There is some pushback against the deprecations going on in various PRs.
>
> The plan has always been to deprecate engines in 3.0 and removing support
> for them 5+ years later.  Originally, the path was to have included an
> engine provider that could load engines and make them appear to be a
> provider.  After a fair amount of investigation, this was deemed to be too
> difficult in the 3.0 time frame.
>
> Do we still want to deprecate engines in 3.0?
> Should we defer until 4.0 instead?
>

I think we should delay the deprecation of engine stuff to 4.0. Personally
I don't have sense of stability of provider API.

The main benefits seem to boil down to continuing to support existing
> engines vs removing the legacy code paths and switching to the provider
> model.
>

For me, both as open-source and commercial engine developer seems
reasonable to delay conversion from engines to providers at least until
3.0.0 feature freeze happens.
But some features I'm interested in imply engine model (and it will be
great if somebody else could look at PR 10904 to avoid it when possible).


-- 
SY, Dmitry Belyavsky


Re: Deprecation

2020-02-14 Thread Matt Caswell



On 14/02/2020 02:30, Dr Paul Dale wrote:
> There is some pushback against the deprecations going on in various PRs.

I've not followed all of the PRs. Can you point at some specific
comments you've received pushing back on this?

Matt


> 
> The plan has always been to deprecate engines in 3.0 and removing
> support for them 5+ years later.  Originally, the path was to have
> included an engine provider that could load engines and make them appear
> to be a provider.  After a fair amount of investigation, this was deemed
> to be too difficult in the 3.0 time frame.
> 
> Do we still want to deprecate engines in 3.0?
> Should we defer until 4.0 instead?
> 
> 
> The main benefits seem to boil down to continuing to support existing
> engines vs removing the legacy code paths and switching to the provider
> model.
> 
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 


Re: Deprecation

2020-02-14 Thread Dmitry Belyavsky
Hello,

I think OPENSSL_NO_DEPRECATED_1_0_2 should be added to this list too.


On Fri, Feb 14, 2020 at 12:31 PM Matthias St. Pierre <
matthias.st.pie...@ncp-e.com> wrote:

>
>
> On 14.02.20 10:21, Matthias St. Pierre wrote:
> >
> > Because deprecation without removal is futile, it increases complexity
> of the code
> > instead of reducing it.
> >
> > Matthias
> >
>
> To underlinie my last statement, I added the initial release dates:
>
>  OPENSSL_NO_DEPRECATED_0_9_8  # 05 Jul 2005
>  OPENSSL_NO_DEPRECATED_1_0_0  # 29 Mar 2010
>  OPENSSL_NO_DEPRECATED_1_0_1  # 14 Mar 2012
>


-- 
SY, Dmitry Belyavsky


Re: Deprecation

2020-02-14 Thread Matthias St. Pierre




On 14.02.20 10:21, Matthias St. Pierre wrote:


Because deprecation without removal is futile, it increases complexity of the 
code
instead of reducing it.

Matthias



To underlinie my last statement, I added the initial release dates:

    OPENSSL_NO_DEPRECATED_0_9_8  # 05 Jul 2005
    OPENSSL_NO_DEPRECATED_1_0_0  # 29 Mar 2010
    OPENSSL_NO_DEPRECATED_1_0_1  # 14 Mar 2012


Re: Deprecation

2020-02-14 Thread Matthias St. Pierre




On 14.02.20 08:24, Tomas Mraz wrote:


I do not understand the pushback too much - Perhaps it could be
resolved by proper explanation that deprecation does not mean a
removal?

Also even if some stuff deprecated in 3.0 is removed in 4.0, it does
not necessarily mean that engines must be removed in the same release.
They can continue to be supported (just deprecated) until 5.0 for
example.

I think that doing the deprecation as early as possible is better - it
properly gives the signal that the engine interface is legacy and it
will disappear at some point. It provides more time for 3rd party
engines to transform into providers.



I agree with Tomas. In addition, I think it is high time to publish a definite 
timescale
for actually *removing* the older deprecated APIs, at least for the first three 
entries
in the list below:

    OPENSSL_NO_DEPRECATED_0_9_8  #
    OPENSSL_NO_DEPRECATED_1_0_0  #
    OPENSSL_NO_DEPRECATED_1_0_1  #

    OPENSSL_NO_DEPRECATED_1_0_2
    OPENSSL_NO_DEPRECATED_1_1_0
    OPENSSL_NO_DEPRECATED_1_1_1
    OPENSSL_NO_DEPRECATED_3_0

Because deprecation without removal is futile, it increases complexity of the 
code
instead of reducing it.

Matthias