Re: Status of the remaining beta1 PRs

2020-09-18 Thread Matt Caswell



On 18/09/2020 16:59, Tomas Mraz wrote:
> On Fri, 2020-09-18 at 16:24 +0100, Matt Caswell wrote:
>>
>> 1 PR which is in a state of "its unclear what we do with this":
>> [WIP] Rename some XXX_ex() related methods to XXX_with_libctx()
>> https://github.com/openssl/openssl/pull/12701
>> With no agreement on a naming convention its unclear if this should
>> go
>> ahead or not
> 
> We should do something consistent - either rename the _ex functions
> that just add libctx (and property query) or rename the existing
> with_libctx functions to _ex functions.
> 
> The current state is a mess having some functions with _ex and some
> with _with_libctx.
> 

The current vote on the policy is still ongoing (the subject of this PR
is the one covered by chapter 6.2 in the proposed coding style).
Unfortunately there is currently no clear outcome on that vote and there
are still a significant number of outstanding votes.

Without a clear decision one way or another on that vote we cannot
decide a way forward. If the vote passes, then it is clear what we have
to do. If the vote fails then we will probably have to make a one off
vote about what to do about existing _with_libctx vs _ex discrepancies
without making a policy out of it.

Please can I encourage OTC members to vote one way or the other so that
we can move forward.

Matt


Status of the remaining beta1 PRs

2020-09-18 Thread Matt Caswell
As of right now we have 13 PRs with the beta1 milestone against them.

Of these there are 4 which really need our focused attention. These are

2 PRs which are in a state of "written but still in review":
WIP: Implement Provider side SM2 Asymmetric Cipher support
https://github.com/openssl/openssl/pull/12913
Needed to complete the SM2 support (WIP because dependent on 12536,
which is in "approval done" status but not yet merged)

ENCODER: Refactor the OSSL_ENCODER API to be more like OSSL_DECODER
https://github.com/openssl/openssl/pull/12873
Still in review

1 PR which is in a state of "we really need to do something about this":
[WIP, Parked] EVP: Adapt EVP_PKEY_set_alias_type() for provider-native
EVP_PKEYs
https://github.com/openssl/openssl/pull/12675
Since this affects a public API we probably really do need to figure out
what to do with the EVP_PKEY_set_alias_type()

1 PR which is in a state of "its unclear what we do with this":
[WIP] Rename some XXX_ex() related methods to XXX_with_libctx()
https://github.com/openssl/openssl/pull/12701
With no agreement on a naming convention its unclear if this should go
ahead or not


The remaining 9 PRs can be split into 2 groups:

6 are in the "approval done" state, so can be pushed soon (one of these
we are actually deliberately holding back until nearer the beta release).

3 PRs are in a state of "probably can be dropped altogether":
WIP: Fix the DRBG reseed propagation [master]
https://github.com/openssl/openssl/pull/12871
This is a bug fix so may not be needed for beta1 at all. Matthias says:
"I put the beta1 milestone against it because I wasn't sure whether I
need to change some public API or not. Currently, I don't think it's
necessary, but I need to check it once more. That will probably not
happen before the weekend, but I'll remember to move the ticket from the
beta1 milestone to the release milestone, if it's not urgent."

Reorder algorithm parameter to be first in *_FETCH() operations.
https://github.com/openssl/openssl/pull/12778
Disputed, and probably not needed. Probably can be dropped.

RAND: allow the primary, public and private of DRBGs to be overriden and
replaced.
https://github.com/openssl/openssl/pull/12754
A feature that current discussion on the PR seems to suggest we may drop
because it may not be needed for 3.0 after all.

Matt




Re: 3.0 beta 1 milestone

2020-09-18 Thread Matt Caswell



On 18/09/2020 08:26, Richard Levitte wrote:
> On Thu, 17 Sep 2020 15:57:52 +0200,
> Tomas Mraz wrote:
>> I do not think the milestone should include nice-to-have items.
> 
> Another view is that beta 1 is feature freeze.  If those nice to have
> items are characterized as new features, then it makes sense to have
> them included in the beta 1 milestone if we want them for 3.0.0.
> Otherwise, we will have to put them on the waiting list for after
> 3.0.0.

Not every feature that someone dreams up and creates a PR for should
mean we have to hold 3.0 beta 1 until it goes in. There are features
that are required, and features that are nice-to-have. If nice-to-have
features don't reach approval in time, then they should be dropped from
3.0, and wait for 3.1 instead.

Matt



> 
> If they are not characterized as new features, that is a different
> matter, of course.  The conclusion is that we will have to take some
> time (I assume that'll be quick in most cases) thinking how we
> characterize certain PRs before beta 1 is released.
> 
> Cheers,
> Richard
> 


3.0 beta 1 milestone

2020-09-17 Thread Matt Caswell
There's been quite a number of PRs added to the 3.0 beta 1 milestone.

Within the PRs there are a couple of bug fixes:

https://github.com/openssl/openssl/pull/12884
https://github.com/openssl/openssl/pull/12874

IMO these would be really nice to get into beta 1, but they should not
be considered blockers for it (i.e. if they didn't go in, it shouldn't
stop us from releasing beta 1).

There are also some nice-to-have items:

https://github.com/openssl/openssl/pull/12777
https://github.com/openssl/openssl/pull/12771
https://github.com/openssl/openssl/pull/12726
https://github.com/openssl/openssl/pull/12669
https://github.com/openssl/openssl/pull/12072

Again - these are nice-to-have, and if they happen to get merged in time
for beta 1 then great. Otherwise, they should wait for 3.1 (possibly
things which are just cleanup/minor refactoring could still be done
within the beta period). So, IMO, these should not be considered
blockers either.

So - this leads me to the question - what is the milestone for? Does it
means these things *must* go in before we can release beta 1? Or does it
mean we would *like* to get these things in for beta 1?

I actually don't mind either way - but if its the latter, then I need a
way of identifying the "must haves". These are the top priority items,
and at the moment I can't easily track their progress.

Matt




Re: Reordering new API's that have a libctx, propq

2020-09-16 Thread Matt Caswell



On 15/09/2020 12:25, Matt Caswell wrote:
> I plan to start two OTC votes on this tomorrow with the following wording:

These votes have now commenced. I'll report back with the results when
they are known.

Matt



> 
> "Adopt the coding style policy on function arguments as shown in chapter
> 6.1 of web PR 194 (commit 7b45b46d71f)"
> 
> "Adopt the coding style policy on extending existing functions as shown
> in chapter 6.2 of web PR 194 (commit 7b45b46d71f)"
> 
> In the event that one vote passes but the other vote does not I will
> remove the section of text that did not pass from the PR and adjust
> chapter numbers accordingly.
> 
> Matt
> 
>>
>> Since we're not yet fully in agreement some compromises will have to be
>> made. I hope I've come up with something which isn't too abhorrent to
>> anyone.
>>
>> Please take a look.
>>
>> Matt
>>
>>
>> On 05/09/2020 04:48, SHANE LONTIS wrote:
>>>
>>>   PR #12778 reorders all the API’s of the form:
>>>
>>>
>>>   EVP_XX_fetch(libctx, algname, propq) 
>>>
>>>
>>>   So that the algorithm name appears first.. 
>>>
>>>
>>>   e.g: EVP_MD_fetch(digestname, libctx, propq);
>>>
>>> This now logically reads as 'search for this algorithm using these
>>> parameters’.
>>>
>>> The libctx, propq should always appear together as a pair of parameters.
>>> There are only a few places where only the libctx is needed, which means
>>> that if there is no propq it is likely to cause a bug in a fetch at some
>>> point. 
>>>
>>> This keeps the API’s more consistent with other existing XXX_with_libctx
>>> API’s that put the libctx, propq at the end of the parameter list..
>>> The exception to this rule is that callback(s) and their arguments occur
>>> after the libctx, propq..
>>>
>>> e.g:
>>> typedef OSSL_STORE_LOADER_CTX *(*OSSL_STORE_open_with_libctx_fn)
>>>     (const OSSL_STORE_LOADER *loader,
>>>      const char *uri, OPENSSL_CTX *libctx, const char *propq,
>>>      const UI_METHOD *ui_method, void *ui_data);
>>>
>>> An otc_hold was put on this.. Do we need to have a formal vote?
>>> This really needs to be sorted out soon so the API’s can be locked down
>>> for beta.
>>>
>>> 
>>> Also discussed in a weekly meeting and numerous PR discussions was the
>>> removal of the long winded API’s ending with ‘with_libctx’
>>> e.g CMS_data_create_with_libctx
>>> The proposed renaming for this was to continue with the _ex notation..
>>> If there is an existing _ex name then it becomes _ex2, etc.
>>> There will most likely be additional parameters in the future for some
>>> API’s, so this notation would be more consistent with current API’s.
>>> Does this also need a vote?
>>>
>>> Regards,
>>> Shane
>>>
>>>


Re: stable branch release cadence

2020-09-15 Thread Matt Caswell



On 15/09/2020 23:10, Tim Hudson wrote:
> The OMC voted to:
> 
> /Release stable branch on the second last Tuesday of the last month in
> each quarter as a regular cadence./
> 
> The vote passed.
> For: 6, against: 9, abstained 0, not voted: 1

That should say against: 0

;-)

Matt

> 
> Thanks,
> Tim.
> 


Forthcoming OpenSSL Release

2020-09-15 Thread Matt Caswell
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


The OpenSSL project team would like to announce the forthcoming release
of OpenSSL version 1.1.1h.

This release will be made available on Tuesday 22nd September 2020
between 1300-1700 UTC.

OpenSSL 1.1.h is a bug-fix release. There are no CVEs addressed in this
release.

Yours

The OpenSSL Project Team
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAl9hObYACgkQ2cTSbQ5g
RJGmDAf+IPnGTpXB6XpHpuvlWWE6v0aTEOHntLgeYbqp9v3/5ay4i0qwFZk2M4Sn
9J5C/057OqqLVMq0UyXXAwhyS52KIR6VfcJKTCc/2NkgPHhee+/W5Q8SgGpXMnOP
60EIrHD5cfkestIO9fvrCHZ19RFFWlFQJnPmc64nLYyhQJ83a/AKGoug459oaxm7
lj90Rd+U4oQvEJyltsA5Urv/IAjQV24EYej1pCLb4zqerW4rLYnoATBrurclWVOa
5AXZgzuhNvtMV3/nVB7aFpfQIsg2FUaTnRW3ok+7e72oiXHndgxYW6TP0GxGOMdu
RDB1ZlWWwt7LzYz8BlWTex+s23SNZA==
=wNcz
-END PGP SIGNATURE-


Re: Reordering new API's that have a libctx, propq

2020-09-15 Thread Matt Caswell



On 14/09/2020 11:30, Matt Caswell wrote:
> In order to try and move this discussion forward I have made a concrete
> proposal for how we should formulate the various ideas in this thread
> into an actual style. Please see here:
> 
> https://github.com/openssl/web/pull/194

I've updated this PR with some of the feedback received so far. Please
take another look.

I plan to start two OTC votes on this tomorrow with the following wording:

"Adopt the coding style policy on function arguments as shown in chapter
6.1 of web PR 194 (commit 7b45b46d71f)"

"Adopt the coding style policy on extending existing functions as shown
in chapter 6.2 of web PR 194 (commit 7b45b46d71f)"

In the event that one vote passes but the other vote does not I will
remove the section of text that did not pass from the PR and adjust
chapter numbers accordingly.

Matt

> 
> Since we're not yet fully in agreement some compromises will have to be
> made. I hope I've come up with something which isn't too abhorrent to
> anyone.
> 
> Please take a look.
> 
> Matt
> 
> 
> On 05/09/2020 04:48, SHANE LONTIS wrote:
>>
>>   PR #12778 reorders all the API’s of the form:
>>
>>
>>   EVP_XX_fetch(libctx, algname, propq) 
>>
>>
>>   So that the algorithm name appears first.. 
>>
>>
>>   e.g: EVP_MD_fetch(digestname, libctx, propq);
>>
>> This now logically reads as 'search for this algorithm using these
>> parameters’.
>>
>> The libctx, propq should always appear together as a pair of parameters.
>> There are only a few places where only the libctx is needed, which means
>> that if there is no propq it is likely to cause a bug in a fetch at some
>> point. 
>>
>> This keeps the API’s more consistent with other existing XXX_with_libctx
>> API’s that put the libctx, propq at the end of the parameter list..
>> The exception to this rule is that callback(s) and their arguments occur
>> after the libctx, propq..
>>
>> e.g:
>> typedef OSSL_STORE_LOADER_CTX *(*OSSL_STORE_open_with_libctx_fn)
>>     (const OSSL_STORE_LOADER *loader,
>>      const char *uri, OPENSSL_CTX *libctx, const char *propq,
>>      const UI_METHOD *ui_method, void *ui_data);
>>
>> An otc_hold was put on this.. Do we need to have a formal vote?
>> This really needs to be sorted out soon so the API’s can be locked down
>> for beta.
>>
>> 
>> Also discussed in a weekly meeting and numerous PR discussions was the
>> removal of the long winded API’s ending with ‘with_libctx’
>> e.g CMS_data_create_with_libctx
>> The proposed renaming for this was to continue with the _ex notation..
>> If there is an existing _ex name then it becomes _ex2, etc.
>> There will most likely be additional parameters in the future for some
>> API’s, so this notation would be more consistent with current API’s.
>> Does this also need a vote?
>>
>> Regards,
>> Shane
>>
>>


Re: SM2 asymmetric encryption

2020-09-15 Thread Matt Caswell
I have started this vote and will report back when I have an answer.

Matt

On 14/09/2020 09:18, Matt Caswell wrote:
> Currently, 1.1.1  supports SM2 asymmetric encryption. Real world use of
> this is currently believed to be low (IIUC it is mainly useful for SM2
> in TLS, which we don't current support). A discussion in PR #12536 is
> proposing to drop this feature from 3.0 (possibly to re-introduce it in
> some later release). This was mentioned on this list by me in this post:
> 
> https://mta.openssl.org/pipermail/openssl-project/2020-August/002138.html
> 
> No feedback has been received on that point. It was also discussed at
> the recent OMC f2f (although no vote was held on the issue). Since
> dropping this support is a breaking change it requires an OMC vote to
> approve it. I'm proposing this vote wording:
> 
> "Support for SM2 asymmetric encryption should be dropped from OpenSSL 3.0"
> 
> Please let me know any thoughts on the vote wording (or any other
> thoughts on the topic). Otherwise I plan to start this vote soon.
> 
> Matt
> 


Re: OTC vote on PR11188

2020-09-14 Thread Matt Caswell
This vote is now closed (actually it closed last week but I forgot to
report it). The vote was not accepted:

for: 0, against: 8, abstained: 3, not voted: 0

On 27/08/2020 11:06, Matt Caswell wrote:
> FYI, I have initiated the following vote on PR11188. Please see the
> comments in that PR for the background. I will report back with the
> result of the vote once known.
> 
> topic: The performance improvements provided in PR11188 should be
>considered a bug fix and therefore acceptable for backport to
>    1.1.1
> Proposed by Matt Caswell
> Public: yes
> opened: 2020-08-27
> closed: 2020-mm-dd
> 
> 
> Matt
> 


Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Matt Caswell



On 14/09/2020 13:14, Tim Hudson wrote:
> On Mon, Sep 14, 2020 at 9:52 PM Matt Caswell  <mailto:m...@openssl.org>> wrote:
> 
> > And that is the point - this is not how the existing CTX functions
> work
> > (ignoring the OPENSSL_CTX stuff).
> 
> Do you have some concrete examples of existing functions that don't work
> this way?
> 
> 
> SSL_new()
> BIO_new_ssl()
> BIO_new_ssl_connect()

All of these three are consistent with my proposed policy.

The policy says:

"the key argument that identifies the thing be constructed should be
placed first (if such an argument exists)"

The parenthetical comes into play here "if such an argument exists". It
doesn't exist in any of these three cases and therefore this part
doesn't apply. In all cases the arguments are a single argument and so
trivially agree with the ordering rules.


> BIO_new_bio_pair()

This one is a slight oddity in that it doesn't create just one object
but two. I'm not aware of anywhere else where we do this, but even if we
do I don't think it is a common pattern, so I don't feel the need to
base our future policy on this precedent.


> etc
> 
> And all the existing method using functions which are also factories.
> 
> But I get the point - if there is only one argument is it logically
> coming first or last - obviously it can be seen both ways.
> 
> IMO, we absolutely MUST have the ability to delete parameters (for
> example ENGINEs). If we can do that, then I don't see why we can't add
> parameters.
> 
> 
> No - that doesn't follow. It is perfectly reasonable to have an ENGINE
> typedef that remains and is passed as NULL as usual - and in fact most
> of the top-level ENGINE stuff handles NULL as meaning no engine usage so
> that would remain consistent. There is no absolute requirement to
> delete a parameter for this or other purposes. If you want to reorder
> parameters I would argue it should be a new function name and not an _ex
> version.

I disagree with you on this.

Matt





Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Matt Caswell



On 14/09/2020 12:46, Tim Hudson wrote:
> On Mon, Sep 14, 2020 at 9:19 PM Matt Caswell  <mailto:m...@openssl.org>> wrote:
> 
> I must be misunderstanding your point because I don't follow your logic
> at all.
> 
> So this is the correct form according to my proposed policy:
> 
> TYPE *TYPE_make_from_ctx(char *name,TYPE_CTX *ctx)
> 
> 
> And that is the point - this is not how the existing CTX functions work
> (ignoring the OPENSSL_CTX stuff).

Do you have some concrete examples of existing functions that don't work
this way?


>  
> 
> > The PR should cover that context of how we name the variations of
> > functions which add the OPENSSL_CTX reference.
> 
> IMO, it does. They should be called "*_ex" as stated in chapter 6.2. I
> don't see why we need to special case OPENSSL_CTX in the policy. Its
> written in a generic way an can be applied to the OPENSSL_CTX case.
> 
> 
> And that means renaming all the with_libctx to _ex which frankly I don't
> think makes a lot of sense to do. 

My understanding of the current consensus from previous discussions is
that most people feel differently to you on this. Maybe I'm wrong.


> Having a naming indicating OPENSSL_CTX added isn't a bad thing to do in
> my view.
> Collecting a whole pile of _ex functions and having to look at each one
> to figure out what the "extension" is does add additional burden for the
> user.
> There is a reason that _with_libctx was added rather than picking _ex as
> the function names - it clearly communicates then intent rather than be
> a generic extension-of-API naming.
> 
> 
> IMO *the* most confusing thing would be to *change* an existing ordering
> 
> (for example swap two existing parameters around). I see no problem with
> adding new ones anywhere that's appropriate.
> 
> 
> Clearly we disagree on that - if you are making an extended version of a
> function and you aren't keeping the same existing parameter order (which
> you are not if you add or remove parameters which is the point I was
> making - the order isn't the same as the existing function because
> you've removed items - what you have is the order of whatever parameters
> remain in the extended function are the same - and that's a pretty
> pointless distinction to make - if you aren't simply adding additional
> items on the end you make for a change over mess and this I think we
> should be trying to avoid). My context there is for the users of the
> existing API.
> It becomes especially problematic if you have type equivalence when the
> order is changed around so there are no compiler warnings if the user
> hasn't picked up on reordering of parameters.

IMO, we absolutely MUST have the ability to delete parameters (for
example ENGINEs). If we can do that, then I don't see why we can't add
parameters.

Matt
 


Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Matt Caswell



On 14/09/2020 11:30, Matt Caswell wrote:
> In order to try and move this discussion forward I have made a concrete
> proposal for how we should formulate the various ideas in this thread
> into an actual style. Please see here:
> 
> https://github.com/openssl/web/pull/194
> 
> Since we're not yet fully in agreement some compromises will have to be
> made. I hope I've come up with something which isn't too abhorrent to
> anyone.
> 
> Please take a look.

I'm planning to gather feedback on this PR today - so if you have any
thoughts then please comment. I will post an updated version tomorrow
incorporating any feedback. I plan to start a vote on this later this
week (perhaps Wednesday).

Possibly I might do two votes: one for the proposed chapter 6.1, and one
for the proposed chapter 6.2.

Given the disagreements on this, I don't believe unanimous agreement is
going to be achievable. I hope to get it into a state where it is
acceptable to as many people as possible.

Matt


> 
> Matt
> 
> 
> On 05/09/2020 04:48, SHANE LONTIS wrote:
>>
>>   PR #12778 reorders all the API’s of the form:
>>
>>
>>   EVP_XX_fetch(libctx, algname, propq) 
>>
>>
>>   So that the algorithm name appears first.. 
>>
>>
>>   e.g: EVP_MD_fetch(digestname, libctx, propq);
>>
>> This now logically reads as 'search for this algorithm using these
>> parameters’.
>>
>> The libctx, propq should always appear together as a pair of parameters.
>> There are only a few places where only the libctx is needed, which means
>> that if there is no propq it is likely to cause a bug in a fetch at some
>> point. 
>>
>> This keeps the API’s more consistent with other existing XXX_with_libctx
>> API’s that put the libctx, propq at the end of the parameter list..
>> The exception to this rule is that callback(s) and their arguments occur
>> after the libctx, propq..
>>
>> e.g:
>> typedef OSSL_STORE_LOADER_CTX *(*OSSL_STORE_open_with_libctx_fn)
>>     (const OSSL_STORE_LOADER *loader,
>>      const char *uri, OPENSSL_CTX *libctx, const char *propq,
>>      const UI_METHOD *ui_method, void *ui_data);
>>
>> An otc_hold was put on this.. Do we need to have a formal vote?
>> This really needs to be sorted out soon so the API’s can be locked down
>> for beta.
>>
>> 
>> Also discussed in a weekly meeting and numerous PR discussions was the
>> removal of the long winded API’s ending with ‘with_libctx’
>> e.g CMS_data_create_with_libctx
>> The proposed renaming for this was to continue with the _ex notation..
>> If there is an existing _ex name then it becomes _ex2, etc.
>> There will most likely be additional parameters in the future for some
>> API’s, so this notation would be more consistent with current API’s.
>> Does this also need a vote?
>>
>> Regards,
>> Shane
>>
>>
> 


Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Matt Caswell



On 14/09/2020 11:52, Tim Hudson wrote:
> Any proposal needs to deal with the constructors consistently - whether
> they come from an OPENSSL_CTX or they come from an existing TYPE_CTX.
> That is absent in your PR.
> 
> Basically this leads to the ability to provide inconsistent argument
> order in functions.
> 
> TYPE *TYPE_make_from_ctx(TYPE_CTX *ctx, char *name)
> or
> TYPE *TYPE_make_from_ctx(char *name,TYPE_CTX *ctx)
> 
> It simply isn't consistent to basically allow both forms of this approach.
> 
> Seeing the OPENSSL_CTX as something different to the other APIs in terms
> of its usage when it is providing the context from which something is
> constructed is the underlying issue here.
> Your PR basically makes rules for "context" arguments which lead to
> allowing both the above forms - and making the current usage of CTX
> values a different logical order than the OPENSSL_CTX.

I must be misunderstanding your point because I don't follow your logic
at all.

In your example above the proposal seems clear (to me):

"Where present the TYPE argument should always be first. In constructors
or object factory type functions (such as a "fetch" function), the key
argument that identifies the thing be constructed should be placed first
(if such an argument exists)."

Your example is a "factory type" or constructor. Therefore the argument
that identifies the thing being constructued should be first, i.e. the name.

The section of the policy says:

"The remaining argument list should be put into order of importance."

Only then does it go on to list where the ctx sits in that order of
importance. The phrase "the remaining argument list" indicates that that
ordering is subordinate to the preceding paragraph.

So this is the correct form according to my proposed policy:

TYPE *TYPE_make_from_ctx(char *name,TYPE_CTX *ctx)


> 
> Separately, we should have consistency in the naming of the functions
> which take an OPENSSL_CTX - the _with_libctx makes no sense now that we
> settled on OPENSSL_CTX rather than OPENSSL_LIBCTX or OPENSSL_LIB_CTX as
> the naming. We also have a pile of _ex functions that were introduced
> just to add an OPENSSL_CTX - those should be consistently named. 

The policy is clear. We are adding extended forms of existing functions.
Therefore "_with_libctx" is not consistent with the policy. "_ex" (or
"_ex2" if necessary), is.


> 
> The PR should cover that context of how we name the variations of
> functions which add the OPENSSL_CTX reference.

IMO, it does. They should be called "*_ex" as stated in chapter 6.2. I
don't see why we need to special case OPENSSL_CTX in the policy. Its
written in a generic way an can be applied to the OPENSSL_CTX case.

> 
> The suggested rules for extended functions is inconsistent in stating
> the order "should be retained" and then allowing for inserting "at any
> point".

This is not inconsistent. The *order* must be retained. Inserting
additional arguments does *not* change the order. So according to the
policy, if you have a function:

FOO_do_something(bar, wibble, blah);

Then this is ok:

FOO_do_something(new, bar, wibble, blah);

As is this:

FOO_do_something(bar, new, wibble, blah);

And this:

FOO_do_something(bar, wibble, blah, new);

But *not* this (because the order of existing arguments is different):

FOO_do_something(blah, wibble, bar, new);



> IHMO either the new function must preserve the existing order and just
> add to the end (to easy migration) or it should conform to the naming
> scheme and parameter argument order for new functions - one of those
> should be the driver rather than basically creating something that is
> neither - not easy to migrate to and also not following the documented
> order. We should be trying to achieve one of those two objectives.

I disagree that that should be an objective.

IMO *the* most confusing thing would be to *change* an existing ordering
(for example swap two existing parameters around). I see no problem with
adding new ones anywhere that's appropriate.

Matt


Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Matt Caswell
In order to try and move this discussion forward I have made a concrete
proposal for how we should formulate the various ideas in this thread
into an actual style. Please see here:

https://github.com/openssl/web/pull/194

Since we're not yet fully in agreement some compromises will have to be
made. I hope I've come up with something which isn't too abhorrent to
anyone.

Please take a look.

Matt


On 05/09/2020 04:48, SHANE LONTIS wrote:
> 
>   PR #12778 reorders all the API’s of the form:
> 
> 
>   EVP_XX_fetch(libctx, algname, propq) 
> 
> 
>   So that the algorithm name appears first.. 
> 
> 
>   e.g: EVP_MD_fetch(digestname, libctx, propq);
> 
> This now logically reads as 'search for this algorithm using these
> parameters’.
> 
> The libctx, propq should always appear together as a pair of parameters.
> There are only a few places where only the libctx is needed, which means
> that if there is no propq it is likely to cause a bug in a fetch at some
> point. 
> 
> This keeps the API’s more consistent with other existing XXX_with_libctx
> API’s that put the libctx, propq at the end of the parameter list..
> The exception to this rule is that callback(s) and their arguments occur
> after the libctx, propq..
> 
> e.g:
> typedef OSSL_STORE_LOADER_CTX *(*OSSL_STORE_open_with_libctx_fn)
>     (const OSSL_STORE_LOADER *loader,
>      const char *uri, OPENSSL_CTX *libctx, const char *propq,
>      const UI_METHOD *ui_method, void *ui_data);
> 
> An otc_hold was put on this.. Do we need to have a formal vote?
> This really needs to be sorted out soon so the API’s can be locked down
> for beta.
> 
> 
> Also discussed in a weekly meeting and numerous PR discussions was the
> removal of the long winded API’s ending with ‘with_libctx’
> e.g CMS_data_create_with_libctx
> The proposed renaming for this was to continue with the _ex notation..
> If there is an existing _ex name then it becomes _ex2, etc.
> There will most likely be additional parameters in the future for some
> API’s, so this notation would be more consistent with current API’s.
> Does this also need a vote?
> 
> Regards,
> Shane
> 
> 


SM2 asymmetric encryption

2020-09-14 Thread Matt Caswell
Currently, 1.1.1  supports SM2 asymmetric encryption. Real world use of
this is currently believed to be low (IIUC it is mainly useful for SM2
in TLS, which we don't current support). A discussion in PR #12536 is
proposing to drop this feature from 3.0 (possibly to re-introduce it in
some later release). This was mentioned on this list by me in this post:

https://mta.openssl.org/pipermail/openssl-project/2020-August/002138.html

No feedback has been received on that point. It was also discussed at
the recent OMC f2f (although no vote was held on the issue). Since
dropping this support is a breaking change it requires an OMC vote to
approve it. I'm proposing this vote wording:

"Support for SM2 asymmetric encryption should be dropped from OpenSSL 3.0"

Please let me know any thoughts on the vote wording (or any other
thoughts on the topic). Otherwise I plan to start this vote soon.

Matt


Re: New GitHub label for release blockers

2020-09-13 Thread Matt Caswell



On 13/09/2020 15:16, Nicola Tuveri wrote:
> ... I still have very confused ideas regarding the "best" conventional
> usage of github features like labels, milestones and projects: I read
> the official documentation about them and I grasp the general ideas
> behind them, but too often the boundaries are too foggy for me to
> navigate and pick the right tool for the job in a consistent and organic
> manner. 

In my head I think of them like this:

- A label: describes a characteristic of a PR/issue
- A milestone: a time based goal (such as a specific release)
- A project: A collection of PRs/issues related by some common long term
objective

There is some confusion between labels and milestones, because you could
think of "a time based goal" set for a pr/issue as a characteristic of
it. So you could use either for the purpose. However since github has
both, it seems more appropriate to use a milestone when talking about a
time based goal since it is the more specific concept.

Matt


> 
> Nicola
> 
> On Sun, Sep 13, 2020, 17:01 Dr. Matthias St. Pierre
> mailto:matthias.st.pie...@ncp-e.com>> wrote:
> 
> Nicola suggested and convinced me, that it would be better to have a
> dedicated
> milestone for the 3.0.0 beta1 release instead of adding a new label.
> 
> So here it is, I already added all the tickets with the release
> blocker label and will
> remove the label again.
> 
> https://github.com/openssl/openssl/milestone/17
> 
> Matthias
> 
> 
> 
> > -Original Message-
> > From: Dr. Matthias St. Pierre
> > Sent: Sunday, September 13, 2020 3:17 PM
> > To: openssl-project@openssl.org 
> > Subject: New GitHub label for release blockers
> >
> > Hi all,
> >
> > taking up again the discussion from openssl-project where I
> suggested to (ab)use
> > the 3.0.0 milestone for release blockers, (see link and citation
> at the end of the mail),
> > I propose to add a new label for this purpose instead. In fact, I
> already created the label
> >
> >       [urgent: release blocker]   (see link below)
> >
> > and will add the mentioned tickets within shortly. So you can take
> a look and tell
> > me whether you like it or not. (If not, no problem. I'll just
> delete the label again.)
> >
> > Matthias
> >
> >
> > BTW: It took me all my force of will to resist the temptation of
> making a pun
> >      by naming the label [urgent: beta blocker].
> >
> >
> > References:
> > ==
> >
> > [urgent: release blocker]:
> >     
>  https://github.com/openssl/openssl/labels/urgent%3A%20release%20blocker
> >
> > [openssl-project message]:
> >     
>  
> https://mta.openssl.org/pipermail/openssl-project/2020-September/002191.html
> >
> >
> > > > > For a more accurate and timely public overview over the
> current state of the blockers,
> > > > > it might be helpful to manage them via the 3.0.0  milestone
> > > > >
> > > > > https://github.com/openssl/openssl/milestone/15
> > > > >
> > > > > Some of the tickets listed below were already associated to
> the milestone, the others
> > > > > were added by me now.
> > > >
> > > > I think the 3.0.0 milestone is what we expect to be in the
> > > > 3.0.0 release, not the beta release. That is bug fixes don't need
> > > > to be in the beta release, but if it adds new functionallity it
> > > > needs to be in the beta release.
> > >
> > > I was aware of this subtlety but I thought that we just could
> (ab-)use the milestone for
> > > the beta1 release and reuse it later for the final release,
> instead of creating a new milestone.
> > >
> > > Practically all of the relevant PRs are associated to the [3.0
> New Core + FIPS] GitHub Project
> > > anyway, so it would be possible to remove the post-beta PRs from
> the milestone and restore
> > > them later.  (In my mind, I see project managers running away
> screeming...)
> > >
> > > Matthias
> > >
> > >
> > > [3.0 New Core + FIPS]: 
> https://github.com/openssl/openssl/projects/2
> 


Monthly Status Report (August)

2020-09-11 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Raised PR to replace the legacy KDF bridge with a provider based KDF
bridge
- Continued work on and merged a fix to ensure that the default config
file has been loaded before working with default properties
- Performed the alpha 6 release
- Implemented a provider based MAC bridge
- Investigated and fixed an issue with stitched ciphersuites in TLSv1.0
- Drafted the job description for the new "Administrator and Manager" role
- Implemented a provider side HMAC implementation that is TLS aware
- Implemented checks in libssl to confirm if we have MD5-SHA1 or not
- Attended the OMC virtual face-2-face
- Wrote patches and security advisory in response to the Raccoon attack


Matt



Re: Beta1 PR deadline

2020-09-10 Thread Matt Caswell



On 10/09/2020 11:40, Matt Caswell wrote:
> 
> 
> On 09/09/2020 13:03, Kurt Roeckx wrote:
>> On Wed, Aug 26, 2020 at 04:58:26PM +0100, Matt Caswell wrote:
>>> Please can anyone with PRs that they wish to have included in OpenSSL
>>> 3.0 beta1 ensure that they are merged to master by 8th September.
>>
>> So that date has passed now. Can someone give an overview of what
>> we think is still needed to be done before the beta 1 release? Are
>> there open PRs for everything? Do we a milestone on github to
>> indicate which PRs need to go in before beta1?
> 
> 
> I believe the "blockers" to be:
> 
> https://github.com/openssl/openssl/pull/12536
> https://github.com/openssl/openssl/pull/12754
> https://github.com/openssl/openssl/pull/12750
> https://github.com/openssl/openssl/pull/12781
> https://github.com/openssl/openssl/pull/12801
> 
> Aside from these there are potential PRs not yet opened but slated for
> beta1 regarding:
> - Refactor serialization? (owner: Richard)
> - lhash refactor? (owner: Me)
> - IG D.9 update? (owner: Shane)

Oh, I think this last one might be:

https://github.com/openssl/openssl/pull/12835

Matt

> 
> There's also this issue - but I'm not sure that it needs to be resolved
> for beta1:
> https://github.com/openssl/openssl/issues/12195
> 
> Matt
> 


Re: Beta1 PR deadline

2020-09-10 Thread Matt Caswell



On 09/09/2020 13:03, Kurt Roeckx wrote:
> On Wed, Aug 26, 2020 at 04:58:26PM +0100, Matt Caswell wrote:
>> Please can anyone with PRs that they wish to have included in OpenSSL
>> 3.0 beta1 ensure that they are merged to master by 8th September.
> 
> So that date has passed now. Can someone give an overview of what
> we think is still needed to be done before the beta 1 release? Are
> there open PRs for everything? Do we a milestone on github to
> indicate which PRs need to go in before beta1?


I believe the "blockers" to be:

https://github.com/openssl/openssl/pull/12536
https://github.com/openssl/openssl/pull/12754
https://github.com/openssl/openssl/pull/12750
https://github.com/openssl/openssl/pull/12781
https://github.com/openssl/openssl/pull/12801

Aside from these there are potential PRs not yet opened but slated for
beta1 regarding:
- Refactor serialization? (owner: Richard)
- lhash refactor? (owner: Me)
- IG D.9 update? (owner: Shane)

There's also this issue - but I'm not sure that it needs to be resolved
for beta1:
https://github.com/openssl/openssl/issues/12195

Matt


OpenSSL is looking for a full time Administrator and Manager

2020-09-05 Thread Matt Caswell
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

The OpenSSL Management Committee are looking to hire a full time
Administrator and Manager. Details of the role can be found here:

https://www.openssl.org/blog/blog/2020/09/05/OpenSSL.ProjectAdminRole/

To apply please send your cover letter and resume to j...@openssl.org by
20th September 2020.

Regards,
The OpenSSL Project Team
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAl9TVxgACgkQ2cTSbQ5g
RJHKGggAn1YGhR7UwtgVXTMWUKiv4jYpXd5OaHonAaUwIFdkXUzBmmEq9PP1Thw/
A4rQ/anDZ6SfRlFaGxQB1Fyz5LRyNDhHA48lM0v/Yw55S6NfSrMaPcGRuU8Odikf
4Nd7zzD3RcOgfhphdHEXz7ykMi90ATVcLTVnaoQtkvw5LHeiXzqzBLT9+WEcENWU
4z2WLJRGTpwIBfYfm6/NQPTDzsy/VBoVW/nl1mx6jkvL2UxuOdp4rfTMz9lu3IPk
CnkujXxDIVSn02xSiRccj3ujnFqOq4lwtSiOzOl/HowlCDY6DmRhIvsu1PnPzJ15
v5JbQhDpk4kHjalCsJq2QfdcP41pDQ==
=h9gU
-END PGP SIGNATURE-


OTC vote on PR11188

2020-08-27 Thread Matt Caswell
FYI, I have initiated the following vote on PR11188. Please see the
comments in that PR for the background. I will report back with the
result of the vote once known.

topic: The performance improvements provided in PR11188 should be
   considered a bug fix and therefore acceptable for backport to
   1.1.1
Proposed by Matt Caswell
Public: yes
opened: 2020-08-27
closed: 2020-mm-dd


Matt



Re: Beta1 PR deadline

2020-08-26 Thread Matt Caswell



On 26/08/2020 17:02, Salz, Rich wrote:
>>Please can anyone with PRs that they wish to have included in OpenSSL
> 3.0 beta1 ensure that they are merged to master by 8th September.
> 
> And how are non-committers supposed to do that
> 

In the same way as normal. Ensure your PRs are raised asap, and
encourage committers/OTC to review them.

Matt


Beta1 PR deadline

2020-08-26 Thread Matt Caswell
Hi all

The OMC had a meeting today.

Please can anyone with PRs that they wish to have included in OpenSSL
3.0 beta1 ensure that they are merged to master by 8th September.

Note in particular that there is no PR at the moment to incorporate SM2
asymmetric encryption into OpenSSL 3.0. This feature currently exists in
1.1.1, so if no PR emerges by the above date then this feature is liable
to be dropped in 3.0. (Note: PRs for SM2 signatures *do* exist and are
expected to be present).


Matt




Monthly Status Report (July)

2020-08-12 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:


- Continued and completed work on PR to fix CMP related msan failures
- Continued and completed work on moving TLS CBC code into the providers
- Continued and completed work on fixing an issue in 
OSSL_PROVIDER_get_capabilities()
- Ongoing review work on the KTLS patches
- Fixed no-dh and no-dsa
- Fixed no-e2m
- Fixed a test failure in test_verify
- Fixed an issue where Digest[Sign|Verify]Init were falling back to legacy code 
paths to easily
- Fixed an issue with test_cmp_cli in the extended tests
- Investigated and fixed and issue with 
EVP_default_properties_is_fips_enabled() not working as expected
- Significant work on moving constant time MAC code out of libssl into providers
- Significant work on moving legacy KDF bridge into the providers

I also took a one week holiday during July.

Matt




Re: RAND_DRBG

2020-07-27 Thread Matt Caswell
I'm ok with option 1 (but it will require a vote). I think the
percentage of our user base that are using the existing API is
sufficiently close to zero that we're not breaking our compatibility
promises.

Matt


On 27/07/2020 02:08, Dr Paul Dale wrote:
> The RAND_DRBG (crypto/rand/drbg_lib) APIs are quite some mess and sit
> badly with the move to provider based infrastructure.
> They are definitely being deprecated in master but without more, the
> extra layer of indirection and additional complexity generating random
> numbers will remain.
> 
> The option I see are:
> 
> 1. Remove, rather than deprecate, RAND_DRBG in 3.0.  This is a breaking
> change.
> 2. Bypass RAND_DRBG and live with a breaking change (refer:
> https://github.com/openssl/openssl/pull/12509#pullrequestreview-455396828)
> 3. Leave things as they currently are in master.
> 
> The first two are breaking changes and will require an OMC vote.
> 
> 
> Some pertinent points:
> 
> 1. RAND_bytes will continue to work as is — this API is perfect for
> almost everyone.
> 2. The RAND_METHOD functionality remains — this allows wholesale
> replacement of OpenSSL’s RNGs if desired.
> 3. The name EVP_RAND is the working name and might change — this is not
> relevant to this discussion.
> 4. The RAND_DRBG APIs are unlikely to be widely used — they were
> introduced in 1.1.1.  The two users I know of (Akamai and NCP) are both
> fine with them being removed.
> 
> 
> Thoughts anyone?
> 
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 


Re: API renaming

2020-07-23 Thread Matt Caswell



On 23/07/2020 16:52, Richard Levitte wrote:
> On Thu, 23 Jul 2020 12:18:10 +0200,
> Dr Paul Dale wrote:
>> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  This seems 
>> reasonable.  Would it
>> also make sense to rename the other new APIs similarly.
>> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and OSSL_KDF respectively?
> 
> This is a good question...
> 
> Historically speaking, even though EVP_MAC and EVP_KDF are indeed new
> APIs, they have a previous history of EVP APIs, through EVP_PKEY.  The
> impact of relocating them outside of the EVP "family" may be small,
> but still, history gives me pause.
> 
> RAND doesn't carry the same sort of history, which makes it much
> easier for me to think "just do it and get it over with"...

I have the same pause - so  I'm thinking just RAND for now.

Matt



Monthly Status Report (June)

2020-07-09 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Continued work on and subsequently merged PR 11834 to check signature
algorithms are available before offering or accepting them
- Continued work on and subsequently merged PR 11898 to avoid
downgrading keys in libssl
- Continued work on and subsequently merged PRs 11972 and 12107 to
ensure CMAC_CTX is correctly initialised
- Performed the alpha 3 release
- Continued work on and subsequently merged PR 11914 for fully pluggable
TLSv1.3 key exchange
- Significant review work on the CMP chunk 12 contribution
- Fixed the s_server -dtls option via PR 12179
- Fixed some SSL_dup issues and provided better documentation for it via
PR 12180 and 12245
- Fixed an issue to ensure we use the passed libctx when loading EC
private keys (PR12159)
- Fixed some man page typos (PR 12185)
- Created PR to revert the KDF/MAC renaming (PR 12186)
- Fixed an issue where the provider ctx was forgotten during a reset
function call (PR 12229)
- Helped out on PR 12228 to ensure the ASYNC code behaved correctly with
respect to OPENSSL_CTX_set0_default()
- Performed the alpha 4 release
- Fixed some CMP related msan failures (PR 12275)
- Created PR to move the TLS CBC code into the providers (PR12288)
- Provided a fix for OSSL_PROVIDER_get_capabilities() (PR12292)


Matt


Re: Cherry-pick proposal

2020-07-09 Thread Matt Caswell



On 02/06/2020 15:29, Matt Caswell wrote:
> 
> There's been no further discussion on this for quite a while, so I will
> start an OTC vote based on the vote text proposed by Matthias and report
> back the results here.

Sorry, I forgot to report back. The final result was:

+1: 4
-1: 4
 0: 3

So the result was tied. According to our bylaws this means that the vote
does not pass.

Matt



Re: Naming conventions

2020-07-06 Thread Matt Caswell



On 06/07/2020 07:41, Richard Levitte wrote:
> On Fri, 03 Jul 2020 11:25:37 +0200,
> Matt Caswell wrote:
>> On 19/06/2020 08:15, Tomas Mraz wrote:
>>> to something like:
>>>
>>> int EVP_MacInit(EVP_MAC_CTX *ctx);
>>>   
>>> int EVP_MacUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
>>> datalen);
>>>  
>>> int EVP_MacFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
>>> outsize);
>>>
>>> or at least
>>>
>>> int EVP_MACInit(EVP_MAC_CTX *ctx);
>>>   
>>> int EVP_MACUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
>>> datalen);
>>>  
>>> int EVP_MACFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
>>> outsize);
>>>
>>> Should we do that? I hope for the sheer ugliness of the supposedly
>>> consistent names that we do not.
>>
>> This pattern is not universally used (as you mention the EVP_PKEY
>> API does something different).
> 
> Er, you've choose what you wanted to read, I suppose...  For fairly
> high level EVP APIs, the CamelCase form is actually quite consistent.
> For EVP_PKEY, we have them covering most if not all higher level
> usages:

I'm actually ok with using the CamelCase versions as well. But in the
interests of choosing my battles I'm for the moment just focusing on
getting us back to where we started.


> 
> EVP_Sign{Init, Update, Final}
> EVP_Verify{Init, Update, Final}
> EVP_Open{Init, Update, Final}
> EVP_Seal{Init, Update, Final}
> 
>> I remain strongly opposed to this renaming as I really don't think it
>> helps to do this sort of thing now. It just introduces significant
>> confusion without a long term strategy. We are too close to 3.0 beta1 to
>> embark on this journey now. I'm also not convinced that strategically
>> this is the right pattern to use.
> 
> What I hear from this is that 3.0 is the wrong time to introduce a new
> (and hopefully better) public API, that we should post-pone that to
> something like 4.0.  I can get along with that line of thought.

Ok. Good.

Matt





Re: Naming conventions

2020-07-03 Thread Matt Caswell



On 19/06/2020 08:15, Tomas Mraz wrote:
> to something like:
> 
> int EVP_MacInit(EVP_MAC_CTX *ctx);
>   
> int EVP_MacUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> datalen);
>  
> int EVP_MacFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> outsize);
> 
> or at least
> 
> int EVP_MACInit(EVP_MAC_CTX *ctx);
>   
> int EVP_MACUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> datalen);
>  
> int EVP_MACFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> outsize);
> 
> Should we do that? I hope for the sheer ugliness of the supposedly
> consistent names that we do not.

This pattern is not universally used (as you mention the EVP_PKEY API
does something different). This is one of the areas of existing
quirkiness, so I'd be ok with leaving these names as they are.

But the fact that there is some strangeness in this area that we may be
willing to accept does not mean we should accept a totally new pattern
in other function names, i.e.

EVP_MAC_CTX_new -> EVP_MAC_new_ctx
EVP_MAC_CTX_free -> EVP_MAC_free_ctx
EVP_MAC_CTX_dup -> EVP_MAC_dup_ctx
EVP_MAC_CTX_mac -> EVP_MAC_get_ctx_mac
EVP_MAC_CTX_get_params -> EVP_MAC_get_ctx_params
EVP_MAC_CTX_set_params -> EVP_MAC_set_ctx_params

I remain strongly opposed to this renaming as I really don't think it
helps to do this sort of thing now. It just introduces significant
confusion without a long term strategy. We are too close to 3.0 beta1 to
embark on this journey now. I'm also not convinced that strategically
this is the right pattern to use.

In the interests of getting to some kind of decision on this I think a
simple OTC vote is probably in order. I already have a PR to revert this
(12186) which has an OTC hold on it. Therefore a simple vote to lift the
hold would be sufficient to at least make a decision for now.

My proposed vote text is:

"We should lift the OTC hold on PR12186 and continue the normal review
process"


Matt


New blog post

2020-06-30 Thread Matt Caswell
I've just published a new blog post by Nicola on the alpha 4 release.
You can read it here:

https://www.openssl.org/blog/blog/2020/06/25/OpenSSL3.0Alpha4/

Thanks Nicola!

Matt


Re: Backports to 1.1.1 and what is allowed

2020-06-25 Thread Matt Caswell



On 25/06/2020 15:33, Nicola Tuveri wrote:
> In light of how the discussion evolved I would say that not only there
> is consensus on supporting the definition of a detailed policy on
> backports and the definitions of what are the requirements for regular
> releases vs LTS releases (other than the longer support timeframe),
> but also highlights a need to do it sooner rather than later!
> 
> This seems a job for the OMC, as it falls under:
> 
>> makes all decisions regarding management and strategic direction of the 
>> project; including:
>> - business requirements;
>> - feature requirements;
>> - platform requirements;
>> - roadmap requirements and priority;
>> - end-of-life decisions;
>> - release timing and requirement decisions;
> 
> My contribution to the discussion is to ask if the OMC has plans for
> addressing this in the very short term.

I think its unlikely we are going to get to a policy in the short term.
It seems to me we are still some way away from a consensus here.

> If working on a policy is going to be a medium-term effort, maybe it
> would be opportune to call an OTC vote specific to #11968 under the
> current release requirements defined by the OMC (or lack thereof).

11968 is already merged and, AFAIK, no one has proposed reverting it. If
such a PR was raised then a vote might be a way forward for it.

11188 is the more pressing problem because that is currently unmerged
and stuck. That has an OTC hold on it (placed there by me), so an OTC
vote seems appropriate. If a vote is held it should be to decide whether
backporting it is consistent with our current understanding of the
policy such as it is. It is for the OMC to decide whether a different
policy should be introduced at some point in the future.

Matt


> 
> We already saw a few comments in favor of evaluating backporting
> #11968 as an exception, in light of the supporting arguments, even if
> it was in conflict with the current policy understanding or the
> upcoming policy formulation.
> So if we could swiftly agree on this being an OTC or OMC vote, I would
> urge to have a dedicated discussion/vote specific to #11968, while
> more detailed policies and definitions are in the process of being
> formulated.
> 
> - What is the consensus on splitting the 2 discussions?
> - If splitting the discussions, is deciding on #11968 an OTC or OMC matter?
> 
> 
> 
> Cheers,
> 
> Nicola
> 


Re: Backports to 1.1.1 and what is allowed

2020-06-22 Thread Matt Caswell



On 20/06/2020 01:11, Tim Hudson wrote:
> I suggest everyone takes a read through 
> https://en.wikipedia.org/wiki/Long-term_support as to what LTS is
> actually meant to be focused on.
> 
> What you (Ben and Matt) are both describing is not LTS but STS ... these
> are different concepts.
> 
> For LTS the focus is *stability *and *reduced risk of disruption *which
> alters the judgement on what is appropriate to put into a release.
> It then becomes a test of "is this bug fix worth the risk" with the
> general focus on lowest possible risk which stops this sort of thing
> happening unless a critical feature is broken.
> 
> All of the "edge case" examples presented all involve substantial
> changes to implementations and carry an inherent risk of breaking
> something else - and that is the issue.
> It would be different if we had a full regression test suite across all
> the platforms and variations on platforms that our users are operating
> on - but we don't.
> 
> We don't compare performance between releases or for any updates in our
> test suite so it isn't part of our scope for release readiness - if this
> is such a critical thing that must be fixed then it is something that we
> should be measuring and checking as a release condition - but we don't -
> because it actually isn't that critical. 
> 

I read the page and I don't think it is contrary to anything that I
said. From the characteristics section:

"At the beginning of a long-term support period, the software developers
impose a feature freeze: They make patches to correct software bugs and
vulnerabilities, but do not introduce new features that may cause
regression. The software maintainer either distributes patches
individually, or packages them in maintenance releases, point releases,
or service packs. At the conclusion of the support period, the product
either reaches end-of-life, or receives a reduced level of support for a
period of time (e.g., high-priority security patches only).[2]"

AFAIK this is exactly what we do. We fix bugs and don't add new
features. After 4 years we transition to security fixes only.

Later, in the Rational section, it says this:

"Long-term support addresses this by assuring users and administrators
that the software will be maintained for a specific period of time, and
that updates selected for publication will carry a significantly reduced
risk of regression.[2] The maintainers of LTS software only publish
updates that either have low IT risk or that reduce IT risk (such as
security patches). Patches for LTS software are published with the
understanding that installing them is less risky than not installing them."

This, IMO, is exactly what we do.

In none of this does it say that we shouldn't fix less serious bugs or
performance regressions. It talks about risk - but that does not
preclude those kinds of fixes. In fact as I said in my answer to Kurt
earlier in this thread I do believe that risk is an important factor in
deciding what things get backported. However I don't believe that,
generally speaking, most minor bug fixes represent a significant source
of risk. I also think the same *might* be true of *some* types of
performance regressions.

Matt


Re: Backports to 1.1.1 and what is allowed

2020-06-19 Thread Matt Caswell



On 19/06/2020 22:58, Kurt Roeckx wrote:
> On Fri, Jun 19, 2020 at 10:29:24PM +0100, Matt Caswell wrote:
>>
>> My immediate reaction to that is no - it shouldn't go to 1.1.1. That
>> would impact a very high proportion of our user base.
> 
> So is risk an important factor? How do you judge the risk? By the
> size of the change?

Yes, I do believe risk is an important factor although putting hard
rules around it is very difficult. There are bound to be some fuzzy
lines here between what is and isn't acceptable.

Matt


Re: Backports to 1.1.1 and what is allowed

2020-06-19 Thread Matt Caswell



On 19/06/2020 23:34, Tim Hudson wrote:
> 
> 
> On Sat, 20 Jun 2020, 8:14 am Benjamin Kaduk,  > wrote:
> 
> On Sat, Jun 20, 2020 at 08:11:16AM +1000, Tim Hudson wrote:
> > The general concept is to only fix serious bugs in stable releases.
> > Increasing performance is not fixing a bug - it is a feature.
> 
> Is remediating a significant performance regression fixing a bug?
> 
> 
> It would be a bug - but not a serious bug. So no.
> It works. It was released. 

I don't recall us ever saying that we would only fix serious bugs.
AFAIK, if its a bug then we will fix it. IMO a serious performance
regression would qualify, within reason (e.g. major rewrites of the
assembler would not be ok).

> Wholesale replacement of implementations of algorithms should not be
> happening in LTS releases.

This I agree with.

Matt


Re: Backports to 1.1.1 and what is allowed

2020-06-19 Thread Matt Caswell



On 19/06/2020 21:42, Kurt Roeckx wrote:
> I think one other thing that has come up is adding support for a
> new target, which can just be some small change to configuration
> files. Is that something we want to accept?

I think previously we have said that a new target is a new feature and
therefore haven't allowed it. But even this I think is a grey area. If a
target could be added simply by adding some lines to Configure, probably
the risk to our existing users is very low. OTOH, if adding a platform
means adding lots of ifdef hackery throughout the codebase then the risk
of something going wrong is significantly higher.

> So we currently also have PR #12201 that adds a new constant time
> P-384 implementation. Do you think we should backport that to the
> 1.1.1 branch? If the answer is different than for the assembler,
> why?

My immediate reaction to that is no - it shouldn't go to 1.1.1. That
would impact a very high proportion of our user base.

Matt


Re: Naming conventions

2020-06-18 Thread Matt Caswell



On 18/06/2020 13:03, Richard Levitte wrote:
> 
> Okie, if we're going to start this discussion with taking a stand, I
> guess I'll declare that while I initially had the exact same concern,
> I now see this change in a positive light.  This comment from #11997
> got me to change my mind:
> 
> The already established patterns is a furphy. I'll reword it more
> precisely: we've done things badly in the past, thus we should do
> them badly moving forwards. We've always been at war with
> Eurasia.oh wrong context, it's Eastasia.
> 

While I can agree with the sentiment here I disagree with the conclusion
that the answer is to change the APIs piecemeal to introduce even more
quirkiness and inconsistency in the APIs.

I'm also not convinced that the currently slightly strange
inconsistencies are worth fixing. Yes, if we were to design it again we
would do it differently. But that does not mean that we have to change
everything to fit into our ideal picture of the way things should be.

To fix it necessarily involves changing the names of a lot of different
functions (how many depends on exactly what convention we eventually
decided upon). Yes, we can lessen the impact by providing backwards
compat macros - but that doesn't mean that a change of name is without
cost. Either applications must change all of the names in their code to
the new form OR they keep the old form and start using the new form for
new code. This pushes the complexity of the name change onto our
downstream application maintainers. They will have all the confusion of
different naming conventions within their application.

It adds further complexity to our documentation. Either we only document
the new forms (which is confusing when application maintainers and
authors see all the old forms being used "out there") - or we have to
ensure all the synonyms are documented everywhere - which is itself
confusing to our users. And our documentation will be at odds with all
the examples and samples and unofficial documentation that our users
undoubtedly use.

I think a wholesale name change can only really be considered if it
brings significant benefits. At the moment I'm struggling to see that
the benefits outweigh the costs.

Even if we decided that we would want to do this, we should also think
twice about doing it in the 3.0 timeframe. We are already pushing a lot
of change onto our users with the change to our architecture. Pushing
more change for dubious benefit doesn't seem like a great idea.

> As for not doing something piecemeal, I actually disagree, I see
> benefit in more than just one person getting their hands dirty (plus
> changing everything in one go may be overwhelming, and would make for
> another huge PR that takes overly much time to get through).  However,
> as strategies go, and if we agree on making the change, we could very
> well create an issue for each affected sub-API and have it point at a
> common page that describes what we agreed upon...  this could be a
> good use of the github wiki tab, for example.

I don't mean piecemeal in the sense of doing it spread over a number of
PRs. I mean piecemeal in the sense of doing it spread over a number of
releases. As far as I can tell #11996 and #11997 were one offs without
any long term strategy in mind to convert the whole API in this way. In
the absence of a strategy we shouldn't be making one off changes like this.

>> There *are* inconsistencies and quirks in our current naming
>> conventions. I am skeptical that now is the time to resolve them given
>> the number of other changes we are already introducing into 3.0.
> 
> When do you see that time being, then?  3.1 (we've talked about it
> being a "cleanup" release)?  4.0?

Perhaps never. But if we do it then either 3.1 or 4.0 could be
considered. I am yet to be convinced that its worth it.

Matt



Naming conventions

2020-06-18 Thread Matt Caswell
PRs #11996 and #11997 made some changes to the EVP_MAC and EVP_KDF API
naming conventions. Specifically (in the MAC case) renaming:

EVP_MAC_CTX_new -> EVP_MAC_new_ctx
EVP_MAC_CTX_free -> EVP_MAC_free_ctx
EVP_MAC_CTX_dup -> EVP_MAC_dup_ctx
EVP_MAC_CTX_mac -> EVP_MAC_get_ctx_mac
EVP_MAC_CTX_get_params -> EVP_MAC_get_ctx_params
EVP_MAC_CTX_set_params -> EVP_MAC_set_ctx_params

There are similar renamings for the KDF APIs.

I am opposed to this renaming on the basis that it is inconsistent with
what we do elsewhere. For example, except for the MAC/KDF APIs I see
there are 26 functions of the form:

FOO_CTX_new

The case is similar for FOO_CTX_free. Aside from the newly introduced
MAC/KDF names there are no other instances of FOO_new_ctx or
FOO_free_ctx. This is totally inconsistent and, IMO, will cause
significant confusion for our users.

I will let others describe the rationale and arguments in favour of the
change, because I wasn't involved in those discussions.

I have proposed a PR to revert the changes (12186) which has an OTC hold
on it pending the discussion that I hope to kick off in this thread. As
I wrote in that PR:

"If we want to change the naming conventions then we should do so only
after agreeing what those conventions should be, and agreeing a strategy
for how we are going to apply them. It should not be done piecemeal (and
hidden away in a PR that supposedly made things more consistent but in
reality did the opposite)."

There *are* inconsistencies and quirks in our current naming
conventions. I am skeptical that now is the time to resolve them given
the number of other changes we are already introducing into 3.0. I'm
also skeptical about introducing changes to well established APIs that
have been used for many years out of some purist sense of order.
However, I'm willing to listen to the arguments on that.

Thoughts please?

Matt






Re: Reducing the security bits for MD5 and SHA1 in TLS

2020-06-18 Thread Matt Caswell
I will start the OMC vote using the text as previously proposed.

Matt

On 18/06/2020 03:20, Tim Hudson wrote:
> Given that this change impacts interoperability in a major way it should
> be a policy vote of the OMC IMHO.
> 
> Tim.
> 
> 
> On Thu, 18 Jun 2020, 5:57 am Kurt Roeckx,  <mailto:k...@roeckx.be>> wrote:
> 
> On Wed, May 27, 2020 at 12:14:13PM +0100, Matt Caswell wrote:
> > PR 10787 proposed to reduce the number of security bits for MD5
> and SHA1
> > in TLS (master branch only, i.e. OpenSSL 3.0):
> >
> > https://github.com/openssl/openssl/pull/10787
> >
> > This would have the impact of meaning that TLS < 1.2 would not be
> > available in the default security level of 1. You would have to
> set the
> > security level to 0.
> >
> > In my mind this feels like the right thing to do. The security bit
> > calculations should reflect reality, and if that means that TLS <
> 1.2 no
> > longer meets the policy for security level 1, then that is just the
> > security level doing its job. However this *is* a significant breaking
> > change and worthy of discussion. Since OpenSSL 3.0 is a major
> release it
> > seems that now is the right time to make such changes.
> >
> > IMO it seems appropriate to have an OMC vote on this topic (or
> should it
> > be OTC?). Possible wording:
> 
> So should that be an OMC or OTC vote, or does it not need a vote?
> 
> 
> Kurt
> 


Backports to 1.1.1 and what is allowed

2020-06-16 Thread Matt Caswell
PR 11188 proposes to backport a series of s390x patches to the 1.1.1
branch. IIUC it includes performance improvements as well as support for
new hardware instructions.

I think we need to have a much clearer and more explicit policy about
exactly what is allowed to be backported to a stable branch. For example
PR #11968 was a significant recent PR that backported AES CTR-DRBG
performance improvements to the 1.1.1 branch and was allowed (should it
have been?).

We have always said that the stable releases should only receive bug and
security fixes. Performance improvements have been a bit of a grey area,
e.g. a few lines of code that significantly improves the performance of
a particular algorithm or codepath could reasonably be justified as
"fixing a performance bug". OTOH bringing in whole new files of
assembler seems to go way beyond that definition.

However, when I tried to find a reference to the policy which says
exactly what we are allowed to backport I could find one. Do we have
such a thing?

In any case I think we should develop a much more explicit policy that
will enable us to decide whether PRs such as 11188 are reasonable or
not. See for example Ubuntu's Stable Release Updates policy:

https://wiki.ubuntu.com/StableReleaseUpdates


Matt


Monthly Status Report (May)

2020-06-11 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Investigated a mysterious perl crash during Configure on some platforms
- Attended regular weekly dev team calls, and fortnightly FIPS sponsor calls
- Created PR to ensure we don't offer accept ciphersuites that we don't
support
- Continued work on PR to centralise environment variables in the tests
- Created PR to document some missing s_server options
- Completed PR fixing raw provider keys
- Published OpenSSL 1.0.2v for premium support customers
- Implemented stricter type discipline in the provider<->libcrypto interface
- Performed the OpenSSL 3.0 alpha 2 release
- Fixed the alignment calculation in ssl3_setup_write
- Continued review work on the CMP submission
- Fixed an issue where we were incorrectly falling back to legacy if a
fetch of the EVP_KEYMGMT failed.
- Deleted the redundant sslprovidertest
- Ensured that we check the availability of sigalgs before we offer or
accept them
- Removed some deliberate downgrading of keys in libssl
- Investigated and fixed issues in the CMAC implementation


Matt


addrev warning

2020-06-08 Thread Matt Caswell
After upgrading Ubuntu over the weekend I'm suddenly seeing this warning
from addrev. Is anyone else getting this?

WARNING: git-filter-branch has a glut of gotchas generating mangled history
 rewrites.  Hit Ctrl-C before proceeding to abort, then use an
 alternative filtering tool such as 'git filter-repo'
 (https://github.com/newren/git-filter-repo/) instead.  See the
 filter-branch manual page for more details; to squelch this warning,
 set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...


MMatt


Alpha 3 Blog post

2020-06-05 Thread Matt Caswell
Nicola's latest blog post on the Alpha 3 release has just been published.

You can read it here:

https://www.openssl.org/blog/blog/2020/06/05/OpenSSL3.0Alpha3/


Matt


Re: Cherry-pick proposal

2020-06-02 Thread Matt Caswell



On 29/04/2020 14:28, Dr. Matthias St. Pierre wrote:
> - First, a pull request needs to be opened against the master branch for
> discussion.
> 
>   Only after that pull request has received the necessary amount of
> approvals,
> 
>   a separate pull request can be opened  against the
> OpenSSL_1_1_1-stable branch.
> 
>  
> 
> - A separate pull request against the OpenSSL_1_1_1-stable branch is
> required.
> 
>   This holds - contrary to common practice - even if the change can be
> cherry-picked
> 
>   without conflicts from the master branch. The only exception from this
> rule are
> 
>   changes which are considered 'CLA: trivial', like e.g. typographical
> fixes.
> 

There's been no further discussion on this for quite a while, so I will
start an OTC vote based on the vote text proposed by Matthias and report
back the results here.

Matt


Re: Reducing the security bits for MD5 and SHA1 in TLS - OTC or OMC vote?

2020-05-27 Thread Matt Caswell



On 27/05/2020 15:33, Tomas Mraz wrote:
> On Wed, 2020-05-27 at 14:16 +, Dr. Matthias St. Pierre wrote:
>>> IMO it seems appropriate to have an OMC vote on this topic (or
>>> should it
>>> be OTC?). Possible wording:
>>
>> Personally, I would prefer if technical questions would by default be
>> discussed (and voted on)
>> by the OTC, unless an OMC member explicitly puts in his veto and
>> claims that higher level
>> strategical interests of the OpenSSL project are affected.
>>
>> But according to the current wording of the bylaws, I would say it is
>> a 'feature requirement' and
>> requires an OMC vote:
> 
> I do not understand this to be a 'feature requirement' - IMO if this
> was a 'feature requirement' it would mean that OMC decides that
> something must be implemented in such and such way that the OpenSSL 3.0
> does this and that as a feature. But we do not do that for every
> feature that is being added to master. So I do not even think this
> requires any formal vote, unless someone from OTC or OMC calls for it
> explicitly.
> 
> Of course it is kind-of API break but again I do not think every API
> break in OpenSSL 3.0 was voted upon by OMC.
> 
> I mean I am definitely not against having a vote if someone feels it
> should be done but if nobody requires it, I do not think it would be a
> violation of anything if this is merged without a vote.

I think there should be a vote. IMO such a significant break should be
done as a result of a positive decision and not on the basis of a very
small number of people approving a PR.

I can see arguments both ways for it being an OTC vote or an OMC vote.
To an extent it is purely a technical decision i.e. to answer the
question: "does it technically make sense to make this change?"

It also has a business requirements aspect to it i.e. to answer the
question "would this have such a significant impact on the OpenSSL user
base that, regardless of its technical merits, we still shouldn't do it?"

On reflection though I'm not sure that the technical merits of this are
particularly controversial. So I'm thinking that the OMC is still the
right forum for this. However if someone else thinks that the
*technical* arguments are controversial there is no reason why we
couldn't have an OTC vote *as well*. I won't be proposing that though.


Matt




Reducing the security bits for MD5 and SHA1 in TLS

2020-05-27 Thread Matt Caswell
PR 10787 proposed to reduce the number of security bits for MD5 and SHA1
in TLS (master branch only, i.e. OpenSSL 3.0):

https://github.com/openssl/openssl/pull/10787

This would have the impact of meaning that TLS < 1.2 would not be
available in the default security level of 1. You would have to set the
security level to 0.

In my mind this feels like the right thing to do. The security bit
calculations should reflect reality, and if that means that TLS < 1.2 no
longer meets the policy for security level 1, then that is just the
security level doing its job. However this *is* a significant breaking
change and worthy of discussion. Since OpenSSL 3.0 is a major release it
seems that now is the right time to make such changes.

IMO it seems appropriate to have an OMC vote on this topic (or should it
be OTC?). Possible wording:

"The TLS security bit values for MD5, MD5_SHA1 and SHA1 should be set to
39, 67 and 65 respectively in OpenSSL 3.0. Consequently TLS < 1.2 will
be disallowed in the default security level"

Thoughts?

Matt



Alpha2 blog post

2020-05-19 Thread Matt Caswell
Nicola has kindly written another blog post for us - this time on the
alpha 2 release. I've just published it, and you can read it here:

https://www.openssl.org/blog/blog/2020/05/16/OpenSSL3.0Alpha2/

Matt


Re: Repo is frozen

2020-05-15 Thread Matt Caswell
The release is now complete, and the repo is unfrozen.

Matt


On 14/05/2020 18:52, Matt Caswell wrote:
> The repo is frozen is readiness for the alpha2 release.
> 
> I'll let you know when it is available again.
> 
> Matt
> 


Repo is frozen

2020-05-14 Thread Matt Caswell
The repo is frozen is readiness for the alpha2 release.

I'll let you know when it is available again.

Matt


Re: Alpha2

2020-05-13 Thread Matt Caswell
On 08/05/2020 09:55, Matt Caswell wrote:
> Various OpenSSL 3.0 developers met (virtually) on Tuesday to discuss
> current progress. It was proposed that we should do the Alpha 2 release
> next week (on Thursday 14th May). Unless I hear objections otherwise, I
> plan to go with that.

I'm currently thinking that we're not quite ready for alpha2 and I would
like to push it by a day. Any objections?

At the dev meeting on Tuesday we discussed a slight change of approach
with the alpha releases, i.e. rather than have a set number of alpha
releases - we would simply do them every 3 weeks or so. I'd like to hear
opinions on that proposal too.

Matt



Alpha2

2020-05-08 Thread Matt Caswell
Various OpenSSL 3.0 developers met (virtually) on Tuesday to discuss
current progress. It was proposed that we should do the Alpha 2 release
next week (on Thursday 14th May). Unless I hear objections otherwise, I
plan to go with that.

Matt


Re: Unexpected EOF handling

2020-05-07 Thread Matt Caswell



On 07/05/2020 20:28, Dmitry Belyavsky wrote:
> From my point of view, if we don't revert the change for the sake of API
> clarity, we need to provide an option restoring old behaviour at least
> for test purposes.

Presumably nginx can already handle the situation where a close_notify
*is* received. So if we have an option to behave as if that had occurred
in the event of eof then nginx should be able to handle it without
requiring special codepaths?? If so then we don't necessarily have to
have an option to restore the old behaviour - just an option to treat
eof like a close_notify.

Matt


> 
> On Thu, May 7, 2020 at 2:52 PM Tomas Mraz  > wrote:
> 
> On Thu, 2020-05-07 at 13:22 +0200, Kurt Roeckx wrote:
> > Hi,
> >
> > We introduced a change in the 1.1.1e release that changed how we
> > handled an unexpected EOF. This broke various software which
> > resulted in the release of 1.1.1f that reverted that change.
> > In master we still have the 1.1.1e behaviour.
> >
> > The change itself is small, it just calls SSLfatal() with a new
> > error code. This has at least 2 effects:
> > - The error code was changed from SSL_ERROR_SYSCALL with errno 0
> >   to SSL_ERROR_SSL with reason code
> >   SSL_R_UNEXPECTED_EOF_WHILE_READING
> > - The session is marked as in error, and so can't be used to
> >   resume.
> >
> > There is code written that now checks for the SSL_ERROR_SYSCALL
> > with errno 0 case, while we never documented that behaviour. The
> > 1.1.1 manpage of SSL_get_error() currently reports this as a bug
> > and that it will change to SSL_ERROR_SSL /
> > SSL_R_UNEXPECTED_EOF_WHILE_READING.
> >
> > Code that checks the SSL_ERROR_SYSCALL with errno 0 seem to fall
> > in at least 2 categories:
> > - Ignore the error
> > - Check for the error, for instance in a test suite
> >
> > Not sending the close notify alert is a violation of the TLS
> > specifications, but there is software out there doesn't send it,
> > so there is a need to be able to deal with peers that don't send
> > it.
> >
> > When the close notify alert wasn't received, but we did get an
> > EOF, there are 2 cases:
> > - It's a fatal error, the protocol needs the close notify alert to
> >   prevent a truncation attack
> > - The protocol running on top of SSL can detect a truncation
> >   attack itself, and does so. It doesn't need the close notify
> >   alert. It's not a fatal error. They just want to check that that
> >   is what happened.
> >
> > The problem is that we can't make a distiction between the 2
> > cases, so the only thing we can do currently is to look at
> > it as a fatal error. So the documentation was changed to say
> > that if you're in the 2nd cases, and need to talk to a peer
> > that doesn't send the close notify alert, you should not wait
> > for the close notify alert, so that you don't get an error.
> >
> > The alternative is that we add an option that does tell us if we
> > should look at it as a fatal error or not.
> >
> > There is at least a request that we keep returning the old error code
> > (SSL_ERROR_SYSCALL with errno 0). I think that from an API point
> > of view this is very confusing. We'd then have SSL_ERROR_SYSCALL
> > as documented that it's fatal, except when errno is 0. If errno is
> > 0, it can either be fatal or not, and we're not telling you which
> > one it is. I think we also can't guarantee that SSL_ERROR_SYSCALL
> > with errno 0 is always the unexpected EOF, we returned that
> > combination because of a bug in OpenSSL.
> >
> > So I think we need at least to agree on:
> > - Do we want an option that makes the unexpected EOF either a fatal
> >   error or a non-fatal error?
> > - Which error should we return?
> 
> From application developer side of view for protocols that do not
> depend on SSL detecting the truncation I think inventing a different
> behavior for this condition than the existing one as of 1.1.1f would be
> clearly wrong. Switching on a SSL_OP if that newly provided OP is a
> trivial change to an application that needs to accommodate various
> versions of OpenSSL (and I am not talking about forks), however
> switching on a SSL_OP and also add another special error case is much
> more complicated change and has potential for bringing regressions in
> the applications that need to do it.
> 
> It is also an API break, however we would do it anyway unless we make
> the legacy behavior the default one, so that is not really relevant
> here.
> 
> Is there really another situation where SSL_ERROR_SYSCALL with errno 0
> could be returned apart from the unclean EOF condition?
> 
> I can't really think of any.
> 
> So I would be just for properly 

Monthly Status Report (April)

2020-05-07 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Ongoing review work on the CMP contribution
- Fixed some issues with the XTS documentation
- Updated WPACKET to be able to do "end first" writing to support
DER_w_* functions
- Make X509_STORE_CTX libctx aware
- Updated the CT code to be library context aware
- Enabled export_to functions to have access to the libctx
- Made PrivateKey loading libctx aware
- Enabled Ed25519/Ed448 signing/verifying to be libctx aware
- Investigated and created a POC for CVE-2020-1967
- Made X509_verify() libctx aware
- PR to run sslapitest with the FIPS module
- PR to run ssl_test_new with the FIPS module
- Investigated and fixed issue on website where the scripts failed if we
only had one tarball
- PR to run ssl_test_old with the FIPS module
- Ensured calls to EC_POINT_point2buf use a libctx
- Ensure import_to functions pass a libctx
- Fixed an issue in libssl which resulted in no alert being sent even
though a fatal error occurred
- Wrote a wiki page about 3.0
- Performed the 1.1.1g release
- Fixed no-des
- Fixed no-ec
- Fixed no-dh and no-dsa
- Fixed no-deprecated tests when the GOST engine is present
- Fixed no-err
- Performed the alpha1 release
- Fixed ssl_test_old when SSLv3 is enabled
- Fixed typo in the makefile templates meaning that fips.so and
legacy.so were not being installed
- Fixed the raw provider key implementation
- Performed the 1.0.2v release for premium support customers
- Updated to the testsuite to centralise environment variable setting
and fix a problem with test_includes


Matt



Re: Technically an API break

2020-05-07 Thread Matt Caswell



On 07/05/2020 16:02, Brian Smith wrote:
> This kind of change might cause memory unsafety issues unless the
> application is recompiled. At least, it's worth investigating that.
> 
> On most platforms the ABI of a function that returns `void` and one that
> returns `int` is the same, from the perspective of a caller that doesn't
> expect or use the return value. I seem to vaguely remember in the past
> that there was at least one common platform where that isn't true
> though. Unfortunately I cannot remember which one it is. I also don't
> remember if it is problematic to change from "int" to "void" or "void"
> to "int" or both.
> 
> Anyway, my point is that you should consider this an ABI-breaking
> change, not just an API breaking one.

Yes - thanks for that Brian. Actually though this change is targeted
only at the master branch (which will become OpenSSL 3.0). That is a
major release and is already ABI breaking - so recompilation is already
a requirement. Actually as a major release we are allowed to be API
breaking too, but we are trying to keep that to a minimum.

Matt



Re: Unexpected EOF handling

2020-05-07 Thread Matt Caswell



On 07/05/2020 12:22, Kurt Roeckx wrote:
> So I think we need at least to agree on:
> - Do we want an option that makes the unexpected EOF either a fatal
>   error or a non-fatal error?
> - Which error should we return?

This is an excellent summary of the current situation.

I am not keen on maintaining the SSL_ERROR_SYSCALL with 0 errno as a
long term solution. It's a very confusing API for new applications to
use. Effectively it means SSL_ERROR_SYSCALL is a fatal error - except
when its not. SSL_ERROR_SYSCALL should mean fatal error.

That said I also recognise that it is apparently commonplace to shut
down a TLS connection without sending close_notify - despite what the
standards may say about it (and TBH we can hardly claim the moral high
ground here since s_server does exactly this - or at least it does in
1.1.1 and did until very recently in master).

But we do have to consider usages beyond HTTPS. I have no idea if this
occurs in other settings or not.

Perhaps what we need is an option for "strict shutdown". With strict
shutdown "off" we could treat EOF as if we had received a close_notify
gracefully (and don't invalidate the session). Presumably existing code
would be able to cope with that???

With strict shutdown "on" we treat it as SSL_ERROR_SSL (as now in master).

I'm not sure though what the default should be.

Matt


Technically an API break

2020-05-07 Thread Matt Caswell
PR11589 makes a change to the public API function
`SSL_set_record_padding_callback` to change its return type from void to
int:

https://github.com/openssl/openssl/pull/11589

This is technically an API break - but it doesn't seem too serious. It's
possible, I suppose, that existing applications that use this will fail
to spot the error return since this function can now fail. The function
itself was only recently added (in 1.1.1), and I suspect real-world
usage is very small (or possibly nil).

Is this considered ok?

Matt


Re: Stale PR stats @May01

2020-05-01 Thread Matt Caswell
This is really nice! Thanks Mark.

Matt

On 01/05/2020 08:52, Mark J Cox wrote:
> Last month I started a script to ping stale PRs that were in certain
> states.  The script has also been collecting statistics (trending and
> snapshot).  I intend to post this monthly and after a few months with
> trends and commentary.
> 
> PRs that have not had any updates in the last 30 days and are not WIP:
> 
> all  ( 148 issues, median  188  days) of which:
> 
>   failed CI  ( 23 issues, median  175  days)
>   deferred after 1.1.1  ( 41 issues, median  188  days)
>   cla required  ( 19 issues, median  422  days)
>   waiting for reporter  ( 15 issues, median  275  days)
>   waiting for review  ( 1 issues, median  38  days)
>   waiting for OMC  ( 2 issues, median  112.0  days)
>   waiting for OTC  ( 1 issues, median  84  days)
>   all other  ( 46 issues, median  239.5  days)
> 
> So, ignoring deferred issues too you could summarise this as:
> 
> Stale PRs waiting for us to do something: 27 (last months: 29)
> Stale PRs waiting for the reporter to do something: 34 (last months: 37)
> Stale PRs with unclear next action: 46 (last months: 42)
> 
> Over time I hope we can triage the "all other" PRs with labels so we
> know what the next action is on each one.
> 
> Details:
> 
> failed CI  ( 23 issues, median  175  days)
> 
> 11349 branch: master, reviewed:commented days:43
> 11327 reviewed:commented days:48
> 11288 branch: master, reviewed:commented days:51
> 11257 branch: 1.1.1, branch: master, reviewed:approved days:49
> 11195 branch: 1.1.1, branch: master, reviewed:commented days:38
> 11151  days:60
> 10626  days:66
> 10556  days:147
> 10465  days:164
> 9926  days:226
> 9603  days:226
> 9155 reviewed:commented days:100
> 8955 branch: 1.1.1, branch: master, reviewed:dismissed days:175
> 8687 reviewed:commented days:210
> 8389  days:415
> 8115 reviewed:commented days:48
> 7921 reviewed:commented days:419
> 7914 reviewed:approved days:478
> 7380 reviewed:commented days:566
> 7051 milestone:Assessed, reviewed:commented days:615
> 6074 milestone:Assessed, reviewed:commented days:658
> 4992 milestone:Assessed, reviewed:commented days:188
> 4486 milestone:Assessed,  days:658
> 
> waiting for reporter  ( 15 issues, median  275  days)
> 
> 11310 reviewed:changes_requested days:43
> 10887 branch: master, reviewed:changes_requested days:37
> 10590 reviewed:changes_requested days:115
> 9677 branch: master, reviewed:changes_requested days:32
> 9575 reviewed:changes_requested days:262
> 9461 reviewed:changes_requested days:275
> 9427 reviewed:changes_requested days:282
> 9243 reviewed:changes_requested days:284
> 9240 reviewed:changes_requested days:309
> 8992 reviewed:changes_requested days:226
> 8962 reviewed:changes_requested days:44
> 8730 reviewed:changes_requested days:323
> 8674 reviewed:changes_requested days:378
> 7961 reviewed:changes_requested days:480
> 7432 reviewed:changes_requested days:559
> 
> waiting for OMC  ( 2 issues, median  112.0  days)
> 
> 10786 approval: done, branch: 1.1.1, branch: master, hold: need
> omc decision, reviewed:approved days:52
> 10195 branch: master, hold: need omc decision, reviewed:commented days:172
> 
> waiting for OTC  ( 1 issues, median  84  days)
> 
> 8300 approval: otc review pending, branch: 1.1.1, branch: master,
> hold: need otc decision, reviewed:approved days:84
> 
> waiting for review  ( 1 issues, median  38  days)
> 
> 10587 approval: review pending, branch: 1.1.1, branch: master,
> reviewed:commented days:38
> 
> all other  ( 46 issues, median  239.5  days)
> 
> 11398 approval: done, branch: master, reviewed:approved days:35
> 11312  days:47
> 11277 resolved: not a bug,  days:50
> 11260 reviewed:commented days:55
> 11154  days:66
> 5  days:56
> 11057  days:80
> 10884  days:102
> 10818  days:108
> 10570 reviewed:commented days:124
> 10541  days:139
> 10338 reviewed:commented days:179
> 10320 branch: 1.1.1, branch: master, reviewed:commented days:170
> 10298  days:183
> 10268  days:185
> 10037  days:147
> 9655  days:157
> 9554  days:266
> 9421 branch: 1.1.1, branch: master, reviewed:approved days:121
> 9223 branch: master, reviewed:commented days:83
> 9206  days:314
> 9051 reviewed:commented days:177
> 8956  days:336
> 8920  days:213
> 8908  days:351
> 8862  days:330
> 8835  days:370
> 8743 branch: master,  days:381
> 8668  days:392
> 8455  days:394
> 8420  days:395
> 8333  days:430
> 8309 branch: master, reviewed:commented days:316
> 8024 reviewed:commented days:81
> 7688  days:516
> 7615  days:520
> 7454 reviewed:commented days:471
> 7274 reviewed:approved days:322
> 7225 reviewed:commented days:587
> 6725 milestone:Assessed, 

Cherry-pick proposal

2020-04-29 Thread Matt Caswell


The OTC have received this proposal and a request that we vote on it:

I would like to request that we do not allow cherry-picks between master
and 1.1.1-stable because these two versions are now very different, if a
cherry-pick succeeds, there is no guarantee that the result will work.
Because we have no CI for the cherry-pick. If a cherry-pick is needed, a
new PR for 1.1.1 should be done and approved independently.

Before starting a vote I'd like to provide opportunity for comments, and
also what the vote text should be.

Thanks

Matt


Re: Alpha1

2020-04-23 Thread Matt Caswell



On 22/04/2020 13:53, Matt Caswell wrote:
> 
> 
> On 22/04/2020 02:46, Benjamin Kaduk wrote:
>> On Tue, Apr 21, 2020 at 11:10:19AM +0100, Matt Caswell wrote:
>>> The 3.0 developers met via conference call this morning. All the
>>> functionality that we had planned for alpha 1 has now been merged, so we
>>> are now thinking that we will do the alpha 1 release on Thursday this
>>> week. That would imply a repo freeze tomorrow.
>>>
>>> Thoughts/opinions/objections to this proposal?
>>
>> Given that the list of required things for alpha 1 are done, it does seem
>> appropriate.  I know of a couple things that would be bug reports against
>> an alpha1 if produced right now, but ... what is an alpha for, if not to
>> trigger people to look and file bug reports? :)
> 
> I've seen no objections and everyone seems to be assuming this is
> happening, so the repo is now frozen ready for the release tomorrow.

The alpha1 release is now done!!

Thanks to everyone that has helped to make this happen.

We had a number of issues during the release itself - which is not
surprising given this is the first of the 3.0 series - but nothing too
significant.

Thanks to Richard for helping out during the release.

Matt



Re: 3.0 wiki page

2020-04-23 Thread Matt Caswell
Fantastic to see that 6 different authors have made contributions to
this page!

Matt

On 22/04/2020 17:31, Matt Caswell wrote:
> The 3.0 wiki page is starting to look good:
> 
> https://wiki.openssl.org/index.php/OpenSSL_3.0
> 
> I'd appreciate it if people could give it a read through before the
> alpha 1 release tomorrow and make any amendments or corrections as
> appropriate.
> 
> Thanks!
> 
> Matt
> 


3.0 wiki page

2020-04-22 Thread Matt Caswell
The 3.0 wiki page is starting to look good:

https://wiki.openssl.org/index.php/OpenSSL_3.0

I'd appreciate it if people could give it a read through before the
alpha 1 release tomorrow and make any amendments or corrections as
appropriate.

Thanks!

Matt


Re: Alpha1

2020-04-22 Thread Matt Caswell



On 22/04/2020 02:46, Benjamin Kaduk wrote:
> On Tue, Apr 21, 2020 at 11:10:19AM +0100, Matt Caswell wrote:
>> The 3.0 developers met via conference call this morning. All the
>> functionality that we had planned for alpha 1 has now been merged, so we
>> are now thinking that we will do the alpha 1 release on Thursday this
>> week. That would imply a repo freeze tomorrow.
>>
>> Thoughts/opinions/objections to this proposal?
> 
> Given that the list of required things for alpha 1 are done, it does seem
> appropriate.  I know of a couple things that would be bug reports against
> an alpha1 if produced right now, but ... what is an alpha for, if not to
> trigger people to look and file bug reports? :)

I've seen no objections and everyone seems to be assuming this is
happening, so the repo is now frozen ready for the release tomorrow.

Matt



Re: Repo is frozen

2020-04-21 Thread Matt Caswell
Repo is now unfrozen. I'm planning on freezing it again tomorrow, ready
for the alpha1 release on Thursday.

Matt


On 21/04/2020 10:23, Matt Caswell wrote:
> FYI, the repo is currently frozen in preparation for the release today.
> I'll let you know when its unfrozen again.
> 
> Matt
> 


Alpha1

2020-04-21 Thread Matt Caswell
The 3.0 developers met via conference call this morning. All the
functionality that we had planned for alpha 1 has now been merged, so we
are now thinking that we will do the alpha 1 release on Thursday this
week. That would imply a repo freeze tomorrow.

Thoughts/opinions/objections to this proposal?

Matt



Repo is frozen

2020-04-21 Thread Matt Caswell
FYI, the repo is currently frozen in preparation for the release today.
I'll let you know when its unfrozen again.

Matt


Forthcoming OpenSSL Release

2020-04-14 Thread Matt Caswell
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

The OpenSSL project team would like to announce the forthcoming release
of OpenSSL version 1.1.1g.

This release will be made available on Tuesday 21st April 2020 between
1300-1700 UTC.

OpenSSL 1.1.g is a security-fix release. The highest severity issue
fixed in this release is HIGH:
https://www.openssl.org/policies/secpolicy.html#high

Yours

The OpenSSL Project Team
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAl6VuVwACgkQ2cTSbQ5g
RJEGGwgAnvbo6LVTEz8PdAOoKPgHiz1ObbB8M/fNANk1Oog1w6CF7a8JPEuB/LlQ
ZS0/31x+69xE+GzD4kPBglG6IVnt7F1mlXSc1YEh5c5zs2T5w5Gak5AIzJNZqEFK
EmplFS8eZCpKJZc+0YKgMisF4Q+VbRjI+KVtYQKBn3sHRNH04z4Ti6jlS14R4pQd
PCB4ftXS/LnISkrxL1uVf1seY+5SpmQjk3FR8ZgrR3vuYAyLcD7aeQNKf+unsS4W
u8VnDmqONHa2JfHjsr5PezLZfWa3YTvK352gamyq5sn6y2ciTcI+fABeSD4OYjvQ
I6t4kQrzfCdMrBNY8G2D5NYOi5cOKQ==
=5CII
-END PGP SIGNATURE-


Re: Revisiting tradition: branches and tags

2020-04-13 Thread Matt Caswell



On 11/04/2020 10:53, Dr. Matthias St. Pierre wrote:
> I love the new naming scheme, in particular the fact that it's all-lowercase 
> and does not
> mix dashes and underscores anymore. I don't recall how often I cursed about 
> the current
> scheme which is so typer unfriendly.
> 
> I'd like to see *all* stable branch names and release tags converted to 
> new-style uniformly
> (keeping the old names for compatibility), so we have an overall consistent 
> scheme and people
> don't need to switch back and forth between naming conventions in the future 
> when doing
> historical investigations. The old names could be deprecated for later 
> removal.

Yes - this aspect was my main hesitation about the proposed new format,
i.e. the fact that we have a set of existing tags/branch names.
Constantly changing between the new format and the old format as we look
at older branches/tags could be painful. Just creating new tags for all
the existing ones would be one way forward.

Typing OpenSSL_1_1_1-stable and getting all the right upper/lower case
characters in the right place and making sure to use _ vs - as
appropriate is a challenge for my fingers which I constantly fail.

Is it possible to set up alias names for the historical branches?

Matt


> 
> Matthias
> 
> 
> 
>> -Original Message-
>> From: openssl-project  On Behalf Of 
>> Richard Levitte
>> Sent: Friday, April 10, 2020 8:28 PM
>> To: openssl-project@openssl.org
>> Subject: Revisiting tradition: branches and tags
>>
>> Once upon a time, there was CVS (*).
>>
>> The story could stop there, but since CVS was what was available,
>> OpenSSL was versioned with CVS.
>>
>> Among limitations that came with CVS was the allowed syntax in tag and
>> branch names; letters, digits, dashes and underscores.  At the time,
>> eveyone that wanted to encode a version number in a tag had to convert
>> periods to some other character.  We chose underscores, ending up with
>> tags like this:
>>
>> OpenSSL_0_9_7-beta2
>> OpenSSL_0_9_7a
>>
>> Furthermore, the culture that we have with git, where branches are
>> being pulled between repositories...  where you can actually have a
>> multitude of repositories and pull data between them, was very hard if
>> not practically impossible with CVS.  So, we occasionally had feature
>> branches for longer term work.  To distinguish our so called stable
>> branches, we had branch names with '-stable' added at the end:
>>
>> OpenSSL_0_9_7-stable
>>
>> We *could* have had something shorter, like 'OpenSSL_0_9_7xx', but
>> I guess that was too easy to mix up with our letter releases.
>>
>> With git, however, there's no technical reason why we must maintain
>> what was originally a technical limitation.  I therefore propose that
>> we start using tags like this starting with OpenSSL 3.0:
>>
>> openssl-3.0.0-alpha1
>> openssl-3.0.0-beta2
>> openssl-3.0.0
>> openssl-3.0.1
>> openssl-3.1.0
>>
>> This is a little more than just avoiding to change the periods with
>> underscores.  However, if you look at the tar files we've released for
>> a long time, that's the naming format they use, and I can see benefits
>> in having the tags for a release match the tar file of the same
>> release:
>>
>> openssl-0.9.7a.tar.gz
>> openssl-0.9.7a.tar.gz.asc
>> openssl-0.9.7a.tar.gz.md5
>>
>> Future tar files would be numbered with the new version scheme, of
>> course:
>>
>> openssl-3.0.0-alpha1.tar.gz
>> openssl-3.0.0-beta2.tar.gz
>> openssl-3.0.0.tar.gz
>> openssl-3.0.1.tar.gz
>> openssl-3.1.0.tar.gz
>>
>> So what about release branches?  We could actually follow a very
>> similar new pattern, just replace the last digit with, say, an 'x', to
>> signify that this branch will contain a series of patch releases:
>>
>> openssl-3.0.x
>>
>> In this branch, one would expect to see the tags 'openssl-3.0.0',
>> 'openssl-3.0.1', 'openssl-3.0.2', ...
>>
>> Earlier today, I submitted a new release script that codifies exactly
>> this:  https://github.com/openssl/openssl/pull/11516
>>
>> Thoughts?
>>
>> Cheers,
>> Richard
>>
>> (*) Well, not quite, there was RCS, there was SCCS, and a few other
>> versioning systems that would make your skin crawl by today's
>> standards, even more so than CVS.
>>
>> --
>> Richard Levitte levi...@openssl.org
>> OpenSSL Project http://www.openssl.org/~levitte/
> 


OpenSSL 3.0 and FIPS talk

2020-04-08 Thread Matt Caswell


I recently gave a talk at the RSA Conference with Ashit Vora from Acumen
Security about OpenSSL 3.0 and FIPS. The conference have just posted the
audio and slides here if you are interested:

https://www.rsaconference.com/usa/agenda/openssl-and-fips-they-are-back-together

Matt


Monthly Status Report (March)

2020-04-03 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Ongoing reviews of the CMP contribution
- Clarified the docs around usage of EVP_PKEY_get_raw_*_key()
- Provided some tweaks/fixes to the Serializer code
- Completed implementation of Ed25519 and Ed448 in the default provider
- Implemented serializers for Ed25519 and Ed448
- Performed and coordinated the release of both 1.1.1e and 1.1.1f
- Fix to handle the case where there is no digest in an EVP_MD_CTX
- Significant effort in getting a simple TLSv1.2 connection working with
FIPS only crypto
- Created PR to make various updates to provider.pod
- Made it possible to easily specify a libctx from EVP_DigestSign*
- Made sure we were using the correct libctx when fetching a MAC in one
scenario
- Ensured we were using RAND_bytes_ex in various calls in crypto/rsa
- Ensured we were using fetched ciphers/digests for TLS tickets
- Fixed a number of spots in libssl where we weren't using the libctx
- Fixed EVP_PKEY_new_mac_key() so that it doesn't fail if the specified
MAC is not available in the default provider
- Wrote code to update libssl to use EVP_MAC for its MAC rather than
EVP_DigestSign*(). This work is currently on hold due to an unexpected
impact on the GOST engine
- Fixed more spots in libssl where fetched ciphers were not being used
- Update to provide better diagnostics in the event of a fetch failure
- Updated test TLS framework to provide better error information if a
connection fails
- Added libctx aware functions OCSP_RESPID_set_by_key_ex() and
OCSP_RESPID_match_ex()
- Added function to explicitly cache X509v3 extensions with a libctx -
and used that function in libssl
- Made the SRP library libctx aware, and updated libssl to use the new
functions
- Updated libssl to give a better error if we can't find a sig alg
- Fixed a bug in libssl to avoid attempting to up-ref a cipher that is NULL
- Fixed a bug to avoid double freeing a DH object in libssl


Matt


Critical Path and Dependencies for Alpha1

2020-03-31 Thread Matt Caswell
Please see attached for what I believe is the critical path as well as
the key dependencies for Alpha 1. Please let me know of any errors or
omissions.

Matt


Forthcoming OpenSSL Release

2020-03-28 Thread Matt Caswell
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

The OpenSSL project team would like to announce the forthcoming release
of OpenSSL version 1.1.1f.

This release will be made available on Tuesday 31st March 2020 between
1200-1600 UTC. This is a bug fix only release.

Yours

The OpenSSL Project Team
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAl5/MPUACgkQ2cTSbQ5g
RJGnaggAjtB2r56ufZaOUAy7/stpy+Cj7R4Jq+RZb8Ja6c9hU9FwHx5/eESxs1lC
XQKr5RGcPZbIvgoDaFCBVXBswl6Ivhde/MuWLoeoag+sl4TBztx/Aash6YAT78ij
h/NvRcYDn2mcBrclxJckh9sags5ei13d+GWug349X8d7dVdfHooFTBgq0Th4ehfZ
UBaNgQTnqnd/8PD2paGkQtHOr8Qr2TTPH6HyQ5Vlea+x0AzjnAbWjbr/wvu0yuFE
2RqE6RnVy65M+Nx1wIXh1ZJT0EfyN4lqRFYuTWViJVPfPDT61UkIKSbxzRtVWEl8
Pu4T2r9cKHl8kFnuA0kqc0/5/jG2EQ==
=KWO3
-END PGP SIGNATURE-


Re: 1.1.1f

2020-03-27 Thread Matt Caswell
There seems to be broad support for a 1.1.1f release. Unless I hear an
OMC objection I will formally announce this tomorrow.

Matt

On 27/03/2020 00:10, Viktor Dukhovni wrote:
> On Thu, Mar 26, 2020 at 11:33:40PM +0000, Matt Caswell wrote:
> 
>> On 26/03/2020 23:15, Viktor Dukhovni wrote:
>>> On Thu, Mar 26, 2020 at 09:13:32PM +0100, Bernd Edlinger wrote:
>>>
>>>> we got into this situation because everything moves so quickly,
>>>> why does everyone here think we should move even faster now?
>>>>
>>>> What is the reason for this?
>>>
>>> We've published a bug-fix release (1.1.1e) that's liable to cause more
>>> problems than it fixes.  In such cases a closely-timed "fixup" (oops)
>>> release is expected.  One that only reverts the (two) problem
>>> EOF-handling commits.
>>
>> Actually a partial revert of one of them is sufficient to resolve the
>> problem.
> 
> Yes, probably so.  I took a sledge-hammer to the problem, and since the
> second commit depended on the first, I reverted both.  If you leave
> the pre-requisites for the second commit in place, and just remove
> the changed error handling, then indeed that may also work.
> 


Re: 1.1.1f

2020-03-26 Thread Matt Caswell



On 26/03/2020 23:15, Viktor Dukhovni wrote:
> On Thu, Mar 26, 2020 at 09:13:32PM +0100, Bernd Edlinger wrote:
> 
>> we got into this situation because everything moves so quickly,
>> why does everyone here think we should move even faster now?
>>
>> What is the reason for this?
> 
> We've published a bug-fix release (1.1.1e) that's liable to cause more
> problems than it fixes.  In such cases a closely-timed "fixup" (oops)
> release is expected.  One that only reverts the (two) problem
> EOF-handling commits.

Actually a partial revert of one of them is sufficient to resolve the
problem.

Matt

>  Further bug-fixes can be queued for later
> releases, or deferred to a major release as appropriate.
> 


Re: 1.1.1f

2020-03-26 Thread Matt Caswell



On 26/03/2020 20:13, Bernd Edlinger wrote:
> 
> 
> On 3/26/20 9:10 PM, Tim Hudson wrote:
>> We don't guarantee constant time.
>>
> 
> #11411 does, I don't see why we hurry so much for 1.1.1f
> 
> we got into this situation because everything moves so quickly,

We waited 6 months to release 1.1.1e. This issue wasn't caused by moving
too quickly.

Matt


> why does everyone here think we should move even faster now?
> 
> What is the reason for this?
> 
> Bernd.
> 
>> Tim.
>>
>> On Fri, 27 Mar 2020, 5:41 am Bernd Edlinger, 
>> wrote:
>>
>>> So I disagree, it is a bug when it is not constant time.
>>>
>>>
>>> On 3/26/20 8:26 PM, Tim Hudson wrote:
>>>> +1 for a release - and soon - and without bundling any more changes. The
>>>> circumstances justify getting this fix out. But I also think we need to
>>>> keep improvements that aren't bug fixes out of stable branches.
>>>>
>>>> Tim.
>>>>
>>>> On Fri, 27 Mar 2020, 3:12 am Matt Caswell,  wrote:
>>>>
>>>>> On 26/03/2020 15:14, Short, Todd wrote:
>>>>>> This type of API-braking change should be reserved for something like
>>>>>> 3.0, not a patch release.
>>>>>>
>>>>>> Despite it being a "incorrect", it is expected behavior.
>>>>>>
>>>>>
>>>>> Right - but the question now is not whether we should revert it (it has
>>>>> been reverted) - but whether this should trigger a 1.1.1f release soon?
>>>>>
>>>>> Matt
>>>>>
>>>>>> --
>>>>>> -Todd Short
>>>>>> // tsh...@akamai.com <mailto:tsh...@akamai.com>
>>>>>> // “One if by land, two if by sea, three if by the Internet."
>>>>>>
>>>>>>> On Mar 26, 2020, at 11:03 AM, Dr. Matthias St. Pierre
>>>>>>> mailto:matthias.st.pie...@ncp-e.com>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I agree, go ahead.
>>>>>>>
>>>>>>> Please also consider reverting the change for the 3.0 alpha release as
>>>>>>> well, see Daniel Stenbergs comment
>>>>>>>
>>> https://github.com/openssl/openssl/issues/11378#issuecomment-603730581
>>>>>>> <
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_issues_11378-23issuecomment-2D603730581=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=djWoIIXyggxwOfbwrmYGrSJdR5tWm06IdzY9x9tDxkA=
>>>>>>
>>>>>>>
>>>>>>> Matthias
>>>>>>>
>>>>>>>
>>>>>>> *From**:* openssl-project >>>>>> <mailto:openssl-project-boun...@openssl.org>> *On Behalf Of *Dmitry
>>>>>>> Belyavsky
>>>>>>> *Sent:* Thursday, March 26, 2020 3:48 PM
>>>>>>> *To:* Matt Caswell mailto:m...@openssl.org>>
>>>>>>> *Cc:* openssl-project@openssl.org <mailto:openssl-project@openssl.org
>>>>
>>>>>>> *Subject:* Re: 1.1.1f
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 26, 2020 at 5:14 PM Matt Caswell >>>>>> <mailto:m...@openssl.org>> wrote:
>>>>>>>
>>>>>>> The EOF issue (https://github.com/openssl/openssl/issues/11378
>>>>>>> <
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_issues_11378=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=MAiLjfGJWaKvnBvqnM4fcyvGVfUyj9CDANO_vh4wfco=
>>>>>> )
>>>>>>> has
>>>>>>> resulted in us reverting the original EOF change in the 1.1.1
>>> branch
>>>>>>> (https://github.com/openssl/openssl/pull/11400
>>>>>>> <
>>>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_pull_11400=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=3hBU2pt84DQlrY1dCnSn9x1ah1gSzH6NEO_bNRH-6DE=
>>>>>> ).
>>>>>>>
>>>>>>> Given that this seems to have broken quite a bit of stuff, I
>>> propose
>>>>>>> that we do a 1.1.1f soon (possibly next Tuesday - 31st March).
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>>
>>>>>>> I strongly support this idea.
>>>>>>>
>>>>>>> --
>>>>>>> SY, Dmitry Belyavsky
>>>>>>
>>>>>
>>>>
>>>
>>
> 


Re: 1.1.1f

2020-03-26 Thread Matt Caswell
On 26/03/2020 15:14, Short, Todd wrote:
> This type of API-braking change should be reserved for something like
> 3.0, not a patch release.
> 
> Despite it being a "incorrect", it is expected behavior.
> 

Right - but the question now is not whether we should revert it (it has
been reverted) - but whether this should trigger a 1.1.1f release soon?

Matt

> --
> -Todd Short
> // tsh...@akamai.com <mailto:tsh...@akamai.com>
> // “One if by land, two if by sea, three if by the Internet."
> 
>> On Mar 26, 2020, at 11:03 AM, Dr. Matthias St. Pierre
>> mailto:matthias.st.pie...@ncp-e.com>>
>> wrote:
>>
>> I agree, go ahead. 
>>  
>> Please also consider reverting the change for the 3.0 alpha release as
>> well, see Daniel Stenbergs comment
>> https://github.com/openssl/openssl/issues/11378#issuecomment-603730581
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_issues_11378-23issuecomment-2D603730581=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=djWoIIXyggxwOfbwrmYGrSJdR5tWm06IdzY9x9tDxkA=>
>>  
>> Matthias
>>  
>>  
>> *From**:* openssl-project > <mailto:openssl-project-boun...@openssl.org>> *On Behalf Of *Dmitry
>> Belyavsky
>> *Sent:* Thursday, March 26, 2020 3:48 PM
>> *To:* Matt Caswell mailto:m...@openssl.org>>
>> *Cc:* openssl-project@openssl.org <mailto:openssl-project@openssl.org>
>> *Subject:* Re: 1.1.1f
>>  
>>  
>> On Thu, Mar 26, 2020 at 5:14 PM Matt Caswell > <mailto:m...@openssl.org>> wrote:
>>
>> The EOF issue (https://github.com/openssl/openssl/issues/11378
>> 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_issues_11378=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=MAiLjfGJWaKvnBvqnM4fcyvGVfUyj9CDANO_vh4wfco=>)
>> has
>> resulted in us reverting the original EOF change in the 1.1.1 branch
>> (https://github.com/openssl/openssl/pull/11400
>> 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openssl_openssl_pull_11400=DwMGaQ=96ZbZZcaMF4w0F4jpN6LZg=QBEcQsqoUDdk1Q26CzlzNPPUkKYWIh1LYsiHAwmtRik=87AtfQDFl1z9cdRP12QeRUizmgnW6ejbufNT40Gip4Q=3hBU2pt84DQlrY1dCnSn9x1ah1gSzH6NEO_bNRH-6DE=>).
>>
>> Given that this seems to have broken quite a bit of stuff, I propose
>> that we do a 1.1.1f soon (possibly next Tuesday - 31st March).
>>
>> Thoughts?
>>
>>  
>> I strongly support this idea.
>>  
>> -- 
>> SY, Dmitry Belyavsky
> 


1.1.1f

2020-03-26 Thread Matt Caswell
The EOF issue (https://github.com/openssl/openssl/issues/11378) has
resulted in us reverting the original EOF change in the 1.1.1 branch
(https://github.com/openssl/openssl/pull/11400).

Given that this seems to have broken quite a bit of stuff, I propose
that we do a 1.1.1f soon (possibly next Tuesday - 31st March).

Thoughts?

Matt


Alpha1 progress

2020-03-25 Thread Matt Caswell
Yesterday a number of us had a teleconference to update our task
tracking for the 3.0 release. The current spreadsheet gives us the
following dates for the various alpha/beta releases:

Alpha1: 2020-04-15
Alpha2: 2020-05-04
Alpha3: 2020-06-10
Beta1:  2020-06-12

Comparing this to the official timeline here:
https://www.openssl.org/policies/releasestrat.html

Which says:

Alpha1: 2020-03-31
Alpha2: 2020-04-21
Alpha3: 2020-05-21
Beta1: 2020-06-02

Until quite recently we were tracking fairly closely to the target
dates, but the last week or so has seen us drift out a bit. As can be
seen from the above we're about 2 weeks out at the moment. This is
primarily due to the key generation work being more complicated and
significant than we had anticipated.

So, right now, it looks to me like we won't be releasing alpha1 next
week as originally planned.

Matt



Re: Release done

2020-03-17 Thread Matt Caswell



On 17/03/2020 18:53, Dr. Matthias St. Pierre wrote:
>> Problems were due to:
>> ...
>> - Rename of CHANGES/NEWS to CHANGES.md/NEWS.md
> 
> Strange  what exactly was your problem?

The website is supposed to update itself automatically when we push
stuff, so things like release notes etc get automatically generated.
>From tools/release/README.md:

Verify that the release notes, which are built from the CHANGES file
in the release, have been updated. This is done automatically by the
commit-hook, but if you see a problem, try the following steps:

cd /var/www/openssl
sudo -u openssl -H make relupd
sudo -u openssl ./bin/purge-one-hour

This didn't happen unfortunately. When I ran "make relupd" manually I got:

make: *** No rule to make target
'/var/cache/openssl/checkouts/openssl/CHANGES', needed by
'news/changelog.txt'.  Stop.

That particular make target is related to the *master* CHANGES file
(there are similar targets I think for other branches).

Matt


Release done

2020-03-17 Thread Matt Caswell
The release is complete and the repo is unfrozen. Thanks to Paul Yang
for doing the release reviewer role.

Loads of problems this time. Looks like the release scripts need some
adjustment. But through a combination of hacking and swearing at my
computer it went through in the end.

Problems were due to:
- ongoing bug in dorelease.pl which doesn't move the old releases out
the way properly
- Rename of CHANGES/NEWS to CHANGES.md/NEWS.md
- Bug in the script to update the source directory (doesn't seem to like
us only having one active release)
- Some problem (on our side) in the scripts related to purging the CDN
cache. As a consequence I had to purge things "manually" as I found
them, so there may be some pages that are un-purged (in particular I'm
thinking the man pages).

Matt


Re: Tools repo ownership

2020-03-16 Thread Matt Caswell
Sorry - I forgot about this vote. It passed:

+1: 6
 0: 0
-1: 0
No vote: 1

Matt

On 19/02/2020 15:06, Matt Caswell wrote:
> Following the discussion in PR 58 [1] about the ownership of the tools
> repo, I've started an OMC vote to see the tools repo split into "tools"
> and "omc-tools", with some tools moving to omc-tools. OMC would have
> commit/approval rights for omc-tools, and the OTC would have
> commit/approval rights for tools.
> 
> I'll report back here once the vote is complete.
> 
> Matt
> 
> [1] https://github.com/openssl/tools/pull/58
> 


Forthcoming OpenSSL release

2020-03-11 Thread Matt Caswell
The OpenSSL project team would like to announce the forthcoming release
of OpenSSL version 1.1.1e.

This release will be made available on Tuesday 17th March 2020 between
1300-1700 UTC. This will contain one LOW severity fix for CVE-2019-1551
previously announced here:
https://www.openssl.org/news/secadv/20191206.txt

Please see the following page for further details of severity levels:
https://www.openssl.org/policies/secpolicy.html

Yours

The OpenSSL Project Team


Monthly Status Report (February)

2020-03-10 Thread Matt Caswell
As well as normal reviews, responding to user queries, wiki user
requests, OMC business, handling security reports, etc., key activities
this month:

- Fixed no-ec builds
- Fixed no-multiblock
- Completed work on moving X25519/X448 to the default provider
- Fixed no-tls1_3
- Fixed no-sm2
- Completed work making libssl provider aware
- Fixed issue with running GOST engine in a no-deprecated build
- Fixed issue where we were attempting to compile AESNI code even if we
weren't AESNI capable
- Completed work to make RSA_ASYM_CIPHER implementation available inside
the FIPS provider
- Fixed no-engine builds
- Attended the RSA Conference 2020
- Prepared for and gave a talk at RSA Conference on OpenSSL and FIPS
- Fixed no-des
- Fixed a mem leak in libssl
- Implemented serializers for X25519/X448
- Introduced the "provider" property
- Contributed to and published the QUIC blog
- Added all the *.d.tmp file to .gitignore


Matt


Re: An OpenSSL cookbook, where and how?

2020-03-09 Thread Matt Caswell



On 07/03/2020 09:01, Richard Levitte wrote:
> As part of a documentation effort, more specifically in some
> discussions in https://github.com/openssl/openssl/pull/11220 (exact
> references below), I've floated the idea that bigger coding examples
> should be placed into a cookbook.
> 
> My reasoning for this is very simple: example code in reference
> manuals should be kept minimal and focused on the functions documented
> in that same page.  Anything that start involving too much other
> functions becomes a distraction, or will leave you wondering why the
> example is in *this* page and not *the other page* that describes
> those other functions.  In my mind, this is education 101.
> 
> However, it's true that people may want to see more complete examples,
> where the interaction between different sets of functions is on
> display, and could serve as code to pick and use, more than the quick
> minimalistic examples.  A cookbook.
> 
> So if we should do this, where do we want that cookbook?  Who keeps it
> up to date, and how?  https://wiki.openssl.org/ could be one answer,
> but is it the answer we want?

I agree we do need to provide this broader "how to" type documentation.
It's a major failing of our current documentation that we don't provide it.

Both the wiki and ossl-book were previous attempts where we've tried to
collect this kind of information in one place.

IMO, for now, the wiki is the best place to put this kind of thing. In
the long term I still dream of getting ossl-book off the ground.


Matt


Re: Deprecations

2020-03-04 Thread Matt Caswell



On 04/03/2020 19:15, Kurt Roeckx wrote:
> On Mon, Mar 02, 2020 at 11:41:57AM +0000, Matt Caswell wrote:
>>
>>
>> On 28/02/2020 23:43, Dr Paul Dale wrote:
>>> Any suggestions for a consensus on this thread?
>>
>> I think we can probably agree that:
>>
>> - Command option deprecations should be handled better
>> - We should look at whether we can resurrect some of the "old" commands
>> (possibly by writing them as wrappers for genpkey, pkey and pkeyutl)
> 
> What about at least pointing to the alternative function in the
> documentation?

Yes, we should do that. Pauli is currently investigating the possibility
of rewriting the old commands using EVP:

https://github.com/openssl/openssl/pull/11225

Matt


Re: Should FIPS_mode() and FIPS_mode_set() be updated, deprecated, or completely removed..

2020-03-04 Thread Matt Caswell



On 04/03/2020 08:15, Tomas Mraz wrote:
> The current implementation in the PR - unconditionally returning error
> - is completely useless. It would make some (not much) sense if we
> aimed for ABI compatibility with 3.0 however that is definitely not the
> case so the only reasonable thing is to either try to emulate the
> behavior as much as possible or remove completely so the users of the
> API would know immediately that they have to be changed.
> 

I don't have a strong view, but I think I tend to agree that removal is
the better option. 3.0 *is* a major release and we've never guaranteed
that there will be *no* breaking changes at all. We've only aimed for
most applications that work in 1.1.1 not having to change. Since these
functions exist in 1.1.1, but always fail, it seems reasonable to think
that very few 1.1.1 application actually call them. If there are any
that do so, then they probably need to re-examine their code anyway to
confirm what the behaviour should be with 3.0.

If we remove them then we should have some good documentation somewhere
that explains what people should do instead.

I also think this will probably need an OMC vote.

Matt


Re: Deprecations

2020-03-02 Thread Matt Caswell



On 28/02/2020 23:43, Dr Paul Dale wrote:
> Any suggestions for a consensus on this thread?

I think we can probably agree that:

- Command option deprecations should be handled better
- We should look at whether we can resurrect some of the "old" commands
(possibly by writing them as wrappers for genpkey, pkey and pkeyutl)

I am slightly concerned that the latter option (rewriting as wrappers)
may turn into a big black hole of effort. It *might* be easier to just
rewrite them as-is to use EVP. Whichever approach we take, I don't think
this should be a goal for alpha1.

Matt

> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> 
> 
> 
>> On 24 Feb 2020, at 5:08 pm, Dr Paul Dale > > wrote:
>>
>> Most of the conversions to using PKEY were straightforward.  One
>> didn’t require any changes (dsa but my memory is suspect).  One seemed
>> quite difficult.  Some I didn’t check.
>>
>> Modifying the commands so that they continue to work and print (to
>> stderr) an alternative pkey based command might be workable too.
>>
>>
>> Pauli
>> -- 
>> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
>> Phone +61 7 3031 7217
>> Oracle Australia
>>
>>
>>
>>
>>> On 24 Feb 2020, at 5:53 am, Viktor Dukhovni
>>> mailto:openssl-us...@dukhovni.org>> wrote:
>>>
 On Feb 22, 2020, at 4:53 AM, Richard Levitte >>> > wrote:

 Something that could be done is to take all those aged commands and
 rewrite them as wrappers for genpkey, pkey and pkeyutl.  Simply create
 and populate a new argv and call genpkey_main(), pkey_main() or
 pkeyutl_main().
>>>
>>> Agreed, that sounds quite reasonable at first blush, and could be
>>> fantastic
>>> if it can be made to work (no immediate obstacles come to mind).
>>>
>>> -- 
>>> Viktor.
>>>
>>
> 


Re: OpenSSL Logo

2020-02-27 Thread Matt Caswell
The design of the logo was deliberately changed back in (I think) 2015.

The OMC have access to the logo in .xcf, .psd and .png formats.

The new logo had these notes associated with it:


This is the official OpenSSL logo.

It was created in Adobe Photoshop 8.0 with the fonts Adobe Palatino
Bold (for "Open"), Bitstream Gothic 725 Bold (for "SSL") and Tahoma
(for "TM" and "Cryptography and SSL/TLS Toolkit"). Its original size
is 30x10cm with a resolution of 300dpi, hence it is suitable for good
quality printing in A4 paper. Higher resolution versions can be created
easily by loading it into Adobe Photoshop and just resizing to the
required size and/or resolution. This is possible because the text is
not rasterized and both the inner bevel and drop shadow effects are
reapplied automatically. Similarily, all elements are on separated
layers, so they can be switched on/off individually. For processing the
image under Unix, load it into The Gimp 2.0. But keep in mind that you
need the above three fonts installed in other to change the original
source.


Matt


On 27/02/2020 22:03, Dr. Matthias St. Pierre wrote:
> 
>> According to that site, it used to be at
>>
>> http://openssl.com/images/openssl-logo.png
>>
> 
> Thanks to the Wayback Machine, nothing gets lost: Here is the historical 
> OpenSSL Logo:
> 
> https://web.archive.org/web/20141231112717/http://openssl.com/images/openssl-logo.png
> 
> Matthias
> 
> 


Re: Deprecations

2020-02-21 Thread Matt Caswell



On 21/02/2020 23:18, Kurt Roeckx wrote:
> On Fri, Feb 21, 2020 at 11:00:10PM +0000, Matt Caswell wrote:
>>
>> dhparam itself has been deprecated. For that reason we are not
>> attempting to rewrite it to use non-deprecated APIs. The informed
>> decision we have made about DH_check use in dhparam is to not build the
>> whole application in a no-deprecated build:
>>
>>   *) The command line utilities dhparam, dsa, gendsa and dsaparam have been
>>  deprecated.  Instead use the pkeyparam, pkey, genpkey and pkeyparam
>>  programs respectively.
>>  [Paul Dale]
> 
> For some reason I seem to have missed various things.
> 
> But I think deprecating tools like dhparam, dsaparam in favour of
> genpkey is something that we should reconsider.

What is your reasoning?

(I just realised that what the CHANGES entry says is that
dhparam/dsaparam are deprecated in favour of pkeyparam - but actually I
think the equivalent functionality is more split between genpkey and
pkeyparam)

Matt



Re: Deprecations

2020-02-21 Thread Matt Caswell



On 21/02/2020 20:33, Matthew Lindner wrote:
> I think any deprecated apps or options that must be kept and not
> implemented in the new interface should print a warning when used in
> that case then, and only do that when it's impossible to implement it
> in the current release.

I agree with this. We already do that if you use a deprecated
application. I'm not so sure we've been quite so careful with the
deprecated options, so we should check that.

> 
> That's not at all clear with the speed app for example. (That speed
> app was deprecated I just found out in this email thread.)
> 
That's not actually what I said. The speed app itself is not deprecated.
There are options to the speed app, which are deprecated. Its quite
possible we don't print a warning for those - which we should do.

Matt



> - Matthew Lindner
> 
>> On Feb 21, 2020, at 12:14 PM, Matt Caswell 
>> wrote:
>> 
>> EXTERNAL MAIL: openssl-project-boun...@openssl.org
>> 
>> 
>> On 21/02/2020 19:45, Matthew Lindner wrote:
>>> As a user I'm strongly in favor of this. It gives an example of
>>> how to implement something using the new interfaces. Even in
>>> 1.1.1 there are things that are impossible to implement without
>>> using low level interfaces. The applications should be guides on
>>> how to correctly implement something and will point out interface
>>> deficiencies. 3.0.0 should not ship with deprecated code in the
>>> apps (and 1.1.1 should stop using deprecated code in its apps).
>> 
>> As I mentioned in my earlier post I don't believe that this is
>> feasible:
>> 
>> - In some cases an API has been deliberately deprecated with no 
>> replacement. This functionality is also deprecated and will
>> eventually be removed from the app at the same time that the
>> deprecated API is removed.
>> 
>> - In the case of the speed app, the whole point of some
>> functionality is to test the speed of the deprecated API. When the
>> deprecated APIs go, so will that functionality from speed.
>> 
>> - In other cases whole apps are deprecated in favour of a different
>> one that does the same job. The deprecated app will be kept around
>> until the deprecated APIs they use are removed, and then the app
>> will also be removed. Application authors looking for an example
>> should refer to the non-deprecated alternative app.
>> 
>> For all of the above reasons there will have to be some deprecated
>> APIs still being used in the apps in 3.0. Any uses that remain are
>> expected to be removed at the same time as the APIs themselves are
>> removed.
>> 
>> Matt
>> 
>> 
>>> 
>>> - Matthew Lindner
>>> 
>>>> On Feb 21, 2020, at 12:06 AM, Kurt Roeckx 
>>>> wrote:
>>>> 
>>>> EXTERNAL MAIL: openssl-project-boun...@openssl.org
>>>> 
>>>> Hi,
>>>> 
>>>> We seem to be deprecating a lot of the old APIs, which I think
>>>> is a good thing. But I think we might either be deprecating too
>>>> much, or not actually using the alternatives ourself.
>>>> 
>>>> In the apps, a lot of the files define
>>>> OPENSSL_SUPPRESS_DEPRECATED, which I think is the wrong way to
>>>> do it. We should stop using the deprecated functions ourself.
>>>> If there is no way to do this using non-deprecated functions,
>>>> the function should probably not have been deprecated in the
>>>> first place.
>>>> 
>>>> The apps might have functionality that we want to deprecate
>>>> too, that depends on the deprecated functions. In which case we
>>>> should also mark that as deprecated, and the apps should always
>>>> build in no-deprecation mode.
>>>> 
>>>> We might also be deprecating APIs that we don't use ourself, so
>>>>  it's harder to know if there are alternatives for it.
>>>> 
>>>> It would also be nice that if you deprecate a function, that
>>>> you point to the alternative way to do the same thing in the
>>>> manual.
>>>> 
>>>> 
>>>> Kurt
>>>> 
>>>> 
>>> 
> 


Re: Deprecations

2020-02-21 Thread Matt Caswell



On 21/02/2020 22:17, Kurt Roeckx wrote:
> On Fri, Feb 21, 2020 at 09:50:10AM +0000, Matt Caswell wrote:
>>
>>
>> On 21/02/2020 08:06, Kurt Roeckx wrote:
>>> In the apps, a lot of the files define
>>> OPENSSL_SUPPRESS_DEPRECATED, which I think is the wrong way to do
>>> it. We should stop using the deprecated functions ourself. If
>>> there is no way to do this using non-deprecated functions, the
>>> function should probably not have been deprecated in the first
>>> place.
>>>
>>> The apps might have functionality that we want to deprecate too,
>>> that depends on the deprecated functions. In which case we should
>>> also mark that as deprecated, and the apps should always build in
>>> no-deprecation mode.
>>
>> I think we have a number of strategies for dealing with deprecated APIs
>> in the apps depending on the situation:
>>
>> 1) Ideally we just rewrite the functionality using non-deprecated APIs
> 
> The problem is that many of the apps already define
> OPENSSL_SUPPRESS_DEPRECATED so that you don't know that something
> you're deprecating is used there without checking for it.

This is not the case. The only effect that OPENSSL_SUPPRESS_DEPRECATED
has is that it suppreses deprecated warnings from the compiler. However,
if you don't handle that deprecated code in some way you will get a
build failure in a no-deprecated build because the deprecated function
doesn't exist at all.

*If* we use any deprecated APIs we *must* make an informed decision
about what to do about that API use to avoid a no-deprecated build
failure. Since our no-deprecated builds are working just fine, I don't
believe there are any instances in the apps where we haven't made that
informed decision.

> 
> The commit I was looking at was ada66e78ef535fe80e422bbbadffe8e7863d457c:
> Deprecate the low level Diffie-Hellman functions.
> 
> At least one of the functions being deprecated is DH_check, which
> is still used by dhparam. Dhparam is our replacement for dh and gendh.
> I don't know if any of the other function that were deprecated are
> still used internally or not.

dhparam itself has been deprecated. For that reason we are not
attempting to rewrite it to use non-deprecated APIs. The informed
decision we have made about DH_check use in dhparam is to not build the
whole application in a no-deprecated build:

  *) The command line utilities dhparam, dsa, gendsa and dsaparam have been
 deprecated.  Instead use the pkeyparam, pkey, genpkey and pkeyparam
 programs respectively.
 [Paul Dale]

> 
> The define was added in commit 1ddf2594e18137aeb7ce861e54f46824db76e36f,
> and so when DH_check later got deprecated, nobody noticed that the
> now deprecated function is still being used.
> 
> I think the replacement function is EVP_PKEY_param_check().
> 
> DH_check is not mentioned as deprecated in the manual.

Yes, it is:

 #include 

Deprecated since OpenSSL 3.0, can be hidden entirely by defining
B with a suitable version value, see
L:

 int DH_generate_parameters_ex(DH *dh, int prime_len, int generator,
BN_GENCB *cb);

 int DH_check(DH *dh, int *codes);
 int DH_check_params(DH *dh, int *codes);

 int DH_check_ex(const DH *dh);
 int DH_check_params_ex(const DH *dh);
 int DH_check_pub_key_ex(const DH *dh, const BIGNUM *pub_key);

=head1 DESCRIPTION

All of the functions described on this page are deprecated.
Applications should instead use L,
L, L and
L.



Matt

> 
> 
> Kurt
> 


Re: Deprecations

2020-02-21 Thread Matt Caswell



On 21/02/2020 19:45, Matthew Lindner wrote:
> As a user I'm strongly in favor of this. It gives an example of how
> to implement something using the new interfaces. Even in 1.1.1 there
> are things that are impossible to implement without using low level
> interfaces. The applications should be guides on how to correctly
> implement something and will point out interface deficiencies. 3.0.0
> should not ship with deprecated code in the apps (and 1.1.1 should
> stop using deprecated code in its apps).

As I mentioned in my earlier post I don't believe that this is feasible:

- In some cases an API has been deliberately deprecated with no
replacement. This functionality is also deprecated and will eventually
be removed from the app at the same time that the deprecated API is removed.

- In the case of the speed app, the whole point of some functionality is
to test the speed of the deprecated API. When the deprecated APIs go, so
will that functionality from speed.

- In other cases whole apps are deprecated in favour of a different one
that does the same job. The deprecated app will be kept around until the
deprecated APIs they use are removed, and then the app will also be
removed. Application authors looking for an example should refer to the
non-deprecated alternative app.

For all of the above reasons there will have to be some deprecated APIs
still being used in the apps in 3.0. Any uses that remain are expected
to be removed at the same time as the APIs themselves are removed.

Matt


> 
> - Matthew Lindner
> 
>> On Feb 21, 2020, at 12:06 AM, Kurt Roeckx  wrote:
>> 
>> EXTERNAL MAIL: openssl-project-boun...@openssl.org
>> 
>> Hi,
>> 
>> We seem to be deprecating a lot of the old APIs, which I think is a
>> good thing. But I think we might either be deprecating too much, or
>> not actually using the alternatives ourself.
>> 
>> In the apps, a lot of the files define OPENSSL_SUPPRESS_DEPRECATED,
>> which I think is the wrong way to do it. We should stop using the
>> deprecated functions ourself. If there is no way to do this using
>> non-deprecated functions, the function should probably not have
>> been deprecated in the first place.
>> 
>> The apps might have functionality that we want to deprecate too, 
>> that depends on the deprecated functions. In which case we should 
>> also mark that as deprecated, and the apps should always build in 
>> no-deprecation mode.
>> 
>> We might also be deprecating APIs that we don't use ourself, so 
>> it's harder to know if there are alternatives for it.
>> 
>> It would also be nice that if you deprecate a function, that you 
>> point to the alternative way to do the same thing in the manual.
>> 
>> 
>> Kurt
>> 
>> 
> 


Re: Deprecations

2020-02-21 Thread Matt Caswell



On 21/02/2020 08:06, Kurt Roeckx wrote:
> In the apps, a lot of the files define
> OPENSSL_SUPPRESS_DEPRECATED, which I think is the wrong way to do
> it. We should stop using the deprecated functions ourself. If
> there is no way to do this using non-deprecated functions, the
> function should probably not have been deprecated in the first
> place.
> 
> The apps might have functionality that we want to deprecate too,
> that depends on the deprecated functions. In which case we should
> also mark that as deprecated, and the apps should always build in
> no-deprecation mode.

I think we have a number of strategies for dealing with deprecated APIs
in the apps depending on the situation:

1) Ideally we just rewrite the functionality using non-deprecated APIs

2) In some cases that isn't possible for some reason. For example in the
"passwd" app the "-crypt" option uses the deprecated API "DES_crypt". We
have chosen not to provide an alternative for this API, so in this case
the option itself is deprecated.[1] There are other examples of this in
the "speed" application where a particular option specifically tests the
speed of the low-level APIs (i.e. the "-evp" option hasn't been used).
Again in those cases the options themselves are deprecated.

3) In some cases we have chosen to deprecate an entire application. This
is usually because the whole application is written depending on the
low-level APIs and there is an alternative application available that
does the same thing and uses the high-level APIs. Given the existence of
the other application there seems little point in spending a lot of time
rewriting an entire app when we have equivalent functionality elsewhere.
Examples of this are dhparam, dsaparam and ecparam which are deprecated
in favour of pkeyparam. The user gets a warning displayed when they use
one of these deprecated applications.

Everything should build in a no-deprecated build. In the case of (1) the
functionality is obviously still present because we've rewritten it. In
(2) the deprecated options are no longer available and in (3) the whole
command is no longer available. This seems reasonable to me since the
user has specifically requested a build without deprecated functionality
in it.

In the case of both (2) and (3) where the user has not requested a
no-deprecated build, the use of the deprecated APIs obviously still
remains and therefore we need to use OPENSSL_SUPPRESS_DEPRECATED. This
will eventually disappear in future releases as the deprecated APIs are
removed.

Matt



[1] I note BTW that although the option is deprecated this doesn't seem
to have been reflected in the documentation - nor do I see any code to
indicate to the end user that a deprecated option has been used. We
should probably do that.




> 
> We might also be deprecating APIs that we don't use ourself, so
> it's harder to know if there are alternatives for it.
> 
> It would also be nice that if you deprecate a function, that you
> point to the alternative way to do the same thing in the manual.
> 
> 
> Kurt
> 


Tools repo ownership

2020-02-19 Thread Matt Caswell
Following the discussion in PR 58 [1] about the ownership of the tools
repo, I've started an OMC vote to see the tools repo split into "tools"
and "omc-tools", with some tools moving to omc-tools. OMC would have
commit/approval rights for omc-tools, and the OTC would have
commit/approval rights for tools.

I'll report back here once the vote is complete.

Matt

[1] https://github.com/openssl/tools/pull/58


QUIC in OpenSSL

2020-02-17 Thread Matt Caswell
The OMC has just published a blog post on our thoughts on QUIC in
OpenSSL. You can read it here:

https://www.openssl.org/blog/blog/2020/02/17/QUIC-and-OpenSSL/

Matt


Re: Errored: openssl/openssl#31939 (master - 34b1676)

2020-02-14 Thread Matt Caswell



On 14/02/2020 12:23, Nicola Tuveri wrote:
> If ASAN is too slow to run in the CI should we restore the previous
> homemade checks for memory leaks as an alternative to be run in regular
> CI runs and leave ASAN builds to run-checker on the master branch only? 

To be clear the build that is timing out uses *msan* not *asan*.

As I understand it msan detects unitialised reads. whereas asan detects
memory corruption, buffer overflows, use-after-free bugs, and memory leaks.

The previous "home-made" checks only detected memory leaks, so it is not
comparable with the functionality offered by msan.

The msan documentation
(https://clang.llvm.org/docs/MemorySanitizer.html) suggests that a slow
down of 3x is typical.

It seems reasonable to me to disable msan checks in Travis entirely, and
have them only in run-checker.

> 
> Here is another idea that would be interesting if we restore the
> previous checks:
> I don't know what kind of options github offers on this, but would it be
> possible to run triggered CI on something that is not Travis and does
> not timeout and still have the results in the PR?

I am sure there are hooks to do this. Richard has been talking for quite
a while about setting up a buildbot infrastructure. If that could be
integrated into github that would be really neat.

Matt


> If something like that would be possible we could move the ASAN builds
> to extended_tests, rely on the previous memleak detection for the
> regular CI runs, and then trigger with a script or Github Action the
> extended_tests when the approval:done label is added. 
> 
> That way, by the time something is ready to be merged we should have a
> full picture! 
> 
> 
> Nicola
> 
> On Wed, Feb 5, 2020, 10:25 Matt Caswell  <mailto:m...@openssl.org>> wrote:
> 
> Since we fixed the Travis builds 4 out of the 8 builds on master that
> have taken place have errored due to a timeout.
> 
> The msan build is consistently taking a *very* long time to run. If it
> gets to 50 minutes then Travis cuts it off and the build fails.
> 
> Should we disable the msan build?
> 
> Matt
> 
> 
>  Forwarded Message 
> Subject:        Errored: openssl/openssl#31939 (master - 34b1676)
> Date:   Wed, 05 Feb 2020 00:02:01 +
> From:   Travis CI mailto:bui...@travis-ci.org>>
> To:     openssl-comm...@openssl.org <mailto:openssl-comm...@openssl.org>
> 
> 
> 
> openssl
> 
> /
> 
> openssl
> 
> 
> <https://travis-ci.org/openssl/openssl?utm_medium=notification_source=email>
> 
> 
> branch iconmaster <https://github.com/openssl/openssl/tree/master>
> 
> build has errored
> Build #31939 has errored
> 
> <https://travis-ci.org/openssl/openssl/builds/646181069?utm_medium=notification_source=email>
> arrow to build time
> clock icon50 mins and 3 secs
> 
> Pauli avatarPauli
> 
> 34b1676 CHANGESET →
> <https://github.com/openssl/openssl/compare/e3b1ccad694a...34b167625af5>
> 
> Make minimum size for secure memory a size_t.
> 
> The minimum size argument to CRYPTO_secure_malloc_init() was an int but
> ought
> to be a size_t since it is a size.
> 
> From an API perspective, this is a change. However, the minimum size is
> verified as being a positive power of two and it will typically be a
> small
> constant.
> 
> Reviewed-by: David von Oheimb  <mailto:david.von.ohe...@siemens.com>>
> (Merged from #11003)
> 
> Want to know about upcoming build environment updates?
> 
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you!
> 
> SIGN UP HERE <http://eepurl.com/9OCsP>
> 
> book icon
> 
> Documentation <https://docs.travis-ci.com/> about Travis CI
> 
> Have any questions? We're here to help.
> <mailto:supp...@travis-ci.com <mailto:supp...@travis-ci.com>>
> Unsubscribe
> 
> <https://travis-ci.org/account/preferences/unsubscribe?repository=5849220_medium=notification_source=email>
> from build emails from the openssl/openssl repository.
> To unsubscribe from *all* build emails, please update your settings
> 
> <https://travis-ci.org/account/preferences/unsubscribe?utm_medium=notification_source=email>.
> 
> black and white travis ci logo <https://travis-ci.com>
> 
> Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
> Jacops | Contact: cont...@travis-ci.com
> <mailto:cont...@travis-ci.com> <mailto:cont...@travis-ci.com
> <mailto:cont...@travis-ci.com>> |
> Amtsgericht Charlottenburg, Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß
> §27 a Umsatzsteuergesetz: DE282002648
> 


Re: Deprecation

2020-02-14 Thread Matt Caswell



On 14/02/2020 10:46, Dr Paul Dale wrote:
> Roumen in
> https://github.com/openssl/openssl/pull/10977#issuecomment-584818517
> Dmitry
> in https://github.com/openssl/openssl/pull/11082#issuecomment-585603911

Thanks.

Having re-read the comments and the thread here, I am still of the
opinion that we should press ahead with the deprecations as planned.

Matt

> And a further one via private email.
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> 
> 
> 
>> On 14 Feb 2020, at 7:37 pm, Matt Caswell > <mailto:m...@openssl.org>> wrote:
>>
>>
>>
>> On 14/02/2020 02:30, Dr Paul Dale wrote:
>>> There is some pushback against the deprecations going on in various PRs.
>>
>> I've not followed all of the PRs. Can you point at some specific
>> comments you've received pushing back on this?
>>
>> Matt
>>
>>
>>>
>>> The plan has always been to deprecate engines in 3.0 and removing
>>> support for them 5+ years later.  Originally, the path was to have
>>> included an engine provider that could load engines and make them appear
>>> to be a provider.  After a fair amount of investigation, this was deemed
>>> to be too difficult in the 3.0 time frame.
>>>
>>> Do we still want to deprecate engines in 3.0?
>>> Should we defer until 4.0 instead?
>>>
>>>
>>> The main benefits seem to boil down to continuing to support existing
>>> engines vs removing the legacy code paths and switching to the provider
>>> model.
>>>
>>>
>>> Pauli
>>> -- 
>>> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
>>> Phone +61 7 3031 7217
>>> Oracle Australia
>>>
> 


Re: Deprecation

2020-02-14 Thread Matt Caswell



On 14/02/2020 02:30, Dr Paul Dale wrote:
> There is some pushback against the deprecations going on in various PRs.

I've not followed all of the PRs. Can you point at some specific
comments you've received pushing back on this?

Matt


> 
> The plan has always been to deprecate engines in 3.0 and removing
> support for them 5+ years later.  Originally, the path was to have
> included an engine provider that could load engines and make them appear
> to be a provider.  After a fair amount of investigation, this was deemed
> to be too difficult in the 3.0 time frame.
> 
> Do we still want to deprecate engines in 3.0?
> Should we defer until 4.0 instead?
> 
> 
> The main benefits seem to boil down to continuing to support existing
> engines vs removing the legacy code paths and switching to the provider
> model.
> 
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 


Re: Should the return result of CRYPTO_UP_REF() / CRYPTO_DOWN_REF() be checked?

2020-02-10 Thread Matt Caswell



On 10/02/2020 00:15, SHANE LONTIS wrote:
> With the new architecture changes there are quite a few new calls to
> 
> CRYPTO_UP_REF()
> CRYPTO_DOWN_REF()
> 
> These methods return an int that is not being checked in lots of places.
> 
> This return value only seems to affect fallback code that calls 
> CRYPTO_atomic_add (which can return 0 on lock or unlock failure)
> 
> SO the question is should we be checking this return value?

Yes, I think we should be.

Matt

> 
> 
> Note that not checking has resulted in a few assumptions in other code…
> e.g the following function returns void.
>  
> /crypto/evp/keymgmt_lib.c: 165 in evp_keymgmt_util_cache_pkey()
> 159 }
> 160 
> 161 void evp_keymgmt_util_cache_pkey(EVP_PKEY *pk, size_t index,
> 162  EVP_KEYMGMT *keymgmt, void *keydata)
> 163 {
> 164 if (keydata != NULL) {
CID 1458170:  Error handling issues  (CHECKED_RETURN)
Calling "EVP_KEYMGMT_up_ref" without checking return value (as is done 
 elsewhere 4 out of 5 times).
> 165 EVP_KEYMGMT_up_ref(keymgmt);
> 
> NOTE: EVP_KEYMGMT_up_ref() just does an CRYPTO_UP_REF() call and always 
> returns 1.
> 
> 


Re: Travis in solid red mode again

2020-02-07 Thread Matt Caswell



On 03/02/2020 19:19, Matthias St. Pierre wrote:
> Mission accomplished, Travis just turned green again on master  :-)

Every one of the last 12 master jobs have failed to complete
successfully due to a travis timeout. Apart from the build that is
timing out (enable-msan), all the other builds in the job are green.

Matt

> 
> https://travis-ci.org/openssl/openssl/builds/645563965?utm_source=github_status_medium=notification
> 
> 
> Thanks to Matt, Richard and everybody else who participated in fixing it.
> 
> 
> Matthias
> 
> 
> 


  1   2   3   4   >