Re: Status of the remaining beta1 PRs

2020-09-20 Thread Richard Levitte
On Fri, 18 Sep 2020 17:24:59 +0200,
Matt Caswell wrote:
> 
> 1 PR which is in a state of "we really need to do something about this":
> [WIP, Parked] EVP: Adapt EVP_PKEY_set_alias_type() for provider-native
> EVP_PKEYs
> https://github.com/openssl/openssl/pull/12675
> Since this affects a public API we probably really do need to figure out
> what to do with the EVP_PKEY_set_alias_type()

There's a counter PR now: #12920

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: 3.0 beta 1 milestone

2020-09-18 Thread Richard Levitte
I've found what's going wrong there, and I agree that it needs to be
fixed ASAP, although I don't view it per se as a beta 1 blocker.

Either way, a fix is coming up.

Cheers,
Richard

On Thu, 17 Sep 2020 19:21:50 +0200,
Dmitry Belyavsky wrote:
> 
> 
> Dear Matt,
> 
> I think #12891 is a significant problem. I'd suggest fixing it before beta1 
> or at least discuss
> it.
> 
> Many thanks!
> 
> On Thu, Sep 17, 2020 at 3:48 PM Matt Caswell  wrote:
> 
> There's been quite a number of PRs added to the 3.0 beta 1 milestone.
>
> Within the PRs there are a couple of bug fixes:
>
> https://github.com/openssl/openssl/pull/12884
> https://github.com/openssl/openssl/pull/12874
>
> IMO these would be really nice to get into beta 1, but they should not
> be considered blockers for it (i.e. if they didn't go in, it shouldn't
> stop us from releasing beta 1).
>
> There are also some nice-to-have items:
>
> https://github.com/openssl/openssl/pull/12777
> https://github.com/openssl/openssl/pull/12771
> https://github.com/openssl/openssl/pull/12726
> https://github.com/openssl/openssl/pull/12669
> https://github.com/openssl/openssl/pull/12072
>
> Again - these are nice-to-have, and if they happen to get merged in time
> for beta 1 then great. Otherwise, they should wait for 3.1 (possibly
> things which are just cleanup/minor refactoring could still be done
> within the beta period). So, IMO, these should not be considered
> blockers either.
>
> So - this leads me to the question - what is the milestone for? Does it
> means these things *must* go in before we can release beta 1? Or does it
> mean we would *like* to get these things in for beta 1?
>
> I actually don't mind either way - but if its the latter, then I need a
> way of identifying the "must haves". These are the top priority items,
> and at the moment I can't easily track their progress.
>
> Matt
> 
> --
> SY, Dmitry Belyavsky
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: 3.0 beta 1 milestone

2020-09-18 Thread Richard Levitte
On Thu, 17 Sep 2020 18:49:19 +0200,
Kurt Roeckx wrote:
> 
> On Thu, Sep 17, 2020 at 01:48:18PM +0100, Matt Caswell wrote:
> > So - this leads me to the question - what is the milestone for? Does it
> > means these things *must* go in before we can release beta 1? Or does it
> > mean we would *like* to get these things in for beta 1?
> 
> We need to have a decision about those before beta 1, which can
> include:
> - It should be in beta 1
> - It doesn't block beta 1, before the 3.0 release is fine too,
>   which just means that you change the milestone to 3.0
> - It doesn't go into 3.0. No idea what the best way to tag them
>   is.

Sorry, I realise I repeated what you said...

Regarding things not going into 3.0.0, one way is not to assign a
milestone to them.  If we decide to do it that way, it probably means
that we should set the 3.0.0 milestone on everything that we consider
a bug fix rather than a new feature.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: 3.0 beta 1 milestone

2020-09-18 Thread Richard Levitte
On Thu, 17 Sep 2020 15:57:52 +0200,
Tomas Mraz wrote:
> I do not think the milestone should include nice-to-have items.

Another view is that beta 1 is feature freeze.  If those nice to have
items are characterized as new features, then it makes sense to have
them included in the beta 1 milestone if we want them for 3.0.0.
Otherwise, we will have to put them on the waiting list for after
3.0.0.

If they are not characterized as new features, that is a different
matter, of course.  The conclusion is that we will have to take some
time (I assume that'll be quick in most cases) thinking how we
characterize certain PRs before beta 1 is released.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: New GitHub label for release blockers

2020-09-15 Thread Richard Levitte
On Tue, 15 Sep 2020 09:46:53 +0200,
Dr. Matthias St. Pierre wrote:
> > If we make them the same, what's the reason to have both?
> 
> I completely agree. According to my understanding,  it would have
> made more sense if we had created an "OpenSSL 3.0" project for
> managing all what needs to go into this major release and three
> milestones '3.0.0 alpha',  '3.0.0 beta', '3.0.0' associated with the
> major events.

And that, exactly that, would create a project that fulfill the exact
same functionality as the 3.0.0 milestone.  Why would that be
necessary?  Just use the milestones.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: New GitHub label for release blockers

2020-09-14 Thread Richard Levitte
On Sun, 13 Sep 2020 22:41:23 +0200,
Dr. Matthias St. Pierre wrote:
> 
> Conversely, there were also pull requests associated with the '3.0.0' or 
> '3.0.0 beta1' milestone,
> without  being associated to the  '3.0 New Core + FIPS' project. This has 
> been fixed now.

I fail to see why the milestones '3.0.0' / '3.0.0 beta1' must be 1:1
with the '3.0 New Core + FIPS' project.  If we make them the same,
what's the reason to have both?

I just looked in the project, and these are issues and PRs the
presence of which I think is questionable:

#10612
#4930
#12860
#11311

Maybe there's a misunderstand of what "3.0 New Core" means.  Please
note the lack of comma.  But if that doesn't help, how about the
project description?

I've seen a bit too much of wanting to pile *everything* into that
project.  That was never the intention.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-14 Thread Richard Levitte
are that the intent
wasn't originally as grand as you make it out to be.  It was a name I
came up with spur of the moment.  Today, I wish I never had, at least
in the public API (I'm less fussy about internal functions).  It was a
bad choice on my part.

> IMO *the* most confusing thing would be to *change* an existing ordering
> 
> (for example swap two existing parameters around). I see no problem with
> adding new ones anywhere that's appropriate.
> 
> Clearly we disagree on that - if you are making an extended version
> of a function and you aren't keeping the same existing parameter
> order (which you are not if you add or remove parameters which is
> the point I was making - the order isn't the same as the existing
> function because you've removed items - what you have is the order
> of whatever parameters remain in the extended function are the same
> - and that's a pretty pointless distinction to make - if you aren't
> simply adding additional items on the end you make for a change over
> mess and this I think we should be trying to avoid).

I think we have some very fundamentaly different understanding of
"order".  I'm with Matt on this; if you have a list A, B, C, D and
remove C, then A, B, D are still in the same order.

As to where to add arguments, I would say that yes, it's good to have
a general guideline, but that it will have to be decided more
carefully on a per function basis.  For example, I wouldn't
necessarily add a property query string at the very end, if the
algorithm name or the object the algorithm name can be derived from
isn't there.  So, I would not at all want to see something like this
(this isn't what we have now, but with the idea that we always place
OPENSSL_CTX as first argument and other added arguments last, this
would be the result):

OSSL_STORE_CTX *
OSSL_STORE_open(OPENSSL_CTX *, const char *uri,
const UI_METHOD *ui_method, void *ui_data,
OSSL_STORE_post_process_info_fn post_process,
void *post_process_data, const char *propq);

Because the property query string is strongly associated with the
implementation name (which is derived from the URI in this case), the
more meaningful placement would be this:

OSSL_STORE_CTX *
OSSL_STORE_open(OPENSSL_CTX *, const char *uri, const char *propq,
const UI_METHOD *ui_method, void *ui_data,
OSSL_STORE_post_process_info_fn post_process,
void *post_process_data, const char *propq);

(that also follows the current _fetch function model)

> My context there is for the users of the existing API.
> It becomes especially problematic if you have type equivalence when
> the order is changed around so there are no compiler warnings if the
> user hasn't picked up on reordering of parameters.

That is indeed a problem, but I fail to see how removed arguments or
the location of added arguments matter much for this.  Unless we get
into +1 -1 situation and that a 'void *' just so happens to end up in
a place that had a different type that gets silently passed anyway, I
doubt that will be much of a problem.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-09 Thread Richard Levitte
On Wed, 09 Sep 2020 16:08:10 +0200,
Tomas Mraz wrote:
> 
> On Wed, 2020-09-09 at 22:29 +1000, Dr Paul Dale wrote:
> > > On 9 Sep 2020, at 9:38 pm, Tomas Mraz  wrote:
> > > 
> > > We could even provide a convenience thread local stack of lib
> > > contexts
> > > so the caller would not have to keep the old value but would just
> > > push
> > > the new libctx when entering and pop the old one when leaving. With
> > > that, I think the changes needed in the application code would be
> > > fairly simple and minimal.
> > 
> > Let’s not overcomplicate things.
> > We went through this discussion back when this was introduced.
> > 
> > 
> > Push is:
> > OPENSSL_CTX *prevctx = OPENSSL_CTX_set0_default(libctx);
> > 
> > Pop is:
> > OPENSSL_CTX_set0_default(prevctx)
> > 
> > 
> > I don’t see having an explicit stack of these is of any benefit to
> > anything but unwarranted complexity.
> 
> There is one thing where it would be IMO helpful - let's say libcurl
> has a callback into a calling application. With the current API in
> libcurl API calls you would put the
> calls OPENSSL_CTX_set0_default(libctx)/OPENSSL_CTX_set0_default(prevctx
> ) at the beginning and end. But you would have to save the prevctx
> somewhere in the libcurl context structure because on callbacks you
> would have to temoprarily reset the context to the prevctx value. If we
> implemented real stack it would not be needed. But yeah, I am not sure
> this convenience is that much worth it.

A stack becomes potentially extra churn in memory allocation for every
push, and wondering what to do with the stack if the stack
functionality fails.  So yeah, I'm thinking that it's not really worth
it.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-09 Thread Richard Levitte
On Wed, 09 Sep 2020 13:38:42 +0200,
Tomas Mraz wrote:
> > Regarding model 3, it must be said that there is potential for
> > confusion
> > on what it's supposed to do, replace the default property query
> > string
> > (settable with EVP_set_default_properties()), or merge with it.
> > Remember that a property query string given through any XXX_fetch()
> > function is temporarly merged with the default properties when
> > looking
> > for fitting algorithms.
> 
> Here I would actually argue, that there should be another property
> query in the libctx in addition to the default, that would have the
> exactly same semantics as the propq parameter has now and for the
> functions with the propq parameter it even would not be applied. How to
> name it so it won't be confused with the default property query, that's
> hard to say though.

"current" was mentioned several times during yesterday's videocall,
that could actually be used for a name.

I'm ok with putting such a property query string into the library
context, As Long As there is the understanding that it's not *tied*
to that library context, i.e. it can be used to pass a property query
string on to a provider implementation, to be used with whatever
library context the provider uses (the FIPS module has its own, for
example).

Side note: the function OPENSSL_CTX_set0_default() is a confusing
name, as there's an internal default ("default default", "ultimate
default"?) library context already, it's possible that we should
rename that to OPENSSL_CTX_set0_current().

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-09 Thread Richard Levitte
ng to be noted is that model 2 and 3 is possible to combine,
which could give us a smoother transition between the current API and
whatever we design going forward.



Regarding the property query string, looking at it separate from the
library context, the question remains where to put it, and a few
proposals are on the table:

1.  Put it last.

2.  Put it next to the algorithm name or the object that an algorithm
name is inferred from.

3.  Set it "globally" with a thread local variable, a bit like what
OPENSSL_CTX_set0_default() does for library contexts.

For this model, it's been argued if it should simply be stored in
the current library context, thereby avoiding to add another
thread local variable (which, for all intents and purposes, is
another actually global thing to deal with, even though on
per-thread level).

In my mind, model 2 would be more sensible than model 1, because of
the implied tie between algorithm name and property query string.
Also, even here, model 3 is possible to combine with model 2, and even
with model 1.

Regarding model 3, it must be said that there is potential for confusion
on what it's supposed to do, replace the default property query string
(settable with EVP_set_default_properties()), or merge with it.
Remember that a property query string given through any XXX_fetch()
function is temporarly merged with the default properties when looking
for fitting algorithms.



Thoughts?

Cheers,
Richard

[*] The OpenSSL API could do with a re-design, but that's for the
farther future and requires a lot of thought and time.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-07 Thread Richard Levitte
On Sun, 06 Sep 2020 23:40:53 +0200,
SHANE LONTIS wrote:
> 
> If it is decided that the libctx is more important then the API
> should really be something like this..
> OSSL_LIBCTX_MD_fetch(), (and I dont know if that is a good idea.)..

I would rather not see that.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Reordering new API's that have a libctx, propq

2020-09-06 Thread Richard Levitte
There are many red herrings in here, and I would argue that trying to
be too uniform in the way you think about all functions may be
harmful, because not all functions are classified the same.

We cannot deny that many of our interfaces have an OOP flair.  We may
be programming in C, but we very obviously have that kind of pattern,
a "poor man's OOP" in C.  So speaking about our interfaces in OOP
terms is not far fetched, as long as we keep in mind that we can only
take the argument so far.

I would argue that EVP_XXX_get_whatever() / EVP_XXX_get_whatever() are
classic accessors, and I assume that we have all been brought up with
the description of "this" as the hidden first argument to any class
method in C++, so it's not very far fetched to emulate that in C by
making the instance you want details from or change details of as the
(explicit) first argument.

I would further argue that EVP_XXX_fetch() is a constructor (of an
instance of EVP_XXX), and that with the assumption that there is no
"this" in a constructor, the discussion about the arguments and their
order must be different.

What I hear, even though not in such terms, is an argument of the
what the library context is and how that should affect the interface.
In relation to EVP_XXX_fetch(), it seems to be more of a factory (or
strictly speaking, the pool of resources, with hidden factory
functions), and depending on the factory model you lean on, it might
or might not be sensible to have it as first argument.  My you, I'm
trying quite hard to see it from a fresh user experience (as far as I
can imagine it...  after all, 20ish years with my fingers deep in
OpenSSL entrails makes you not so fresh).

I think, BTW, that this really comes down to how we view the library
context.  So far, I've seen all these interpretations (not all said
explicitly, but clearly visible in code or how we argue about it):

- framework
- scope (I'm unsure if that differs much from framework)
- factory / factory pool
- bag of resources

Personally, I have zero problems viewing it as all of these combined,
but that requires us to be clear on how it's used in different places.

Cheers,
Richard

On Sat, 05 Sep 2020 23:18:21 +0200,
Tim Hudson wrote:
> 
> 
> On Sun, Sep 6, 2020 at 6:55 AM Richard Levitte  wrote:
> 
> I'd rank the algorithm name as the most important, it really can't do
> anything of value without it.
> 
> It also cannot do anything without knowing which libctx to use. Look at the 
> implementation.
> Without the libctx there is no "from-where" to specify. 
> 
> This is again hitting the concept of where do things come from and what is a 
> default.
> Once "global" disappears as such, logically everything comes from a libctx.
> 
> Your argument is basically "what" is more important than "from" or "where".
> And the specific context here is where you see "from" or "where" can be 
> defaulted to a value so it
> can be deduced so it isn't (as) important in the API.
> 
> That would basically indicate you would (applying the same pattern/rule in a 
> different context)
> change:
> 
> int EVP_PKEY_get_int_param(EVP_PKEY *pkey, const char *key_name, int *out);
> 
> To the following (putting what you want as the most important rather than 
> from):
> 
> int EVP_PKEY_get_int_param(char *key_name, EVP_PKEY *pkey, int *out);
> 
> Or pushing it right to the end after the output parameter:
> 
> int EVP_PKEY_get_int_param(char *key_name, int *out,EVP_PKEY *pkey);
> 
> The context of where things come from is actually the most critical item in 
> any of the APIs we
> have.
> Even though what you want to get from where you want to get it is in the 
> point of the API call you
> need to specify where from first as that context sets the frame of the call.
> 
> Think of it around this way - we could have an implementation where we 
> remember the last key that
> we have used and allow you to simply indicate you use the last key or if we 
> didn't want the last
> key used to be able to specify it via a non-NULL pointer. This isn't that 
> unusual an API but not
> something I'm suggesting we add - just trying to get the point across that 
> you are still thinking
> of global and libctx as something super special with an exception in its 
> handling rather than
> applying a general rule which is pretty much what we use everywhere else.
> 
> And in which case where you generally don't provide a reference as there is 
> some default meaning
> for it in general and can provide a reference for that sort of API would this 
> make sense to you:
> 
> int EVP_PKEY_get_int_param(char *key_name, int *out,EVP_PKEY *pkey);
> 
> If pkey is NULL then you use the last key that you referenced, if it is not 

Re: Reordering new API's that have a libctx, propq

2020-09-05 Thread Richard Levitte
the caller needs to always 
> specify a value for each
> argument, which are always positional) in the sense that they can pass NULL 
> as a value rather than
> a pointer to a fully instantiated object of the required type.
> Even more so given that, excluding a minority of cases, we can expect 
> consumers of the APIs taking
> libctx and propq as arguments to pass NULL for both of them and rely on the 
> openssl config
> mechanism.
> 
> So while I agree with Tim that sometime it is valuable to make a difference 
> among the consequences
> of passing NULL as arguments in the context of one kind of function and 
> another, I believe the
> place for that is the documentation not its signature.
> The signature of the function should be designed for consumer usability, and 
> the conventional
> pattern there seems to be
> - required args
> - optional args
> - callback+cb_args
> and inside each group the "importance" factor should be the secondary sorting 
> criteria. 
> 
> "importance" is probably going to be affected by the difference you are 
> making (or my
> understanding of it): e.g. if a function took both libctx and bnctx, the fact 
> that a valid
> pre-existing libctx is required to work (and a global already existing one 
> will be retrieved in
> case none is given), while a fresh short-lived bnctx is going to be created 
> only for the lifetime
> of the called function in case none is given seems to indicate that libctx is 
> of vital importance
> for the API functionality, while bnctx is of minor relevance. 
> 
> But... going this way as a generalized approach, would bring us to the "add 
> in the middle"
> scenario that we'd like to avoid. 
> I recognize that this is a point you already made in your original writeup, 
> as the tendency of
> "add to the end" to naturally degrade into "add in the middle".
> 
> So, my question is: if "degradable add to the end" (where "degradable" only 
> happens rarely and for
> good reasons) seems the one that in the end produces signatures matching 
> (IMHO) the conventional
> usability patterns expected by consumers of the API, is it such a dramatic 
> conclusion that we want
> to exclude it?
> 
> Or is your point that we are writing in C, all the arguments are positional, 
> none is ever really
> optional, there is no difference between passing a `(void*) NULL` or a valid 
> `(TYPE*) ptr` as the
> value of a `TYPE *` argument, so "importance" is the only remaining sorting 
> criteria, hence
> (libctx, propq) are always the most important and should go to the beginning 
> of the args list
> (with the exception of the `self/this` kind of argument that always goes 
> first)? 
> 
> Nicola
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: OTC VOTE in progress: Rename OSSL_SERIALIZER / OSSL_DESERIALIZER to OSSL_ENCODE / OSSL_DECODE

2020-08-20 Thread Richard Levitte
Votes have been cast, and the verdict is:

accepted:  yes  (for: 5, against: 1, abstained: 4, not voted: 1)

Cheers,
Richard

On Tue, 18 Aug 2020 13:15:46 +0200,
Richard Levitte wrote:
> 
> topic: Rename OSSL_SERIALIZER / OSSL_DESERIALIZER to OSSL_ENCODE / OSSL_DECODE
> 
>The rationale is that it makes things easier on programmers
>(encode / decode is easier to write than serialize / deserialize),
>and also avoids disputes on what is and isn't serialization.
> 
>Associated issues and PRs: #12455, #12659 and #12660
> 
> Cheers,
> Richard
> 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> ___
> otc mailing list
> o...@openssl.org
> https://mta.openssl.org/mailman/listinfo/otc
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


OTC VOTE in progress: Rename OSSL_SERIALIZER / OSSL_DESERIALIZER to OSSL_ENCODE / OSSL_DECODE

2020-08-18 Thread Richard Levitte
topic: Rename OSSL_SERIALIZER / OSSL_DESERIALIZER to OSSL_ENCODE / OSSL_DECODE

   The rationale is that it makes things easier on programmers
   (encode / decode is easier to write than serialize / deserialize),
   and also avoids disputes on what is and isn't serialization.

   Associated issues and PRs: #12455, #12659 and #12660

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
___
otc mailing list
o...@openssl.org
https://mta.openssl.org/mailman/listinfo/otc



Re: API renaming

2020-07-24 Thread Richard Levitte
Why?  Just because some of us used such variable names when there was
a need to distinguish it from other contexts that are sometimes
juggled with at the time time?

(and yeah, I've made that a habit...  but why that would determine the
type name, I cannot understand)

Cheers,
Richard

On Fri, 24 Jul 2020 09:42:59 +0200,
SHANE LONTIS wrote:
> 
> I was thinking OSSL_LIBCTX?
> 
> > On 24 Jul 2020, at 5:38 pm, Dr. Matthias St. Pierre 
> >  wrote:
> > 
> > I like the OSSL_ prefix for new APIs as proposed by Richard. And I agree 
> > with Shane
> > that we should go for a single prefix and not have two alternatives. 
> > Existing prefixes
> > for things like feature macros should remain as they are, but if the OSSL_ 
> > prefix is
> > our choice for the future, we should start using it consistently for _all_ 
> > new APIs in 3.0.
> > And not make it a random choice (pun intended) depending on whether the API 
> > went
> > into master early or late. So my favorite choice is a consistent renaming, 
> > i.e.
> > 
> > OSSL_MAC, OSSL_KDF, OSSL_RAND, OSSL_CTX, ...
> > 
> > OTOH, it would be ok for me if we would make an exception for EVP_MAC and 
> > EVP_KDF,
> > because they have a long EVP history, as Matt pointed out. But using the 
> > EVP_ prefix
> > for the new RAND interface never made sense to me.
> > 
> > What bothers me about OPENSSL_CTX in particular is the fact that it is a 
> > mixture of
> > a non-abbreviated and an abbreviated word. IMHO it should be either 
> > OSSL_CTX or
> > OPENSSL_CONTEXT, and the former is obviously preferrable for its length.
> > 
> > Matthias
> > 
> > 
> >> -Original Message-
> >> From: openssl-project  On Behalf Of 
> >> Richard Levitte
> >> Sent: Friday, July 24, 2020 8:34 AM
> >> To: openssl-project@openssl.org
> >> Subject: Re: API renaming
> >> 
> >> I'm fine with that, really.
> >> 
> >> I have a preference for OSSL_, but if we look through the source,
> >> there's reason for either.  So far, we've majorly used OPENSSL_ for
> >> things like feature macros (disabling macros, really), environment
> >> variables, that sort of thing, while OSSL_ has become the primary
> >> choice for new APIs ("less klunky", I think the judgement was in that
> >> past discussion I keep referring to).
> >> 
> >> And yeah, I hear you from all the way around the planet, there are
> >> some recent name choice that were made that give pause and are
> >> arguably a mistake in this regard.  EVP_MAC and EVP_KDF could have
> >> been OSSL_MAC and OSSL_KDF.  OPENSSL_CTX could have been OSSL_CTX.
> >> I have no problem recognising that.  But, they are there, even though
> >> only in master (*).  This is question of what we do going forward (a
> >> yet unmerged PR with a new API is as good a target as any).
> >> 
> >> Cheers,
> >> Richard
> >> 
> >> (*) I'm not sure I see master as something untouchable in this regard,
> >> as the development is still not frozen (alpha), so I for one could
> >> easily see a rename fest happening, should we decide for it.  That
> >> must happen before we enter the beta phase, though...
> >> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: API renaming

2020-07-24 Thread Richard Levitte
We're talking APIs (*), that includes the types.  So yes, that's a
safe assumption.

Cheers,
Richard

(*) if people stopped using "API" when they mean "function", that
would save the world from a pile of confusion.

On Thu, 23 Jul 2020 18:45:49 +0200,
Short, Todd wrote:
> 
> 
> They also correspond directly to EVP_MAC and EVP_KDF types. Would the types 
> change as well?
> --
> -Todd Short
> // tsh...@akamai.com
> // “One if by land, two if by sea, three if by the Internet."
> 
> On Jul 23, 2020, at 11:56 AM, Matt Caswell  wrote:
> 
> On 23/07/2020 16:52, Richard Levitte wrote:
>
> On Thu, 23 Jul 2020 12:18:10 +0200,
> Dr Paul Dale wrote:
>
> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  
> This seems reasonable.
>  Would it
> also make sense to rename the other new APIs similarly.
> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and OSSL_KDF 
> respectively?
> 
> This is a good question...
>
> Historically speaking, even though EVP_MAC and EVP_KDF are indeed new
> APIs, they have a previous history of EVP APIs, through EVP_PKEY.  The
> impact of relocating them outside of the EVP "family" may be small,
> but still, history gives me pause.
>
> RAND doesn't carry the same sort of history, which makes it much
> easier for me to think "just do it and get it over with"...
> 
> I have the same pause - so  I'm thinking just RAND for now.
>
> Matt
> 
> 
> No public key for CFC553A2BA1A0ED1 created at 2020-07-23T18:45:49+0200 using 
> RSA
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: API renaming

2020-07-24 Thread Richard Levitte
I'm fine with that, really.

I have a preference for OSSL_, but if we look through the source,
there's reason for either.  So far, we've majorly used OPENSSL_ for
things like feature macros (disabling macros, really), environment
variables, that sort of thing, while OSSL_ has become the primary
choice for new APIs ("less klunky", I think the judgement was in that
past discussion I keep referring to).

And yeah, I hear you from all the way around the planet, there are
some recent name choice that were made that give pause and are
arguably a mistake in this regard.  EVP_MAC and EVP_KDF could have
been OSSL_MAC and OSSL_KDF.  OPENSSL_CTX could have been OSSL_CTX.
I have no problem recognising that.  But, they are there, even though
only in master (*).  This is question of what we do going forward (a
yet unmerged PR with a new API is as good a target as any).

Cheers,
Richard

(*) I'm not sure I see master as something untouchable in this regard,
as the development is still not frozen (alpha), so I for one could
easily see a rename fest happening, should we decide for it.  That
must happen before we enter the beta phase, though...

On Fri, 24 Jul 2020 07:55:10 +0200,
SHANE LONTIS wrote:
> 
> For (1) the use of either ‘OPENSSL_' OR ‘OSSL_’  is not a particularly great 
> rule either.
> We should decide which one to use and stick to it.
> 
> > On 24 Jul 2020, at 3:20 pm, Richard Levitte  wrote:
> > 
> > A couple of points:
> > 
> > 1.  Quite a while ago, we (the team at the time) made a decision to
> >have all new APIs prefixed with 'OPENSSL_' or 'OSSL_'.  It seems
> >that we never voted on it, though, but still.
> > 
> > 2.  The new RAND API hasn't been merged yet, so it's not like we're
> >renaming something that already exists.
> > 
> > So in terms of "it's just a prefix", OSSL_ would be just as suitable.
> > It's a bit more blatantly "OpenSSL", though.
> > 
> > Cheers,
> > Richard
> > 
> > On Thu, 23 Jul 2020 23:30:25 +0200,
> > Tim Hudson wrote:
> >> Placing everything under EVP is reasonable in my view. It is just a prefix 
> >> and it really has no
> >> meaning these days as it became nothing more than a common prefix to use.
> >> 
> >> I don't see any significant benefit in renaming at this point - even for 
> >> RAND.
> >> 
> >> Tim.
> >> 
> >> On Fri, 24 Jul 2020, 1:56 am Matt Caswell,  wrote:
> >> 
> >>On 23/07/2020 16:52, Richard Levitte wrote:
> >>> On Thu, 23 Jul 2020 12:18:10 +0200,
> >>> Dr Paul Dale wrote:
> >>>> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  This seems 
> >>>> reasonable.  Would
> >>it
> >>>> also make sense to rename the other new APIs similarly.
> >>>> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and OSSL_KDF 
> >>>> respectively?
> >>> 
> >>> This is a good question...
> >>> 
> >>> Historically speaking, even though EVP_MAC and EVP_KDF are indeed new
> >>> APIs, they have a previous history of EVP APIs, through EVP_PKEY.  The
> >>> impact of relocating them outside of the EVP "family" may be small,
> >>> but still, history gives me pause.
> >>> 
> >>> RAND doesn't carry the same sort of history, which makes it much
> >>> easier for me to think "just do it and get it over with"...
> >> 
> >>I have the same pause - so  I'm thinking just RAND for now.
> >> 
> >>Matt
> >> 
> >> 
> > -- 
> > Richard Levitte levi...@openssl.org
> > OpenSSL Project 
> > https://urldefense.com/v3/__http://www.openssl.org/*levitte/__;fg!!GqivPVa7Brio!KL97HvjYmS7a3QKC8tJzRlM2dM4t9WLQOYHSX50pDVuxB5XrRy5zA3onhN1dMVGCCw$
> >  
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: API renaming

2020-07-23 Thread Richard Levitte
Er, I don't feel like I was part of this "we".

I was very much part of the discussion that introduced OSSL_ and
OPENSSL_ as a common prefix, thought...  actually only three years
ago.

(historical note: I had written the STORE API, using STORE_ as a
prefix, but that was judged too common, and that's what sparked the
discussion at the time...  and that's why we now have a OSSL_STORE)

Cheers,
Richard

On Fri, 24 Jul 2020 07:26:23 +0200,
Dr Paul Dale wrote:
> The exact same points apply to EVP_MAC & EVP_KDF.
> 
> We have also been telling people “use EVP” for ages.
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> On 24 Jul 2020, at 3:20 pm, Richard Levitte  wrote:
>
> A couple of points:
>
> 1.  Quite a while ago, we (the team at the time) made a decision to
>have all new APIs prefixed with 'OPENSSL_' or 'OSSL_'.  It seems
>that we never voted on it, though, but still.
>
> 2.  The new RAND API hasn't been merged yet, so it's not like we're
>renaming something that already exists.
>
> So in terms of "it's just a prefix", OSSL_ would be just as suitable.
> It's a bit more blatantly "OpenSSL", though.
>
> Cheers,
> Richard
>
> On Thu, 23 Jul 2020 23:30:25 +0200,
> Tim Hudson wrote:
>
> Placing everything under EVP is reasonable in my view. It is just a 
> prefix and it really
> has no
> meaning these days as it became nothing more than a common prefix to 
> use.
>
> I don't see any significant benefit in renaming at this point - even 
> for RAND.
>    
> Tim.
>
> On Fri, 24 Jul 2020, 1:56 am Matt Caswell,  wrote:
>
>On 23/07/2020 16:52, Richard Levitte wrote:
>
> On Thu, 23 Jul 2020 12:18:10 +0200,
> Dr Paul Dale wrote:
>
> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  
> This seems
> reasonable.  Would
>
>it
>
> also make sense to rename the other new APIs similarly.
> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and 
> OSSL_KDF respectively?
> 
> This is a good question...
>
> Historically speaking, even though EVP_MAC and EVP_KDF are indeed 
> new
> APIs, they have a previous history of EVP APIs, through EVP_PKEY. 
>  The
> impact of relocating them outside of the EVP "family" may be 
> small,
> but still, history gives me pause.
>
> RAND doesn't carry the same sort of history, which makes it much
> easier for me to think "just do it and get it over with"...
> 
>I have the same pause - so  I'm thinking just RAND for now.
>
>Matt
> 
> --
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: API renaming

2020-07-23 Thread Richard Levitte
A couple of points:

1.  Quite a while ago, we (the team at the time) made a decision to
have all new APIs prefixed with 'OPENSSL_' or 'OSSL_'.  It seems
that we never voted on it, though, but still.

2.  The new RAND API hasn't been merged yet, so it's not like we're
renaming something that already exists.

So in terms of "it's just a prefix", OSSL_ would be just as suitable.
It's a bit more blatantly "OpenSSL", though.

Cheers,
Richard

On Thu, 23 Jul 2020 23:30:25 +0200,
Tim Hudson wrote:
> Placing everything under EVP is reasonable in my view. It is just a prefix 
> and it really has no
> meaning these days as it became nothing more than a common prefix to use.
> 
> I don't see any significant benefit in renaming at this point - even for RAND.
> 
> Tim.
> 
> On Fri, 24 Jul 2020, 1:56 am Matt Caswell,  wrote:
> 
> On 23/07/2020 16:52, Richard Levitte wrote:
> > On Thu, 23 Jul 2020 12:18:10 +0200,
> > Dr Paul Dale wrote:
> >> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  This 
> seems reasonable.  Would
> it
> >> also make sense to rename the other new APIs similarly.
> >> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and OSSL_KDF 
> respectively?
> >
> > This is a good question...
> >
> > Historically speaking, even though EVP_MAC and EVP_KDF are indeed new
> > APIs, they have a previous history of EVP APIs, through EVP_PKEY.  The
> > impact of relocating them outside of the EVP "family" may be small,
> > but still, history gives me pause.
> >
> > RAND doesn't carry the same sort of history, which makes it much
> > easier for me to think "just do it and get it over with"...
>
> I have the same pause - so  I'm thinking just RAND for now.
>
> Matt
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: API renaming

2020-07-23 Thread Richard Levitte
On Thu, 23 Jul 2020 12:18:10 +0200,
Dr Paul Dale wrote:
> There has been a suggestion to rename EVP_RAND to OSSL_RAND.  This seems 
> reasonable.  Would it
> also make sense to rename the other new APIs similarly.
> More specifically, EVP_MAC and EVP_KDF to OSSL_MAC and OSSL_KDF respectively?

This is a good question...

Historically speaking, even though EVP_MAC and EVP_KDF are indeed new
APIs, they have a previous history of EVP APIs, through EVP_PKEY.  The
impact of relocating them outside of the EVP "family" may be small,
but still, history gives me pause.

RAND doesn't carry the same sort of history, which makes it much
easier for me to think "just do it and get it over with"...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Naming conventions

2020-07-06 Thread Richard Levitte
On Fri, 03 Jul 2020 11:25:37 +0200,
Matt Caswell wrote:
> On 19/06/2020 08:15, Tomas Mraz wrote:
> > to something like:
> > 
> > int EVP_MacInit(EVP_MAC_CTX *ctx);
> >   
> > int EVP_MacUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> > datalen);
> >  
> > int EVP_MacFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> > outsize);
> > 
> > or at least
> > 
> > int EVP_MACInit(EVP_MAC_CTX *ctx);
> >   
> > int EVP_MACUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> > datalen);
> >  
> > int EVP_MACFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> > outsize);
> > 
> > Should we do that? I hope for the sheer ugliness of the supposedly
> > consistent names that we do not.
> 
> This pattern is not universally used (as you mention the EVP_PKEY
> API does something different).

Er, you've choose what you wanted to read, I suppose...  For fairly
high level EVP APIs, the CamelCase form is actually quite consistent.
For EVP_PKEY, we have them covering most if not all higher level
usages:

EVP_Sign{Init, Update, Final}
EVP_Verify{Init, Update, Final}
EVP_Open{Init, Update, Final}
EVP_Seal{Init, Update, Final}

> I remain strongly opposed to this renaming as I really don't think it
> helps to do this sort of thing now. It just introduces significant
> confusion without a long term strategy. We are too close to 3.0 beta1 to
> embark on this journey now. I'm also not convinced that strategically
> this is the right pattern to use.

What I hear from this is that 3.0 is the wrong time to introduce a new
(and hopefully better) public API, that we should post-pone that to
something like 4.0.  I can get along with that line of thought.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Naming conventions

2020-06-29 Thread Richard Levitte
This discussion seems to have gone stale.

As far as I can read the thread, there are three lines of thought at
play (in no special order):

- the API put forth in #11996 and #11997
- the API exemplified with EVP_MAC and EVP_KDF before #11996 and #11997
- the API exemplified by functions in CamelCase

I'm uncertain if we mean to say that only new EVP features (sometimes
called sub-APIs) should be affected by whatever we decide, or if we
should make appropriate aliases for older EVP features as well (one
might argue that the CamelCase functions pave a way that avoids such
aliases).

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Naming conventions

2020-06-19 Thread Richard Levitte
On Fri, 19 Jun 2020 09:15:22 +0200,
Tomas Mraz wrote:
> The problem is that there is not really fully consistent convention
> followed currently (even in 1.1.1, not counting the API additions in
> 3.0).
> 
> For example if we would like to follow the convention _fully_ we would
> have to rename these new EVP_MAC functions:
> 
> int EVP_MAC_init(EVP_MAC_CTX *ctx);
>   
> int EVP_MAC_update(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> datalen);
>  
> int EVP_MAC_final(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> outsize);
> 
> to something like:
> 
> int EVP_MacInit(EVP_MAC_CTX *ctx);
>   
> int EVP_MacUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> datalen);
>  
> int EVP_MacFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> outsize);
> 
> or at least
> 
> int EVP_MACInit(EVP_MAC_CTX *ctx);
>   
> int EVP_MACUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
> datalen);
>  
> int EVP_MACFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, size_t 
> outsize);

Someone's walking memory lane ;-)
I had to look back at OpenSSL_0_9_1c to remind myself, 'cause today,
we're quite mixed up, what with the slightly lower level functions
EVP_PKEY_sign_init, EVP_PKEY_sign, etc etc etc...

I would say, thought, that if you want to do the higher level function
thing (which are all CamelCase as far as I can see), there's another
naming convention going on, at least with EVP_PKEY, it seems to be
done according to the higher level operation you want to perform, not
the type of data used to do it...  what do you normally to with a MAC? 
So:

int EVP_AuthenticateInit(EVP_MAC_CTX *ctx);
int EVP_AUthenticateUpdate(EVP_MAC_CTX *ctx, const unsigned char *data, size_t 
datalen);
int EVP_AUthenticateFinal(EVP_MAC_CTX *ctx, unsigned char *out, size_t *outl, 
size_t outsize);

... or something like that.  Quite frankly, a naming convention that
is about the operation rather than the type of any type is something I
could play along with.

> And I actually hope we could add at least non-CamelCase aliases to the
> existing (non-deprecated) CamelCase functions because they were always
> the worst offender of the API consistency.

I agree with you...  but we have to recognise that would be a bigger
remake, and if we do look at just the CamelCase API, it's actually
fairly consistent (turning a blind eye at DigestSign quirkiness when I
say that).

(I suppose that CamelCase was inspired by the time, it was quite
popular in certain circles in the 90's, and got especially popularised
by Microsoft...  and well, there are quite a few Microsoft ideas that
have infiltrated OpenSSL...  need I mention '_ex'?)

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Naming conventions

2020-06-18 Thread Richard Levitte
On Thu, 18 Jun 2020 16:36:30 +0200,
Matt Caswell wrote:
> On 18/06/2020 13:03, Richard Levitte wrote:
> > As for not doing something piecemeal, I actually disagree, I see
> > benefit in more than just one person getting their hands dirty (plus
> > changing everything in one go may be overwhelming, and would make for
> > another huge PR that takes overly much time to get through).  However,
> > as strategies go, and if we agree on making the change, we could very
> > well create an issue for each affected sub-API and have it point at a
> > common page that describes what we agreed upon...  this could be a
> > good use of the github wiki tab, for example.
> 
> I don't mean piecemeal in the sense of doing it spread over a number of
> PRs. I mean piecemeal in the sense of doing it spread over a number of
> releases. As far as I can tell #11996 and #11997 were one offs without
> any long term strategy in mind to convert the whole API in this way.

Ah, ok.  I agree with you there, if we're doing this, we should do it
consistently for the same release.

> > When do you see that time being, then?  3.1 (we've talked about it
> > being a "cleanup" release)?  4.0?
> 
> Perhaps never. But if we do it then either 3.1 or 4.0 could be
> considered. I am yet to be convinced that its worth it.

I actually have a different idea, but that's much more further in the
future: a consistent libcrypto API across the board, where all
libcrypto functions are in the "namespaces" (i.e. are prefixed with)
OSSL or OPENSSL.  No exception.
That idea would be a fairly complete API remake, and I do think it
would be worth the while.
So uhm, "never" isn't a line of thinking that I'm ready to accept.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Naming conventions

2020-06-18 Thread Richard Levitte
On Thu, 18 Jun 2020 12:36:52 +0200,
Matt Caswell wrote:
> 
> EVP_MAC_CTX_new -> EVP_MAC_new_ctx
> EVP_MAC_CTX_free -> EVP_MAC_free_ctx
> EVP_MAC_CTX_dup -> EVP_MAC_dup_ctx
> EVP_MAC_CTX_mac -> EVP_MAC_get_ctx_mac
> EVP_MAC_CTX_get_params -> EVP_MAC_get_ctx_params
> EVP_MAC_CTX_set_params -> EVP_MAC_set_ctx_params
> 
> There are similar renamings for the KDF APIs.
> 
> I am opposed to this renaming on the basis that it is inconsistent with
> what we do elsewhere.

Okie, if we're going to start this discussion with taking a stand, I
guess I'll declare that while I initially had the exact same concern,
I now see this change in a positive light.  This comment from #11997
got me to change my mind:

The already established patterns is a furphy. I'll reword it more
precisely: we've done things badly in the past, thus we should do
them badly moving forwards. We've always been at war with
Eurasia.oh wrong context, it's Eastasia.

Ref: https://github.com/openssl/openssl/pull/11997#issuecomment-636658901

For historical background, changing the function name pattern isn't a
new discussion, it's even been brought up on this list more than a
year ago: 
https://mta.openssl.org/pipermail/openssl-project/2019-March/001280.html
Unfortunately, those discussion never got much traction.  I'm not at
all surprised that #11996 and #11997 came to be, it follows a fairly
human pattern that when talking leads nowhere and someone is still
engaged enough, action will happen.

When it comes to EVP_MAC and EVP_KDF, those sub-APIs already break
with existing patterns, albeit more subtly.  They have a _dup
function, where previous sub-APIs have a _copy function.  But that's
perhaps small enough to be acceptable.

> For example, except for the MAC/KDF APIs I see
> there are 26 functions of the form:
> 
> FOO_CTX_new
> 
> The case is similar for FOO_CTX_free. Aside from the newly introduced
> MAC/KDF names there are no other instances of FOO_new_ctx or
> FOO_free_ctx. This is totally inconsistent and, IMO, will cause
> significant confusion for our users.

Sure, but it counters the confusion with when to use FOO and when to
use FOO_CTX as a prefix.  You and I seem to be pretty clear on this,
but I've seen enough questions on the matter to see the benefit with
this change, even though it breaks the old pattern.

> "If we want to change the naming conventions then we should do so only
> after agreeing what those conventions should be, and agreeing a strategy
> for how we are going to apply them. It should not be done piecemeal (and
> hidden away in a PR that supposedly made things more consistent but in
> reality did the opposite)."

I agree that it would have been good to discuss it further first.
However, those discussion have been met with utter silence before, so
that wasn't very fruitful.  I guess that #11996 and #11997 was enough
of a slap to wake us up.

As for not doing something piecemeal, I actually disagree, I see
benefit in more than just one person getting their hands dirty (plus
changing everything in one go may be overwhelming, and would make for
another huge PR that takes overly much time to get through).  However,
as strategies go, and if we agree on making the change, we could very
well create an issue for each affected sub-API and have it point at a
common page that describes what we agreed upon...  this could be a
good use of the github wiki tab, for example.

> There *are* inconsistencies and quirks in our current naming
> conventions. I am skeptical that now is the time to resolve them given
> the number of other changes we are already introducing into 3.0.

When do you see that time being, then?  3.1 (we've talked about it
being a "cleanup" release)?  4.0?

Each choice has its pros and cons:

- Pros for 3.1: we get the added API variants out fairly early.
- Cons for 3.1: because of ABI compatibility concerns, the old
  functions will have to be preserved as is and the new will have to
  be implemented as macros or new functions, making for a bigger
  libcrypto.num.

- Pros for 4.0: ABI concerns aren't a factor, so the old functions can
  be renamed, and we can preserve the old names as macros.  No need
  for double entries in libcrypto.num.
- Cons for 4.0: By that's far away, which means that we solidify
  the old pattern even more before remaking it.


Side note: if we get through a rename fest, can we get rid of
CamelCase?

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Monthly Status Report (May 2020)

2020-06-16 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - [not_yet_merged] [WIP] Refactor CPUID code
(PR openssl/openssl#11311)
  - EVP: Only use the engine when one is defined, in pkey_mac_ctrl()
(PR openssl/openssl#11674)
  - WPACKET: don't write DER length when we don't want to
(PR openssl/openssl#11703)
  - util/perl/OpenSSL/OID.pm: remove the included unit test
(PR openssl/openssl#11704)
  - Fix reason code clash
(PR openssl/openssl#11708)
  - PSS pack: Add provider support for PSS parameters
(PR openssl/openssl#11710)
  - Configure: avoid perl regexp bugs
(PR openssl/openssl#11737)
  - EVP: when setting the operation to EVP_PKEY_OP_UNDEFINED, clean up!
(PR openssl/openssl#11750)
  - Warn that objects with opaque types must never be passed on.
(PR openssl/openssl#11754)
  - OSSL_STORE additions
(PR openssl/openssl#11756)
  - Fix some misunderstandings in our providers' main modules
(PR openssl/openssl#11777)
  - Fix d2i_PrivateKey_ex() to work as documented
(PR openssl/openssl#11787)
  - Fix CHANGES.md issues reported by markdownlint
(PR openssl/openssl#11788)
  - Remove explicit dependency on configdata.pm when processing .in files
(PR openssl/openssl#11790)
  - PROV: Add a proper provider context structure for OpenSSL providers
(PR openssl/openssl#11803)
  - SSL: refactor ssl_cert_lookup_by_pkey() to work with provider side keys
(PR openssl/openssl#11828)
  - test/evp_extra_test.c: Add OPENSSL_NO_CMAC around CMAC test
(PR openssl/openssl#11833)
  - CORE: Fix a couple of bugs in algorithm_do_this()
(PR openssl/openssl#11837)
  - CORE: query for operations only once per provider (unless no_store is true)
(PR openssl/openssl#11842)
  - Add OSSL_PROVIDER_do_all()
(PR openssl/openssl#11858)
  - Refactor the provider side DER constants and writers
(PR openssl/openssl#11868)
  - rsa_padding_add_PKCS1_OAEP_mgf1_with_libctx(): fix check of |md|
(PR openssl/openssl#11869)
  - STORE: Make try_decode_PrivateKey() ENGINE aware
(PR openssl/openssl#11872)
  - Fix d2i_PrivateKey() to work as documented [1.1.1]
(PR openssl/openssl#11888)
  - PROV: Fix RSA-OAEP memory leak
(PR openssl/openssl#11927)
  - Add header file docs for openssl/core_numbers.h and openssl/core_names.h
(PR openssl/openssl#11963)
  - util/mkpod2html.pl: Fix unbalanced quotes
(PR openssl/openssl#11969)
  - Fix EVP_CIPHER_fetch race condition
(PR openssl/openssl#11977)
  - [not_yet_merged] [WIP] EVP: retrieve EVP_CIPHER constants in the 
evp_cipher_from_dispatch()
(PR openssl/openssl#11980)

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (April 2020)

2020-06-16 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - EXPERIMENTAL: remove all the legacy bits from a EVP_PKEY
(PR openssl/openssl#10797)
  - DOC: Add more description of EVP_PKEY_fromdata(), and examples
(PR openssl/openssl#11220)
  - [not_yet_merged] [WIP] Incorporate system guessing in Configure
(PR openssl/openssl#11230)
  - EC  param / key generation
(PR openssl/openssl#11328)
  - EVP & TLS: Add necessary EC_KEY data extraction functions, and use them
(PR openssl/openssl#11358)
  - [not_yet_merged] [WIP] PKCS#7 and CMS with provider side keys
(PR openssl/openssl#11422)
  - PROV: DER writer
(PR openssl/openssl#11450)
  - CMS KARI: Temporarly downgrade newly generated EVP_PKEYs to legacy
(PR openssl/openssl#11501)
  - OpenSSL::OID: Don't use List::Util
(PR openssl/openssl#11503)
  - [URGENT] Add common internal crypto/ modules in liblegacy.a
(PR openssl/openssl#11504)
  - PROV: Remove duplicate in .../ciphers/build.info
(PR openssl/openssl#11513)
  - Rename FIPS_MODE to FIPS_MODULE
(PR openssl/openssl#11539)
  - DOC: Refactor provider-keymgmt(7) to give the keytypes their own pages
(PR openssl/openssl#11546)
  - EVP: Fix calls to evp_pkey_export_to_provider()
(PR openssl/openssl#11550)
  - INSTALL: document 'no-ui-console' rather than 'no-ui'
(PR openssl/openssl#11553)
  - TEST: make and use a fipsinstall script
(PR openssl/openssl#11565)
  - Build files: add module installation targets
(PR openssl/openssl#11566)
  - EVP: Fix EVP_Digest{Sign,Verify}Init() to handle no default digest 
(PR openssl/openssl#11576)
  - Make test/fipsinstall.pl when running via the test harness
(PR openssl/openssl#11591)
  - Fix dev/release-aux-openssl-announce-pre-release.tmpl
(PR openssl/openssl#11617)
  - Configure: Allow quoted values in VERSION
(PR openssl/openssl#11624)
  - Configurations/windows-makefile.tmpl: Fix template code for INSTALL_MODULES
(PR openssl/openssl#11629)
  - fuzz/asn1.c: Add missing #include
(PR openssl/openssl#11640)
  - Configurations/unix-Makefile.tmpl: fix typo
(PR openssl/openssl#11654)
  - include/openssl/x509v3.h: restore previous stack definition arrangement
(PR openssl/openssl#11655)
  - crypto/x509/v3_alt.c: make 'othername' a bit bigger
(PR openssl/openssl#11656)
  - Configure: change all references to INSTALL to INSTALL.md
(PR openssl/openssl#11657)
  - EVP: Fix evp_keymgmt_util_copy() for to->keymgmt == NULL
(PR openssl/openssl#11668)

* Administration

  - Create the omc-tools repo, as per vote concluded 2020-03-04
  - Move a set of directories to omc-tools, as per vote concluded 2020-03-04
  - Move release-tools/do-release.pl to omc-tools
  - Give OTC push access to tools
  - mediawiki: add the NumberHeadings extension

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (March 2020)

2020-06-16 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - [not_yet_merged] WIP: apps: Switch to using OSSL_STORE for loading keys, 
certs, ...
(PR openssl/openssl#7390)
  - Implement domparam and key generation
(PR openssl/openssl#10289)
  - DOC: Add documentation related to X509_LOOKUPs [1.1.1]
(PR openssl/openssl#11120)
  - Refactor CRMF_poposigningkey_init() to work with provider keys
(PR openssl/openssl#11126)
  - Configuration: Add post module building check
(PR openssl/openssl#11170)
  - build.info extensions: variable value substitutions and multi-item 
statements
(PR openssl/openssl#11185)
  - .travis.yml: where it matters, have build and source nesting levels differ
(PR openssl/openssl#11186)
  - crypto/perlasm/x86_64-xlate.pl: detect GNU as to deal with quirks
(PR openssl/openssl#11191)
  - EVP: Check that key methods aren't foreign when exporting
(PR openssl/openssl#11193)
  - Andoid cross compile: change ANDROID_NDK_HOME to ANDROID_NDK_ROOT
(PR openssl/openssl#11206)
  - config, Configure: move the check of removed crypto/ sub-systems
(PR openssl/openssl#11217)
  - Configurations: Fix "android" configuration target
(PR openssl/openssl#11238)
  - util/wrap.pl: do not look at EXE_SHELL
(PR openssl/openssl#11258)
  - Restructure documentation of implementations in providers
(PR openssl/openssl#11270)
  - DOCS: Fix documentation on asymmetric keydata types
(PR openssl/openssl#11275)
  - DOCS: Fix the description of OSSL_PARAM_allocate_from_text()
(PR openssl/openssl#11279)
  - Refactor sm2_id
(PR openssl/openssl#11302)
  - Fix RSA structure
(PR openssl/openssl#11315)
  - Fix legacy_ctrl_to_param() to pay better attention to keytype
(PR openssl/openssl#11329)
  - Refactor sm2_id - addendum
(PR openssl/openssl#11335)
  - EVP: fetch the EVP_KEYMGMT earlier
(PR openssl/openssl#11343)
  - [WIP] EVP: Fix EVP_PKEY_copy_parameters() for a newly allocated |to|
(PR openssl/openssl#11368)
  - DH, DSA, EC_KEY: Fix exporters to allow domain parameter keys
(PR openssl/openssl#11374)
  - EVP: Downgrade keys rather than upgrade
(PR openssl/openssl#11375)
  - util/wrap.pl: Correct exit code when signalled
(PR openssl/openssl#11379)
  - EC: Refactor ec_curve_name2nid() to accept NIST curve names
(PR openssl/openssl#11391)
  - PROV: Fix EC_KEY exporters to allow domain parameter keys
(PR openssl/openssl#11394)

* Web

  - Fix 'make relupd'

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (February 2020)

2020-06-16 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - PROV: add RSA signature implementation
(PR openssl/openssl#10557)
  - Disable devcryptoeng on newer OpenBSD versions [1.1.1]
(PR openssl/openssl#10565)
  - EVP: Adapt EVP_PKEY checking, comparing and copying for provider keys
(PR openssl/openssl#10807)
  - DOC: document in more detail what a BIO_read_ex() via BIO_f_buffer() does
(PR openssl/openssl#10890)
  - Refactor SM2
(PR openssl/openssl#10942)
  - Decentralize legacy_ctrl_str_to_param()
(PR openssl/openssl#10947)
  - config: ensure the perl Configure run is the last statement
(PR openssl/openssl#10953)
  - EVP: Small refactor of keymgmt library code
(PR openssl/openssl#10963)
  - DOC: Add documentation related to X509_LOOKUPs
(PR openssl/openssl#10986)
  - Redesign the KEYMGMT libcrypto <-> provider interface
(PR openssl/openssl#11006)
  - EVP: Adapt EVP_PKEY checking, comparing and copying for provider keys, take 
2
(PR openssl/openssl#11025)
  - Configure: Add easy to use disabled deprecated functionality indicators
(PR openssl/openssl#11027)
  - PROV: Ensure the AlgorithmIdentifier registers in DSA signature impl
(PR openssl/openssl#11037)
  - X509_PUBKEY_set(): Fix memory leak
(PR openssl/openssl#11038)
  - test/recipes/80-test_ssl_old.t: use 'openssl genpkey'
(PR openssl/openssl#11041)
  - Make util/find-doc-nits runnable from the build tree
(PR openssl/openssl#11045)
  - Refactor OSSL_PARAM_allocate_from_text() for better flexibility
(PR openssl/openssl#11046)
  - Add OSSL_SERIALIZER_PUBKEY_TO_DER_PQ and friends
(PR openssl/openssl#11055)
  - Adapt i2d_PrivateKey for provider only keys
(PR openssl/openssl#11056)
  - Document OSSL_SERIALIZER_PUBKEY_TO_DER_PQ and friends
(PR openssl/openssl#11071)
  - Refactor evp_pkey_make_provided() to do legacy to provider export
(PR openssl/openssl#11074)
  - Adapt i2d_PUBKEY for provider only keys
(PR openssl/openssl#11078)
  - TEST: Create test specific output directories
(PR openssl/openssl#11080)
  - include/openssl/whrlpool.h: correct unbalanced deprecation guards
(PR openssl/openssl#11087)
  - Fix VMS build [1.1.1 only]
(PR openssl/openssl#11088)
  - PROV: Build the main FIPS module code with FIPS_MODE defined
(PR openssl/openssl#11090)
  - TEST: add util/wrap.pl and use it #0
(PR openssl/openssl#0)
  - Rethink the EVP_PKEY cache of provider side keys
(PR openssl/openssl#11148)
  - VMS: mitigate for the C++ compiler that doesn't understand certain pragmas
(PR openssl/openssl#11159)
  - Deprecate ASN1_sign(), ASN1_verify() and ASN1_digest()
(PR openssl/openssl#11161)
  - Fix util/mktar.sh to use the new VERSION information
(PR openssl/openssl#11190)

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (January 2020)

2020-06-16 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - [not_yet_merged] WIP: OSSL_STORE for providers
(PR openssl/openssl#9389)
  - CORE & EVP: Adapt KEYEXCH, SIGNATURE and ASYM_CIPHER to handle key types 
better
(PR openssl/openssl#10647)
  - Configuration: synchronise the variables on the build file templates
(PR openssl/openssl#10753)
  - EVP: Fix method to determine if a PKEY is legacy or not
(PR openssl/openssl#10758)
  - DOCS: The interpretation of OPENSSL_API_COMPAT has changed, update docs
(PR openssl/openssl#10765)
  - Add missing inclusion of "internal/deprecated.h"
(PR openssl/openssl#10766)
  - EVP: If a key can't be exported to provider, fallback to legacy
(PR openssl/openssl#10771)
  - Add the DSA serializers to the default provider tools
(PR openssl/openssl#10772)
  - EVP: make EVP_PKEY_{bits,security_bits,size} work with provider only keys
(PR openssl/openssl#10778)
  - PROV: Fix mixup between general and specialized GCM implementations
(PR openssl/openssl#10783)
  - Configure: use $list_separator_re only for defines and includes
(PR openssl/openssl#10793)
  - Eliminate some EVP_PKEY_size() uses
(PR openssl/openssl#10798)
  - EVP: clear error when falling back from failed EVP_KEYMGMT_fetch()
(PR openssl/openssl#10803)
  - CORE: renumber OSSL_FUNC_KEYMGMT macros
(PR openssl/openssl#10804)
  - Fix documentation for EVP_DigestSign* and EVP_DigestVerify*
(PR openssl/openssl#10805)
  - Fix EVP_Digest{Sign,Verify}Final() and EVP_Digest{Sign,Verify}() for 
provider only keys
(PR openssl/openssl#10806)
  - EVP: Adapt EVP_PKEY Seal and Open for provider keys
(PR openssl/openssl#10808)
  - Move the definition of OPENSSL_BUILDING_OPENSSL
(PR openssl/openssl#10813)
  - Change returned -2 to 0 in EVP_Digest{Sign,Verify}Init()
(PR openssl/openssl#10815)
  - Add EVP_PKEY_get_default_digest_name()
(PR openssl/openssl#10824)
  - CRYPTO: Remove support for ex_data fields when building the FIPS module
(PR openssl/openssl#10837)
  - Modify EVP_CIPHER_is_a() and EVP_MD_is_a() to handle legacy methods too
(PR openssl/openssl#10845)
  - Move the stored namemap pre-population to namemap construction
(PR openssl/openssl#10846)
  - [1.1.1] Fix documentation of return value for EVP_Digest{Sign,Verify}Init()
(PR openssl/openssl#10847)
  - Build file templates: Use explicit files instead of $< or $? for pods
(PR openssl/openssl#10849)
  - EVP: Add evp_pkey_make_provided() and refactor around it
(PR openssl/openssl#10850)
  - Adapt X509_PUBKEY_set() for use with provided implementations
(PR openssl/openssl#10851)
  - For all assembler scripts where it matters, recognise clang > 9.x
(PR openssl/openssl#10855)
  - Add GNU properties note for Intel CET in x86_64-xlate.pl
(PR openssl/openssl#10875)
  - Configure: Better detection of '-static' in @{$config{LDFLAGS}}
(PR openssl/openssl#10878)
  - PROV: Fix bignum printout in text serializers
(PR openssl/openssl#10891)
  - OpenSSL::Test: bring back the relative paths
(PR openssl/openssl#10913)
  - Adapt ASN1_item_sign_ctx() for use with provided keypairs
(PR openssl/openssl#10920)
  - Add internal maxsize macros
(PR openssl/openssl#10928)
  - test/recipes/30-test_evp.t: Fix multiple definition of @bffiles
(PR openssl/openssl#10944)

* Administration

  - Stop making snaps for 1.1.0 and 1.0.2, and make 3.0-dev snaps
  - Switch final review to be for OTC rather than OMC

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: The hold on PR 12089

2020-06-10 Thread Richard Levitte
On Wed, 10 Jun 2020 10:55:36 +0200,
Nicola Tuveri wrote:
> The matter of OMC Vs OTC vote also depends on what kind of hold Tim
> is applying with his - 1: is it a OMC or a OTC hold?

The label that Tim added should make it clear: "hold: need omc decision"

I've started a discussion within the OMC.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: addrev warning

2020-06-08 Thread Richard Levitte
Yes.

I propose telling git to shut up, i.e. FILTER_BRANCH_SQUELCH_WARNING=1
Our use of git-filter-branch isn't one that will generate a gotcha, or
we would have seen that a long time ago...  in other words, we treat
it gently.

I was looking at git-filter-repo, but realised that it's not a
standard git thing, so if we changed addrev to use that instead, we'd
have addrev starting to fail completely for everyone that hasn't
installed that one yet.  I'm not prepared to spring that surprise on
anyone.

Cheers,
Richard

On Mon, 08 Jun 2020 12:16:40 +0200,
Matt Caswell wrote:
> 
> After upgrading Ubuntu over the weekend I'm suddenly seeing this warning
> from addrev. Is anyone else getting this?
> 
> WARNING: git-filter-branch has a glut of gotchas generating mangled history
>rewrites.  Hit Ctrl-C before proceeding to abort, then use an
>alternative filtering tool such as 'git filter-repo'
>(https://github.com/newren/git-filter-repo/) instead.  See the
>filter-branch manual page for more details; to squelch this warning,
>set FILTER_BRANCH_SQUELCH_WARNING=1.
> Proceeding with filter-branch...
> 
> 
> MMatt
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Travis jobs not getting started

2020-06-06 Thread Richard Levitte
This is something that has been ongoing for a while...  it was in the
back of my mind a while ago, but then I got distracted.

Thanks for finding that document...  I failed to when I looked a
couple of weeks ago.

Migration is now done.  It won't show on existing PRs, but does show
on new ones.

The new Travis is a Github App, which means that it's much more
integrated into Github; among others, that shows when you press
"details", and it's also available via the new per-PR "Checks" tab.

Cheers,
Richard

On Fri, 05 Jun 2020 13:27:20 +0200,
Tomas Mraz wrote:
> 
> Apparently the travis jobs are not getting started anymore for some
> reason.
> 
> It happened to me on the GitHub linux-pam project and the solution (or
> workaround) was to migrate the project to the travis-ci.com.
> 
> I just did it by following the instructions in this document:
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration/
> 
> -- 
> Tomáš Mráz
> No matter how far down the wrong road you've gone, turn back.
>   Turkish proverb
> [You'll know whether the road is wrong if you carefully listen to your
> conscience.]
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: OMC VOTE: Drop the 'openssl' interactive mode as per GH #12023

2020-06-05 Thread Richard Levitte
4 voted for, none against, no abstencions, 3 have not yet voted.

This PR will be merged.

Cheers,
Richard

On Wed, 03 Jun 2020 12:46:41 +0200,
Richard Levitte wrote:
> 
> topic: Drop the 'openssl' interactive mode as per GH #12023
> comment: It is undocumented, pretty much unused, and hard to keep
>  stable.
> Proposed by Richard Levitte
> Public: yes
> 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


OMC VOTE: Drop the 'openssl' interactive mode as per GH #12023

2020-06-03 Thread Richard Levitte
topic: Drop the 'openssl' interactive mode as per GH #12023
comment: It is undocumented, pretty much unused, and hard to keep
 stable.
Proposed by Richard Levitte
Public: yes

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/



Re: error conversion macro's

2020-05-15 Thread Richard Levitte
We can, given time.  This is one of those things that can be saved
for the beta period, since it doesn't affect anything public.  But
sure, if someone feel inclined to do that earlier, I don't see why
that should be a problem.

Cheers,
Richard

On Fri, 15 May 2020 16:00:49 +0200,
Salz, Rich wrote:
> Perhaps the next Alpha release can do the error macro conversion?
> 
> Add script to convert XXXerr() to XXXerror, 
> https://github.com/openssl/openssl/pull/9441
> 
> In particular, 
> https://github.com/openssl/openssl/pull/9441#issuecomment-522171044
> 
> It might need updating if any new “error facility” values have been added, 
> but I didn’t notice any
> of them.
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Alpha2

2020-05-14 Thread Richard Levitte
On Wed, 13 May 2020 18:00:35 +0200,
Matt Caswell wrote:
> 
> On 08/05/2020 09:55, Matt Caswell wrote:
> > Various OpenSSL 3.0 developers met (virtually) on Tuesday to discuss
> > current progress. It was proposed that we should do the Alpha 2 release
> > next week (on Thursday 14th May). Unless I hear objections otherwise, I
> > plan to go with that.
> 
> I'm currently thinking that we're not quite ready for alpha2 and I would
> like to push it by a day. Any objections?

Not from me...  among others, I want to see #11758 in there...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Alpha2

2020-05-08 Thread Richard Levitte
You might want to consider reviewing #11630, an reminding me what else
needs to get fixed in there.

On Fri, 08 May 2020 10:55:19 +0200,
Matt Caswell wrote:
> 
> Various OpenSSL 3.0 developers met (virtually) on Tuesday to discuss
> current progress. It was proposed that we should do the Alpha 2 release
> next week (on Thursday 14th May). Unless I hear objections otherwise, I
> plan to go with that.
> 
> Matt
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Technically an API break

2020-05-07 Thread Richard Levitte
On Thu, 07 May 2020 10:31:42 +0200,
Tomas Mraz wrote:
> 
> On Thu, 2020-05-07 at 09:24 +0100, Matt Caswell wrote:
> > PR11589 makes a change to the public API function
> > `SSL_set_record_padding_callback` to change its return type from void
> > to
> > int:
> > 
> > https://github.com/openssl/openssl/pull/11589
> > 
> > This is technically an API break - but it doesn't seem too serious.
> > It's
> > possible, I suppose, that existing applications that use this will
> > fail
> > to spot the error return since this function can now fail. The
> > function
> > itself was only recently added (in 1.1.1), and I suspect real-world
> > usage is very small (or possibly nil).
> > 
> > Is this considered ok?
> 
> I would say this is an acceptable thing if it is documented (which it
> is in the PR). Especially because the error return can happen only when
> the application sets up the SSL to use kernel TLS.

I agree with this assessment.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Cherry-pick proposal

2020-04-29 Thread Richard Levitte
; On Wed, 29 Apr 2020 at 14:08, Matt Caswell  wrote:
> 
> The OTC have received this proposal and a request that we vote on it:
>
> I would like to request that we do not allow cherry-picks between master
> and 1.1.1-stable because these two versions are now very different, if a
> cherry-pick succeeds, there is no guarantee that the result will work.
> Because we have no CI for the cherry-pick. If a cherry-pick is needed, a
> new PR for 1.1.1 should be done and approved independently.
>
> Before starting a vote I'd like to provide opportunity for comments, and
> also what the vote text should be.
>
> Thanks
>
> Matt
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Revisiting tradition: branches and tags

2020-04-14 Thread Richard Levitte
On Tue, 14 Apr 2020 09:28:26 +0200,
Dr. Matthias St. Pierre wrote:
> 
> > > Is it possible to set up alias names for the historical branches?
> > 
> > It's possible yes.  The hard part is 1.1.1.  There *is* something
> > called 'git symbolic-ref', but they can only be added in repos we have
> > physical access to, so while can add those on our git server, and they
> > will work, we cannot add them in github.
> > 
> > Ref git help symbolic-ref
> 
> Symbolic references are *not* the right solution to the problem IMO. They are 
> not equivalent to branches.
> Checking out a symbolic reference leaves you in the 'detached HEAD' state:
> 
> msp@msppc:~/src/openssl$ git symbolic-ref ossl111 
> refs/heads/OpenSSL_1_1_1-stable
> msp@msppc:~/src/openssl$ cd ../openssl-1.1.1
> msp@msppc:~/src/openssl-1.1.1$ git checkout ossl111
> Note: switching to 'ossl111'.
> 
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in this
> state without impacting any branches by switching back to a branch.

Okie.  I hadn't experimented with that, so didn't know.  That manual
is fantastically unclear on what this command does too, at least the
way I read it.  I guess that's another nail in its coffin.

> The proper way to do it IMO is to create new branch and tag
> references for all the stable branches resp. release tags.
...
> Currently, the only old-style branch which needs to be synchronized
> is `OpenSSL_1_1_1-stable`. This could easily be done by the git
> post-receive hook on git.openssl.org. In fact, all old-style branch
> and tag references for all eol-branches could be removed
> immediately.

Good point.  The posst-update hook should be usable for this.

> We change the GitHub setup such that pull requests to stable
> branches need to be based onto the new-style branches, but keep the
> old-style stable branches in sync with the new-style branches for a
> little while.

Er...  how do you change that Github setup?  I thought it simply
showed a list of all available branches regardless.  So if we got a
duplicate set, it's going to show both, isn't it?  Considering that
the github repo is a --mirror of the git.openssl.org repo, I'm not
sure how the old branch refs would be filtered out...

It seems to me like this discussion is splitting into two:

1)  Start with new names for 3.0 and on (I can only see positive
responses so far, even with Matt's hesitance)
2)  Rename of the old names.

As far as I can see, those two don't have to be in absolute sync, even
though that would be desirable.  In other words, we can figure out the
details of 2 even after we've started 1.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Revisiting tradition: branches and tags

2020-04-13 Thread Richard Levitte
On Mon, 13 Apr 2020 10:48:35 +0200,
Matt Caswell wrote:
> On 11/04/2020 10:53, Dr. Matthias St. Pierre wrote:
> > I love the new naming scheme, in particular the fact that it's 
> > all-lowercase and does not
> > mix dashes and underscores anymore. I don't recall how often I cursed about 
> > the current
> > scheme which is so typer unfriendly.
> > 
> > I'd like to see *all* stable branch names and release tags converted to 
> > new-style uniformly
> > (keeping the old names for compatibility), so we have an overall consistent 
> > scheme and people
> > don't need to switch back and forth between naming conventions in the 
> > future when doing
> > historical investigations. The old names could be deprecated for later 
> > removal.
> 
> Yes - this aspect was my main hesitation about the proposed new format,
> i.e. the fact that we have a set of existing tags/branch names.
> Constantly changing between the new format and the old format as we look
> at older branches/tags could be painful. Just creating new tags for all
> the existing ones would be one way forward.

New tags is perfectly possible to do.

> Typing OpenSSL_1_1_1-stable and getting all the right upper/lower case
> characters in the right place and making sure to use _ vs - as
> appropriate is a challenge for my fingers which I constantly fail.

I constantly fail the *current* names.

> Is it possible to set up alias names for the historical branches?

It's possible yes.  The hard part is 1.1.1.  There *is* something
called 'git symbolic-ref', but they can only be added in repos we have
physical access to, so while can add those on our git server, and they
will work, we cannot add them in github.

Ref git help symbolic-ref

Cheers,
Richard ( still thinks it's worth making the change with 3.0 )

> 
> Matt
> 
> 
> > 
> > Matthias
> > 
> > 
> > 
> >> -Original Message-
> >> From: openssl-project  On Behalf Of 
> >> Richard Levitte
> >> Sent: Friday, April 10, 2020 8:28 PM
> >> To: openssl-project@openssl.org
> >> Subject: Revisiting tradition: branches and tags
> >>
> >> Once upon a time, there was CVS (*).
> >>
> >> The story could stop there, but since CVS was what was available,
> >> OpenSSL was versioned with CVS.
> >>
> >> Among limitations that came with CVS was the allowed syntax in tag and
> >> branch names; letters, digits, dashes and underscores.  At the time,
> >> eveyone that wanted to encode a version number in a tag had to convert
> >> periods to some other character.  We chose underscores, ending up with
> >> tags like this:
> >>
> >> OpenSSL_0_9_7-beta2
> >> OpenSSL_0_9_7a
> >>
> >> Furthermore, the culture that we have with git, where branches are
> >> being pulled between repositories...  where you can actually have a
> >> multitude of repositories and pull data between them, was very hard if
> >> not practically impossible with CVS.  So, we occasionally had feature
> >> branches for longer term work.  To distinguish our so called stable
> >> branches, we had branch names with '-stable' added at the end:
> >>
> >> OpenSSL_0_9_7-stable
> >>
> >> We *could* have had something shorter, like 'OpenSSL_0_9_7xx', but
> >> I guess that was too easy to mix up with our letter releases.
> >>
> >> With git, however, there's no technical reason why we must maintain
> >> what was originally a technical limitation.  I therefore propose that
> >> we start using tags like this starting with OpenSSL 3.0:
> >>
> >> openssl-3.0.0-alpha1
> >> openssl-3.0.0-beta2
> >> openssl-3.0.0
> >> openssl-3.0.1
> >> openssl-3.1.0
> >>
> >> This is a little more than just avoiding to change the periods with
> >> underscores.  However, if you look at the tar files we've released for
> >> a long time, that's the naming format they use, and I can see benefits
> >> in having the tags for a release match the tar file of the same
> >> release:
> >>
> >> openssl-0.9.7a.tar.gz
> >> openssl-0.9.7a.tar.gz.asc
> >> openssl-0.9.7a.tar.gz.md5
> >>
> >> Future tar files would be numbered with the new version scheme, of
> >> course:
> >>
> >> openssl-3.0.0-alpha1.tar.gz
> >> openssl-3.0.0-beta2.tar.gz
> >> openssl-3.0.0.tar.gz
> >> openssl-3.0.1.tar.gz
> >> openssl-3.1.0.tar.gz
> >>
> >> So 

Revisiting tradition: branches and tags

2020-04-10 Thread Richard Levitte
Once upon a time, there was CVS (*).

The story could stop there, but since CVS was what was available,
OpenSSL was versioned with CVS.

Among limitations that came with CVS was the allowed syntax in tag and
branch names; letters, digits, dashes and underscores.  At the time,
eveyone that wanted to encode a version number in a tag had to convert
periods to some other character.  We chose underscores, ending up with
tags like this:

OpenSSL_0_9_7-beta2
OpenSSL_0_9_7a

Furthermore, the culture that we have with git, where branches are
being pulled between repositories...  where you can actually have a
multitude of repositories and pull data between them, was very hard if
not practically impossible with CVS.  So, we occasionally had feature
branches for longer term work.  To distinguish our so called stable
branches, we had branch names with '-stable' added at the end:

OpenSSL_0_9_7-stable

We *could* have had something shorter, like 'OpenSSL_0_9_7xx', but
I guess that was too easy to mix up with our letter releases.

With git, however, there's no technical reason why we must maintain
what was originally a technical limitation.  I therefore propose that
we start using tags like this starting with OpenSSL 3.0:

openssl-3.0.0-alpha1
openssl-3.0.0-beta2
openssl-3.0.0
openssl-3.0.1
openssl-3.1.0

This is a little more than just avoiding to change the periods with
underscores.  However, if you look at the tar files we've released for
a long time, that's the naming format they use, and I can see benefits
in having the tags for a release match the tar file of the same
release:

openssl-0.9.7a.tar.gz
openssl-0.9.7a.tar.gz.asc
openssl-0.9.7a.tar.gz.md5

Future tar files would be numbered with the new version scheme, of
course:

openssl-3.0.0-alpha1.tar.gz
openssl-3.0.0-beta2.tar.gz
openssl-3.0.0.tar.gz
openssl-3.0.1.tar.gz
openssl-3.1.0.tar.gz

So what about release branches?  We could actually follow a very
similar new pattern, just replace the last digit with, say, an 'x', to
signify that this branch will contain a series of patch releases:

openssl-3.0.x

In this branch, one would expect to see the tags 'openssl-3.0.0',
'openssl-3.0.1', 'openssl-3.0.2', ...

Earlier today, I submitted a new release script that codifies exactly
this:  https://github.com/openssl/openssl/pull/11516

Thoughts?

Cheers,
Richard

(*) Well, not quite, there was RCS, there was SCCS, and a few other
versioning systems that would make your skin crawl by today's
standards, even more so than CVS.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


RE: Release done

2020-03-17 Thread Richard Levitte
The problem was running "make relupd" in the web worktree. It assumes CHANGES 
and NEWS in all branches.
We could have discovered this earlier... 

"Dr. Matthias St. Pierre"  skrev: (17 mars 2020 
19:53:12 CET)
>> Problems were due to:
>> ...
>> - Rename of CHANGES/NEWS to CHANGES.md/NEWS.md
>
>Strange  what exactly was your problem?
>
>I anticipated this problem on the weekend and checked the release
>scripts.
>After some investigation, I came to the conclusion that it wouldn't be
>a problem
>for the OpenSSL_1_1_1-stable branch yet, because the markdown revamping
>happened only on master. What did I miss?
>
>Matthias
>
>
>P.S.: As a byproduct of my investigations I created
>https://github.com/openssl/tools/pull/64

-- 
Skickat från min Android-enhet med K-9 Mail. Ursäkta min fåordighet.


Re: An OpenSSL cookbook, where and how?

2020-03-07 Thread Richard Levitte
On Sat, 07 Mar 2020 10:01:59 +0100,
Richard Levitte wrote:
> 
> As part of a documentation effort, more specifically in some
> discussions in https://github.com/openssl/openssl/pull/11220 (exact
> references below), I've floated the idea that bigger coding examples
> should be placed into a cookbook.

I forgot the references:

https://github.com/openssl/openssl/pull/11220#discussion_r386751635
https://github.com/openssl/openssl/pull/11220#issuecomment-596060267

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


An OpenSSL cookbook, where and how?

2020-03-07 Thread Richard Levitte
As part of a documentation effort, more specifically in some
discussions in https://github.com/openssl/openssl/pull/11220 (exact
references below), I've floated the idea that bigger coding examples
should be placed into a cookbook.

My reasoning for this is very simple: example code in reference
manuals should be kept minimal and focused on the functions documented
in that same page.  Anything that start involving too much other
functions becomes a distraction, or will leave you wondering why the
example is in *this* page and not *the other page* that describes
those other functions.  In my mind, this is education 101.

However, it's true that people may want to see more complete examples,
where the interaction between different sets of functions is on
display, and could serve as code to pick and use, more than the quick
minimalistic examples.  A cookbook.

So if we should do this, where do we want that cookbook?  Who keeps it
up to date, and how?  https://wiki.openssl.org/ could be one answer,
but is it the answer we want?

Thoughts welcome!

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecations

2020-02-22 Thread Richard Levitte
On Sat, 22 Feb 2020 00:51:17 +0100,
Kurt Roeckx wrote:
> Some equivalants:
> openssl dhparam 2048
> openssl genpkey -genparam --algorithm DH -pkeyopt dh_paramgen_prime_len:2048
> 
> openssl dsaparam 2048
> openssl genpkey -genparam -algorithm DSA -pkeyopt dsa_paramgen_bits:2048

Side note: I never quite understood why we had to have such verbose
pkey opts.  "prime_len" and "bits" would have been enough, the rest is
known by context (the command line already specifies that it wants to
generate domain parameters and that the algorithm is DH, or DSA)

I have to agree with Viktor that some of those pkey commands are
overly complicated at times...  it's a bit hard to undo at this point,
though, apart from creating an entirely new openssl command with a
different, and possibly more intuitive interface.

Something that could be done is to take all those aged commands and
rewrite them as wrappers for genpkey, pkey and pkeyutl.  Simply create
and populate a new argv and call genpkey_main(), pkey_main() or
pkeyutl_main().

std::mantra: PR welcome!

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecations

2020-02-22 Thread Richard Levitte
On Sat, 22 Feb 2020 00:03:05 +0100,
Matt Caswell wrote:
> On 21/02/2020 20:33, Matthew Lindner wrote:
> > I think any deprecated apps or options that must be kept and not
> > implemented in the new interface should print a warning when used in
> > that case then, and only do that when it's impossible to implement it
> > in the current release.
> 
> I agree with this. We already do that if you use a deprecated
> application. I'm not so sure we've been quite so careful with the
> deprecated options, so we should check that.

We haven't been very careful at all, all we've done so far is to say
that they are deprecated, nothing else said.  So yeah, I completely
agree.  We also need to decide what such an update should look like.

Looking through some manuals on my Linux installation, I find
deprecation notices in:

- the DESCRIPTION section, which seems to happen when all the
  functions described on the page are deprecated.
  Ref: freehostent(3), gets(3)
- the NOTES sections, which seems to be typically done when only some
  of the functions described on the page are deprecated.
  Ref: dlopen(3),
- as part of the function description
  Ref: getwd(3)
- the CONFORMING TO section, when an external standardisation
  (typically POSIX, LSB, ...) has deprecated a function.
  Ref: bzero(3), gets(3)
- a dedicated deprecation section like DEPRECATED INTERFACES
  Ref: ldap(3) (all applicable LDAP manpages, actually, such as
  ldap_abandon(3))

For those that don't have direct access to a Linux box:

http://man7.org/linux/man-pages/man3/freehostent.3.html
http://man7.org/linux/man-pages/man3/gets.3.html
http://man7.org/linux/man-pages/man3/dlopen.3.html
http://man7.org/linux/man-pages/man3/getwd.3.html
http://man7.org/linux/man-pages/man3/bzero.3.html
http://man7.org/linux/man-pages/man3/ldap.3.html
http://man7.org/linux/man-pages/man3/ldap_abandon.3.html

My personal preference is either the whole page deprecation for pages
where all described functions are deprecated (so, early in DESCRIPTION,
like freehostent(3)), or a dedicated section like the LDAP manpages.

> > 
> > That's not at all clear with the speed app for example. (That speed
> > app was deprecated I just found out in this email thread.)
> > 
> That's not actually what I said. The speed app itself is not deprecated.
> There are options to the speed app, which are deprecated. Its quite
> possible we don't print a warning for those - which we should do.

I wholeheartedly think that we should replace the asymmetric tests
with the EVP variants, to be used with the '-evp' option.  Just
dropping them isn't a great solution.
(as a matter of fact, I would turn everything to go through the EVP
interface, whether the '-evp' option has been given or not)

Cheers,
Richard ( std::mantra: PR welcome! )

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecations

2020-02-21 Thread Richard Levitte



"Salz, Rich"  skrev: (21 februari 2020 17:17:54 CET)
>>A small note about the 'no-deprecated' configuration option: this
>is an option to simulate the far future when the deprecated stuff has
>been removed from the public eye. Deprecated openssl commands should
>not build with such a configuration. 
>  
>How can that work?  If the deprecated command calls deprecated API's
>then you'll get build failures.

It works by not building the deprecated command in that configuration.

-- 
Skickat från min Android-enhet med K-9 Mail. Ursäkta min fåordighet.


Re: Deprecations

2020-02-21 Thread Richard Levitte
A small note about the 'no-deprecated' configuration option: this is an option 
to simulate the far future when the deprecated stuff has been removed from the 
public eye. Deprecated openssl commands should not build with such a 
configuration. 

Cheers 
Richard 



Kurt Roeckx  skrev: (21 februari 2020 09:06:39 CET)
>Hi,
>
>We seem to be deprecating a lot of the old APIs, which I think is
>a good thing. But I think we might either be deprecating too much,
>or not actually using the alternatives ourself.
>
>In the apps, a lot of the files define
>OPENSSL_SUPPRESS_DEPRECATED, which I think is the wrong way to do
>it. We should stop using the deprecated functions ourself. If
>there is no way to do this using non-deprecated functions, the
>function should probably not have been deprecated in the first
>place.
>
>The apps might have functionality that we want to deprecate too,
>that depends on the deprecated functions. In which case we should
>also mark that as deprecated, and the apps should always build in
>no-deprecation mode.
>
>We might also be deprecating APIs that we don't use ourself, so
>it's harder to know if there are alternatives for it.
>
>It would also be nice that if you deprecate a function, that you
>point to the alternative way to do the same thing in the manual.
>
>
>Kurt

-- 
Richard by mobile


Re: Deprecation

2020-02-14 Thread Richard Levitte
On Fri, 14 Feb 2020 12:17:30 +0100,
Dmitry Belyavsky wrote:
> 
> 
> Dear Richard,
> 
> On Fri, Feb 14, 2020 at 1:37 PM Richard Levitte  wrote:
> 
> It should be pointed out that the engine stuff isn't as stable as you
> might think, because it shares internal structures with libcrypto.
> Even if we handle those structures via function calls, all it takes is
> loading an engine linked against - say - 1.1.0 in 1.1.1 to run a real
> risk of a spectacular KABOOM if any of the structure that are touched
> changed between those OpenSSL versions.
> 
> Does ABI compatibility cover a significant share of these cases?

Yes.  Any structure change between those OpenSSL version potentially
means an ABI incompatibility between an engine and the libcrypto that
loads it.

This is precisely the reason why we have forbidden shared opaque
structures over the libcrypto <-> provider boundary.

...

> > But some features I'm interested in imply engine model (and it will
> > be great if somebody else could look at PR 10904 to avoid it when
> > possible).
>
> Apart from a few details, that PR looks rather innocuous.  I frankly
> haven't had time to look at it too deeply (I just discovered that I
> was prompted), as I haven't dived in the CMS code yet...
> 
> Well, the problem is introducing some new control values. I don't
> feel policy here very well. I also plan to submit at least one
> TLS-related PR significantly more innocent...

We haven't formed a clear policy on those, it's a bit of play as we go
and do what makes most sense.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Deprecation

2020-02-14 Thread Richard Levitte
On Fri, 14 Feb 2020 10:41:05 +0100,
Dmitry Belyavsky wrote:
> 
> 
> Hello,
> 
> On Fri, Feb 14, 2020 at 5:30 AM Dr Paul Dale  wrote:
> 
> There is some pushback against the deprecations going on in various PRs.
>
> The plan has always been to deprecate engines in 3.0 and removing support 
> for them 5+ years later. 
> Originally, the path was to have included an engine provider that could 
> load engines and make them appear
> to be a provider.  After a fair amount of investigation, this was deemed 
> to be too difficult in the 3.0
> time frame.
>
> Do we still want to deprecate engines in 3.0?
> Should we defer until 4.0 instead?
> 
> I think we should delay the deprecation of engine stuff to 4.0. Personally I 
> don't have sense of stability of
> provider API.

It should be pointed out that the engine stuff isn't as stable as you
might think, because it shares internal structures with libcrypto.
Even if we handle those structures via function calls, all it takes is
loading an engine linked against - say - 1.1.0 in 1.1.1 to run a real
risk of a spectacular KABOOM if any of the structure that are touched
changed between those OpenSSL versions.
This is a consequence of making structures opaque and feeling much
more liberty in changing them, and we didn't quite pay attention.

The whole model around providers is intentionally designed to allow
providers to run flexibly with any OpenSSL version, even if they are
linked with some (possibly different) libcrypto version.

So the question of stability depends on what you look at.

It's true that some parts of the provider API is fluctuating a bit, as
early designs need modifications to better meet demands.  We're still
in development.  Come beta1 (schedule for June), I expect that it will
have stabilized.  Come the release, it must have.

> The main benefits seem to boil down to continuing to support existing 
> engines vs removing the legacy code
> paths and switching to the provider model.
> 
> For me, both as open-source and commercial engine developer seems
> reasonable to delay conversion from engines to providers at least
> until 3.0.0 feature freeze happens.

According to our policy, a deprecation starts at the release that has
those deprecations, and removal of deprecated stuff can only happen 5
years later at the earliest.  Seeing deprecations in the source now
doesn't change that, the clock will start ticking when we release
3.0, so consider what you see happening on github as a pre-deprecation
warning.

> But some features I'm interested in imply engine model (and it will
> be great if somebody else could look at PR 10904 to avoid it when
> possible).

Apart from a few details, that PR looks rather innocuous.  I frankly
haven't had time to look at it too deeply (I just discovered that I
was prompted), as I haven't dived in the CMS code yet...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: fips mode and key management

2020-01-21 Thread Richard Levitte
This doesn't affect applications.  Our FIPS module holds its own
keys, but reuse the same structures as libcrypto to hold them
internally, and *there*, the EX_DATA field is irrelevant.
Applications will never get that far in.  The EX_DATA added by the
application is still valid.

I think that the misunderstanding lies in when FIPS_MODE is defined.
It's defined when the FIPS provider module is being built, never
otherwise.

Cheers,
Richard

On Sat, 18 Jan 2020 12:19:25 +0100,
Roumen Petrov wrote:
> 
> Hello,
> 
> Recently I note that when build is in FIPS_MODE some functionality is
> lost. For instance RSA_{g|s}et_ex_data is not available.
> 
> Reading the code I expect that in FIPS mode use of external keys is
> forbidden.
> Remark: ex_data is used to store reference information for external keys.
> 
> Please confirm that in FIPS mode we could use external keys?
> 
> 
> Regards
> Roumen Petrov
> 
> P.S. If is not allowed this regression to previous FIPS
> releases(validations).
> Neither OpenSSL nor Red Hat nor Solaris FIPS validation forbid use of
> "external" keys.
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: crypt(3)

2020-01-17 Thread Richard Levitte
Right. Such a KDF could be implemented elsewhere, as a separate project.

Cheers
Richard


Kurt Roeckx  skrev: (17 januari 2020 21:35:00 CET)
>On Fri, Jan 17, 2020 at 04:31:06PM +1000, Dr Paul Dale wrote:
>> I’ve got several choices:
>> Leave them public and unchanged — that is, don’t deprecate these two
>functions yet.
>> Deprecate them and add KDFs to replace them.
>> Deprecate them, leave them alone and hope they go away painlessly at
>some point.
>
>I really see no point in adding something that we at the same time
>would like to remove. Just deprecate it.
>
>
>Kurt

-- 
Richard by mobile


Re: crypt(3)

2020-01-17 Thread Richard Levitte
I don't really see a reason to actually *remove* them.  Deprecation
should suffice.

Cheers,
Richard

On Fri, 17 Jan 2020 09:19:40 +0100,
Dr Paul Dale wrote:
> 
> 
> My experience with embedded systems is that crypt(3) is in the standard 
> library and not accessed
> via OpenSSL.  Apps/password uses DES_crypt() and password crackers used to 
> use OpenSSL for
> performance reasons.  Neither seems like a huge deal.  I.e. I can’t think of 
> a good reason to keep
> them.
> 
> Removing these calls will require an OMC vote as a breaking API change.  I’m 
> fine to call one if
> it seems justified.
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> On 17 Jan 2020, at 5:41 pm, Viktor Dukhovni  
> wrote:
>
> On Fri, Jan 17, 2020 at 04:31:06PM +1000, Dr Paul Dale wrote:
> 
> There are two functions (DES_crypt and DES_fcrypt) which implement the
> old crypt(3) password algorithm.  Once these are deprecated, they will
> no longer be reachable via EVP.  The confounding point is that they
> aren’t quite DES ― close but not identical.  I would be surprised if
> they aren’t still in use for /etc/passwd files on old and/or embedded
> systems.
> 
> Generally speaking, on Unix-like systems that use crypt(3) for
> /etc/passwd I'd expect to find a standaline crypt() implementation in
> libc, that is independent of OpenSSL.  That is, if your system still
> uses crypt() for passwords, you don't need OpenSSL to compute crypt
> hashes.
>
> That said, this is experience from general-purpose computers running
> Unix-like OSes, not embedded systems, where I have no idea whether
> crypt() is popular, and whether it is provided by a port of libcrypto
> to that platform.
> 
> I’ve got several choices: Leave them public and unchanged ― that is,
> don’t deprecate these two functions yet.  Deprecate them and add KDFs
> to replace them.  Deprecate them, leave them alone and hope they go
> away painlessly at some point.
> 
> I would not expect to find many users of OpenSSL's crypt(), except
> internally within OpenSSL itself.
> 
> The apps/password.c applet calls these which is how I stumbled over
> the complication.  I’m fine refactoring this based on the solution
> chosen.  I’d also be okay with factoring out all the password
> derivation functions into KDFs if necessary.
>
> Thoughts?  Other alternatives?
> 
> I don't know enough about embedded systems to speak about what if
> anything we need to do for those with respect to crypt().
>
> --
>Viktor.
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Legacy provider

2020-01-15 Thread Richard Levitte
On Wed, 15 Jan 2020 21:07:54 +0100,
Benjamin Kaduk wrote:
> 
> Hi Pauli,
> 
> On Tue, Jan 14, 2020 at 09:34:40PM +1000, Dr Paul Dale wrote:
> > The OMC vote is closed.
> > 
> > The vote text being:
> > 
> > The legacy provider should be disabled by default in 3.0
> > 
> > With the clarification that "disabled" in this context means "not loaded”.
> > 
> > The vote passed (two for, one against, four abstain)
> 
> It's good to have a decision here, but I'm kind of worried about the four
> abstains -- it's easy for me to leap to a conclusion that the individuals
> in question just didn't want to to spend the time to come to a considered
> position, even though this issue has substantial potential impact for our
> userbase.  I'm trying to not make faulty assumptions, so some greater
> clarity on the circumstances would be helpful, if possible.

This was a vote that I found extremely difficult.  This topic has been
disputed on and off for quite a while, both on github and within the
OMC, and I could never decide between the two sides.  Both have pros
and cons that outweigh each other.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


monthly Status Report (December 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Meetings, workshops, etc

  - Continued work through older issues together with Matt Caswell,
primarly to triage them, but in some cases to also review them.
We will continued with the seriously aged issues in 2020.

* Development

  - util/mkerr.pl: don't stop reading conserved symbols from the state file
(PR openssl/openssl#10549)
  - Use leak sanitizer instead of internal mdebug to check for memory leaks
(PR openssl/openssl#9294)
  - Disable devcryptoeng on newer OpenBSD versions
(PR openssl/openssl#10566)
  - Move providers/common/{ciphers,digests}/* to providers/implementations
(PR openssl/openssl#10564)
  - PROV: Move AES_CCM and AES_GCM specialisation away from common
cipher header
(PR openssl/openssl#10606)
  - Add better support for using deprecated symbols internally
(PR openssl/openssl#10608)
  - EVP: make it possible to init EVP_PKEY_CTX with provided EVP_PKEY
(PR openssl/openssl#10618)
  - BIO: Add BIO_f_prefix(), a text line prefixing filter
(PR openssl/openssl#10531)
  - CRYPTO: split cipher_platform.h into algorithm specific headers
(PR openssl/openssl#10662)
  - util/find-doc-nits: Better checking of missing documentation
(PR openssl/openssl#10621)
  - util/find-doc-nits: when loading "missing" files, check if documented
(PR openssl/openssl#10683)
  - Configurations/windows-makefile.tmpl: HTMLDOCS are files, not directories
(PR openssl/openssl#10555)
  - [1.1.1] Disable devcryptoeng on newer OpenBSD versions
(PR openssl/openssl#10565)
  - [not yet merged] Implement domparam and key generation
(PR openssl/openssl#10289)
  - [not yet merged] apps: Switch to using OSSL_STORE for loading
keys, certs, ...
(PR openssl/openssl#7390)
  - [not yet merged] PROV: add RSA signature implementation
(PR openssl/openssl#10557)
  - [not yet merged] CORE & EVP: Adapt KEYEXCH, SIGNATURE and
ASYM_CIPHER to handle key types better 
(PR openssl/openssl#10647)
  - [unpublished] Support serializers in 'openssl provider' and
'openssl list'
  - [unpublished] Experiments with builbbot-travis

* Administration

  - Set up initial OTC memberships

  - Form gitolite @otc group
  - Create the public OTC repo
  - Block pushes to branches on the main repo that are now EOL (1.1.0
and 1.0.2)
  - Added xymon entry for internal github instance

  - Set up infrastructure for premium customers

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (November 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Meetings, workshops, etc

  - Worked through a significant amount of older issues together with
Matt Caswell, primarly to triage them, but in some cases to also
review them.  We will continued with the seriously aged issues in
2020.

* Development

  - BIO_s_connect: add an error state and use it
(PR openssl/openssl#7630)
  - Configure: Make --strict-warnings meaningful with MSVC cl
(PR openssl/openssl#10287)
  - VMS: Added new method to gather entropy on VMS, based on SYS$GET_ENTROPY.
(PR openssl/openssl#8926)
  - Fix OSSL_PARAM_set_BN() to fill the given buffer correctly.
(PR openssl/openssl#10326)
  - Make EVP_PKEY_CTX initialization more precise
(PR openssl/openssl#10308)
  - OSSL_STORE: constify the criterion parameter a bit more
(PR openssl/openssl#8442)
  - X509_LOOKUP_store: new X509_LOOKUP_METHOD that works by OSSL_STORE URI
(PR openssl/openssl#8442)
  - EVP: Make the KEYEXCH implementation leaner
(PR openssl/openssl#10305)
  - EVP: Make the SIGNATURE implementation leaner
(PR openssl/openssl#10303)
  - Rework ordinals
(PR openssl/openssl#10348)
  - Change the logic and behaviour surrounding '--api' and 'no-deprecated'
(PR openssl/openssl#10364)
  - Add EVP functionality to create domain params and keys by user data
(PR openssl/openssl#10187)
  - Refactor PEM_read_bio_{PrivateKey,Parameters,DHparams}
(PR openssl/openssl#2746)
  - Cleanup include/openssl/opensslv.h.in
(PR openssl/openssl#10218)
  - Configuration: make Solaris builds with gcc recognise GNU ld
(PR openssl/openssl#8548)
  - Final cleanup after move to leaner EVP_PKEY methods
(PR openssl/openssl#10309)
  - Reinstate the KDF error macros
(PR openssl/openssl#10368)
  - Add a .pragma directive for configuration files
(PR openssl/openssl#8882)
  - [master] SSL: Document SSL_add_{file,dir,store}_cert_subjects_to_stack()
(PR openssl/openssl#10402)
  - [1.1.1] SSL: Document SSL_add_{file,dir}_cert_subjects_to_stack()
(PR openssl/openssl#10403)
  - CORE: Add a generic callback function type
(PR openssl/openssl#10412)
  - CORE & PROV: make export of key data leaner through callback
(PR openssl/openssl#10414)
  - PEM: constify PEM_write_ routines
(PR openssl/openssl#10452)
  - Replumbing: pre-populate the EVP namemap with commonly known names
(PR openssl/openssl#8984)
  - UI_UTIL_wrap_read_pem_callback(): when |cb| is NULL, use PEM_def_callback
(PR openssl/openssl#10447)
  - doc/man7/proxy-certificates.pod: New guide for proxy certificates
(PR openssl/openssl#10507)
  - configdata.pm.in, util/dofile.pl: load 'platform' unconditionally
(PR openssl/openssl#10514)
  - Generate docs at build time instead of install time
(PR openssl/openssl#6236)
  - SERIALIZER: New API for serialization of objects through providers
(PR openssl/openssl#10394)
  - [1.1.1 only] i2b_PVK(): Use Encrypt, not Decrypt
(PR openssl/openssl#10521)
  - [not yet merged] Implement domparam and key generation
(PR openssl/openssl#10289)
  - [not yet merged] apps: Switch to using OSSL_STORE for loading
keys, certs, ...
(PR openssl/openssl#7390)
  - [unpublished] Support serializers in 'openssl provider' and
'openssl list'

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/



Late Monthly Status Report (September 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - Rework the documentation of our individual MAC implementations
(PR openssl/openssl#9713)
  - Refactor how KEYMGMT methods get associated with other methods
(PR openssl/openssl#9678)
  - test/errtest.c: more conditions for checking __FILE__ and __LINE__
(PR openssl/openssl#9755)
  - New functions EVP_MD_free() and EVP_CIPHER_free()
(PR openssl/openssl#9758)
  - Move libapps.a source to apps/lib
(PR openssl/openssl#9723)
  - Move KDFs and PRFs into providers
(PR openssl/openssl#9662)
  - Rework the perl fallback functionality
(PR openssl/openssl#9826)
  - test/evp_test.c: try fetching algorithms
(PR openssl/openssl#9121)
  - doc/man3/OSSL_PARAM.pod: add details about multiple elements with
same key
(PR openssl/openssl#9741)
  - util/perl/OpenSSL/Test.pm: Disable stdout/stderr redirection on
non-verbosity
(PR openssl/openssl#9862)
  - Rework test/run_tests.pl to support selective verbosity and TAP copy
(PR openssl/openssl#9862)
  - ERR fixups and additions
(PR openssl/openssl#9765)
  - Refactor configdata.pm to be generated by template
(PR openssl/openssl#9693)
  - Deprecate the public definition of ERR_STATE
(PR openssl/openssl#9462)
  - Unify assembler scripts
(PR openssl/openssl#9884)
  - crypto/bn/build.info: Correct use of SSE2 definition
(PR openssl/openssl#9879)
  - Refactor TLS1-PRF to create the MAC contexts early
(PR openssl/openssl#9930)
  - Use name identity instead of name in diverse methods
(PR openssl/openssl#9897)
  - Refactor TLS-PRF's kdf_tls1_prf_mkmacctx() to a provider utility
(PR openssl/openssl#9946)
  - Refactor SSKDF to create the MAC contexts early
(PR openssl/openssl#9946)
  - include/openssl/macros.h: Rework OPENSSL_FUNC for div C standards
(PR openssl/openssl#9913)
  - include/openssl/macros.h: better OPENSSL_FUNC fallback
(PR openssl/openssl#9976)
  - Rework cipher / digest fetching for legacy nids with multiple name support
(PR openssl/openssl#9969)
  - Configure, build.info: make it possible to use variables in indexes
(PR openssl/openssl#9637)
  - When building of modules is disabled, build the legacy provider
into libcrypto
(PR openssl/openssl#9637)
  - OSSL_PARAM.pod: document the mechanism to figure out buffer sizes
(PR openssl/openssl#10025)
  - Make doc/man7/ and doc/internal/man3/ conform with man-pages(7)
(PR openssl/openssl#10034)
  - Make relevant tests more sensitive to 'no-fips'
(PR openssl/openssl#10047)
  - Make ASYNC manuals conform with man-pages(7)
(PR openssl/openssl#10043)
  - [not yet merged] Adapt EVP_CIPHER_{param_to_asn1,asn1_to_param} for use
with provider.
(PR openssl/openssl#10008)
  - [not yet merged] Replumbing: pre-populate the EVP namemap with
commonly known names
(PR openssl/openssl#8984)
  - [not yet merged] Display multiple names
(PR openssl/openssl#9979)
  - [unpublished] Continued work on flexible installation commands for
Makefiles
  - [not yet merged] X509_LOOKUP_store: new X509_LOOKUP_METHOD that
works by OSSL_STORE URI
(PR openssl/openssl#8442)

* System administration

  - Installed and set up internal github EE instance
  - Fixed letsencrypt issues with our gitlab instance
  - Modernized our apache config template usage

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (September 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - Rework the documentation of our individual MAC implementations
(PR openssl/openssl#9713)
  - Refactor how KEYMGMT methods get associated with other methods
(PR openssl/openssl#9678)
  - test/errtest.c: more conditions for checking __FILE__ and __LINE__
(PR openssl/openssl#9755)
  - New functions EVP_MD_free() and EVP_CIPHER_free()
(PR openssl/openssl#9758)
  - Move libapps.a source to apps/lib
(PR openssl/openssl#9723)
  - Move KDFs and PRFs into providers
(PR openssl/openssl#9662)
  - Rework the perl fallback functionality
(PR openssl/openssl#9826)
  - test/evp_test.c: try fetching algorithms
(PR openssl/openssl#9121)
  - doc/man3/OSSL_PARAM.pod: add details about multiple elements with
same key
(PR openssl/openssl#9741)
  - util/perl/OpenSSL/Test.pm: Disable stdout/stderr redirection on
non-verbosity
(PR openssl/openssl#9862)
  - Rework test/run_tests.pl to support selective verbosity and TAP copy
(PR openssl/openssl#9862)
  - ERR fixups and additions
(PR openssl/openssl#9765)
  - Refactor configdata.pm to be generated by template
(PR openssl/openssl#9693)
  - Deprecate the public definition of ERR_STATE
(PR openssl/openssl#9462)
  - Unify assembler scripts
(PR openssl/openssl#9884)
  - crypto/bn/build.info: Correct use of SSE2 definition
(PR openssl/openssl#9879)
  - Refactor TLS1-PRF to create the MAC contexts early
(PR openssl/openssl#9930)
  - Use name identity instead of name in diverse methods
(PR openssl/openssl#9897)
  - Refactor TLS-PRF's kdf_tls1_prf_mkmacctx() to a provider utility
(PR openssl/openssl#9946)
  - Refactor SSKDF to create the MAC contexts early
(PR openssl/openssl#9946)
  - include/openssl/macros.h: Rework OPENSSL_FUNC for div C standards
(PR openssl/openssl#9913)
  - include/openssl/macros.h: better OPENSSL_FUNC fallback
(PR openssl/openssl#9976)
  - Rework cipher / digest fetching for legacy nids with multiple name support
(PR openssl/openssl#9969)
  - Configure, build.info: make it possible to use variables in indexes
(PR openssl/openssl#9637)
  - When building of modules is disabled, build the legacy provider
into libcrypto
(PR openssl/openssl#9637)
  - OSSL_PARAM.pod: document the mechanism to figure out buffer sizes
(PR openssl/openssl#10025)
  - Make doc/man7/ and doc/internal/man3/ conform with man-pages(7)
(PR openssl/openssl#10034)
  - Make relevant tests more sensitive to 'no-fips'
(PR openssl/openssl#10047)
  - Make ASYNC manuals conform with man-pages(7)
(PR openssl/openssl#10043)
  - [not yet merged] Adapt EVP_CIPHER_{param_to_asn1,asn1_to_param} for use
with provider.
(PR openssl/openssl#10008)
  - [not yet merged] Replumbing: pre-populate the EVP namemap with
commonly known names
(PR openssl/openssl#8984)
  - [not yet merged] Display multiple names
(PR openssl/openssl#9979)
  - [unpublished] Continued work on flexible installation commands for
Makefiles
  - [not yet merged] X509_LOOKUP_store: new X509_LOOKUP_METHOD that
works by OSSL_STORE URI
(PR openssl/openssl#8442)

* System administration

  - Installed and set up internal github EE instance
  - Fixed letsencrypt issues with our gitlab instance
  - Modernized our apache config template usage

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/



Late Monthly Status Report (October 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Meetings, workshops, etc

  - Attended the OMC f2f, and committers f2f meetings in Nuremburg

* Development

  - Make ASN1 manuals conform with man-pages(7)
(PR openssl/openssl#10042)
  - Make manuals with TYPE conform with man-pages(7)
(PR openssl/openssl#10041)
  - Adapt EVP_CIPHER_{param_to_asn1,asn1_to_param} for use with provider.
(PR openssl/openssl#10008)
  - util/find-doc-nits: more precise option and function name checker
(PR openssl/openssl#10073)
  - Replumbing: make it possible for providers to specify multiple names
(PR openssl/openssl#8985)
  - Move all SHA digests completely to the default provider
(PR openssl/openssl#10059)
  - Move MD5-SHA1 digest completely to the default provider
(PR openssl/openssl#9076)
  - EVP_{CIPHER,MD}_CTX_ctrl(): make sure to return 0 or 1
(PR openssl/openssl#10108)
  - Add documentation for PEM_{read,write}_bio_Parameters()
(PR openssl/openssl#10113)
  - More doc/man1 fixes
(PR openssl/openssl#10065)
  - Document build.info syntax internally
(PR openssl/openssl#10121, openssl/openssl#10148)
  - Reorganize providers, which also involved:
Configure: rework build.info grammar and attributes
Configure: Implement attributes for DEPEND[xxx]
Configurations/common.tmpl: Rework dependency resolution
Build files: Make it possible to source libraries into other libraries
  - POD: stop abusing comment
(PR openssl/openssl#10048)
  - Fix EVP_Cipher() for provided cipher implementations
(PR openssl/openssl#10137)
  - KDF: clean away old EVP_KDF declarations
(PR openssl/openssl#10170)
  - Restore MD5-SHA1 in legacy method database
(PR openssl/openssl#10176)
  - Building: Add modules with DEPENDs to GENERATEd files
(PR openssl/openssl#10162)
  - Move MD2, MD4 and MD5 digests completely to the providers
(PR openssl/openssl#10164)
  - Add EVP_PKEY_CTX_new_provided()
(PR openssl/openssl#10184)
  - EVP_{CIPHER,MD}_CTX_ctrl(): make extra sure to return 0 or 1
(PR openssl/openssl#10163)
  - Display multiple names
(PR openssl/openssl#9979)
  - Configure: break long lines in build files
(PR openssl/openssl#8990)
  - KEYMGMT: export/import domain parameters
(PR openssl/openssl#10169)
  - Export / import RSA to / from provider
(PR openssl/openssl#10190)
  - Move BLAKE2 digests completely to the default provider
(PR openssl/openssl#9075)
  - Centralize version data
(PR openssl/openssl#10205)
  - Doc for the added internal RSA functions
(PR openssl/openssl#10206)
  - windows-makefile.tmpl: Convert all /I and /D to -I and -D
(PR openssl/openssl#10222)
  - crypto/s390xcap.c: Add guards around the GETAUXVAL checks
(PR openssl/openssl#9892)
  - crypto/evp/evp_fetch.c: Make it more prominent that these functions are EVP
(PR openssl/openssl#10257)
  - evp_pkey_ctx_free_old_ops(): Make sure to assign NULL to freed pointers
(PR openssl/openssl#10292)
  - [not yet merged] Replumbing: pre-populate the EVP namemap with
commonly known names
(PR openssl/openssl#8984)
  - [1.1.1 only] Define AESNI_ASM if AESNI assembler is included, and
use it
(PR openssl/openssl#10080)
  - [not yet merged] Implement domparam and key generation
(PR openssl/openssl#10289)
  - [not yet merged] Template fipsprov
(PR openssl/openssl#10124)
  - [not yet merged] Add EVP functionality to create domain params and
keys by user data
(PR openssl/openssl#10187)

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/



Late Monthly Status Report (August 2019)

2019-12-28 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, small fixes, etc., key activities
this month:

* Development

  - Configurations/unix-Makefile.tmpl: Don't clean away dotted files
(PR openssl/openssl#9573)
  - Add text to OSSL_PARAM constructors
(PR openssl/openssl#9303)
  - Rework EVP_MD support, enhance existing functions
(PR openssl/openssl#9391)
  - Enhance param handling with param type descriptors
(PR openssl/openssl#9576)
  - Rename provider and core get_param_types functions
(PR openssl/openssl#9591)
  - Move all MACs to the providers
(PR openssl/openssl#8877)
  - Rename ctx_{get,set}_params to {get,set}_ctx_params
(PR openssl/openssl#9612)
  - Windows UWP builds: determine automatically if asm should be disabled
(PR openssl/openssl#9440)
  - Untangle / retangle opensslv.h, openssslconf.h and macros.h
(PR openssl/openssl#9626)
  - Use macros internally for algorithm names
(PR openssl/openssl#9635)
  - Modify ossl_method_store_add() to accept an OSSL_PROVIDER and check for it
(PR openssl/openssl#9650)
  - Configure: Allow 'DEFINE[]=def'
crypto/bn/build.info: define OPENSL_IA32_SSE2 globally when needed
(PR openssl/openssl#9679)
  - Clean up MAC implementations
(PR openssl/openssl#9667)
  - testing: set OPENSSL_MODULES to the providers directory by default
(PR openssl/openssl#9618)
  - OPENSSL_info(): add the item OPENSSL_INFO_SEED_SOURCE and use it
(PR openssl/openssl#9689)
  - openssl provider: New sub-command, for provider discovery
(PR openssl/openssl#9697)
  - [not yet merged] When building of modules is disabled, build the legacy
provider into libcrypto
(PR openssl/openssl#9637)

* System administration

  - Installed and set up internal github EE instance
  - Fixed letsencrypt issues with our gitlab instance
  - Modernized our apache config template usage

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Flaw in our process for dealing with trivial changes

2019-12-13 Thread Richard Levitte
https://github.com/openssl/tools/pull/49

Quite an exercise, I think my python-fu has increased significantly.

I have *no* idea how to debug CGI stuff in a sensible way, suggestions
welcome!

Cheers,
Richard

On Fri, 13 Dec 2019 12:08:42 +0100,
Richard Levitte wrote:
> 
> clacheck modification coming up!
> 
> Cheers,
> Richard
> 
> On Fri, 13 Dec 2019 04:48:38 +0100,
> Dr Paul Dale wrote:
> > 
> > 
> > A better example of this problem: #10607.  Both Paul and I approved it 
> > yesterday and I merged it
> > today without noticing until too late that it was tagged “CLA: trivial” :(
> > I’ve not reverted it at this point but will if necessary.
> > 
> > Let’s get the label in.
> > 
> > Pauli
> > -- 
> > Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> > Phone +61 7 3031 7217
> > Oracle Australia
> > 
> > On 13 Dec 2019, at 11:02 am, Richard Levitte  
> > wrote:
> >
> > On Thu, 12 Dec 2019 22:31:10 +0100,
> > Dr Paul Dale wrote:
> > 
> > A red blocker along the lines of: “Triviality Unconfirmed”. One of
> > the reviewers needs to remove this before the PR can be merged.
> >
> > It’s in our face, it prevent accidental merges and its low overhead.
> > 
> > I still think simply adding the label should be sufficient.  I dunno
> > about you, but I look at labels all the time, for all sorts of
> > reasons, and one saying [cla: trivial] would certainly attract my
> > attention.
> >
> > Let's make it bright red-orange, that'll catch anyone's eye (even mine)
> >
> > Also, removing that label will rapidly be annoying as soon as someone
> > closes and re-opens a PR...  or whatever other action that triggers
> > the "pull_request" event (and there's a lot that does that...  our
> > script is being kept busy!).
> >
> > Cheers,
> >     Richard
> >
> > --
> > Richard Levitte levi...@openssl.org
> > OpenSSL Project http://www.openssl.org/~levitte/
> > 
> > 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Flaw in our process for dealing with trivial changes

2019-12-13 Thread Richard Levitte
clacheck modification coming up!

Cheers,
Richard

On Fri, 13 Dec 2019 04:48:38 +0100,
Dr Paul Dale wrote:
> 
> 
> A better example of this problem: #10607.  Both Paul and I approved it 
> yesterday and I merged it
> today without noticing until too late that it was tagged “CLA: trivial” :(
> I’ve not reverted it at this point but will if necessary.
> 
> Let’s get the label in.
> 
> Pauli
> -- 
> Dr Paul Dale | Distinguished Architect | Cryptographic Foundations 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> On 13 Dec 2019, at 11:02 am, Richard Levitte  wrote:
>
> On Thu, 12 Dec 2019 22:31:10 +0100,
> Dr Paul Dale wrote:
> 
> A red blocker along the lines of: “Triviality Unconfirmed”. One of
> the reviewers needs to remove this before the PR can be merged.
>
> It’s in our face, it prevent accidental merges and its low overhead.
> 
> I still think simply adding the label should be sufficient.  I dunno
> about you, but I look at labels all the time, for all sorts of
> reasons, and one saying [cla: trivial] would certainly attract my
> attention.
>
> Let's make it bright red-orange, that'll catch anyone's eye (even mine)
>
> Also, removing that label will rapidly be annoying as soon as someone
> closes and re-opens a PR...  or whatever other action that triggers
> the "pull_request" event (and there's a lot that does that...  our
> script is being kept busy!).
>
> Cheers,
> Richard
>
> --
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Flaw in our process for dealing with trivial changes

2019-12-13 Thread Richard Levitte
On Fri, 13 Dec 2019 08:58:27 +0100,
Dr. Matthias St. Pierre wrote:
> 
> 
> > On Thu, 12 Dec 2019 22:31:10 +0100,
> > Dr Paul Dale wrote:
> > >
> > > A red blocker along the lines of: "Triviality Unconfirmed". One of
> > > the reviewers needs to remove this before the PR can be merged.
> > >
> > > It's in our face, it prevent accidental merges and its low overhead.
> > 
> > I still think simply adding the label should be sufficient.  I dunno
> > about you, but I look at labels all the time, for all sorts of
> > reasons, and one saying [cla: trivial] would certainly attract my
> > attention.
> > 
> > Let's make it bright red-orange, that'll catch anyone's eye (even mine)
> > 
> > Also, removing that label will rapidly be annoying as soon as someone
> > closes and re-opens a PR...  or whatever other action that triggers
> > the "pull_request" event (and there's a lot that does that...  our
> > script is being kept busy!).
> > 
> > Cheers,
> > Richard
> 
> 
> This seems to be implied already by my last proposal, with just one color 
> changed:   ;-)

Yes and no...  you talked about addrev, which should not be affected
by this.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Flaw in our process for dealing with trivial changes

2019-12-12 Thread Richard Levitte
On Thu, 12 Dec 2019 22:31:10 +0100,
Dr Paul Dale wrote:
> 
> A red blocker along the lines of: “Triviality Unconfirmed”. One of
> the reviewers needs to remove this before the PR can be merged.
> 
> It’s in our face, it prevent accidental merges and its low overhead.

I still think simply adding the label should be sufficient.  I dunno
about you, but I look at labels all the time, for all sorts of
reasons, and one saying [cla: trivial] would certainly attract my
attention.

Let's make it bright red-orange, that'll catch anyone's eye (even mine)

Also, removing that label will rapidly be annoying as soon as someone
closes and re-opens a PR...  or whatever other action that triggers
the "pull_request" event (and there's a lot that does that...  our
script is being kept busy!).

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Flaw in our process for dealing with trivial changes

2019-12-12 Thread Richard Levitte
On Thu, 12 Dec 2019 12:15:30 +0100,
Dr. Matthias St. Pierre wrote:
> 
> As for a possible semi-automated solution:
> 
> The problem is more fundamental: currently both the GitHub bot and
> the git commit hook only watch out for the 'CLA: trivial' marker.

Correct re the clacheck hook (that's the Github hook we're talking
about), but re the update hook on our git server, it does a little
more than that, as already shown.

> One possible solution to this problem could be the following procedure: 
> 
> Add three mutually exclusive [cla: *] labels:
> 
>   [cla: ok] (green)
>   [cla: trivial]   (green)
>   [cla: missing](red)
> 
> The CLA bot *always* sets the [cla: ok] label if it finds a  CLA on
> file. Otherwise, it sets the [cla: missing] label, unless the [cla:
> trivial] label is already set.

I'm not sure why the [cla: ok] or [cla: missing] labels are needed.
We already have a [hold: cla required] label that comes up when
there's a lack of CLA and of "CLA: trivial" marker, so [cla: ok]
and [cla: missing] seem redundant.

However, contrary to you, I would have the [cla: trivial] label added
automatically!...  whenever clacheck finds a "CLA: trivial" line in
any of the commits.

> The [cla: trivial] label can only be set manually by a committer,
> and only after the consent between contributor and both reviewers
> has been reached.

Sounds superfluous, considering there's already a need for two
approvals, as well as the [approved: done] label set.  Yet another
manual label will make zero difference, as long as all reviewers can
clearly see that there's a "CLA: trivial" commit (i.e. that the [cla:
trivial] label has been set by clacheck).

After all, the problem we have hit is that "CLA: trivial" can go
undetected, so let's make sure it doesn't, without adding a lot of
other redundant mechanisms that only make our lives harder, yeah?

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Flaw in our process for dealing with trivial changes

2019-12-12 Thread Richard Levitte
On Thu, 12 Dec 2019 13:06:33 +0100,
Dr. Matthias St. Pierre wrote:
> 
> > > The server-side commit hook ensures that
> > >
> > > - the "CLA: trivial" annotation is present in *all* commits of the PR if 
> > > and only if the [cla: trivial]
> > >   label is set.
> > > - the [cla: ok] label is set if and only if the CLA is on file
> > > - the pull request is accepted only if the [cla: ok] or [cla: trivial] 
> > > label is set
> > 
> > 
> > One issue with this bit is that it requires the git hooks to connect to
> > github to check the labels. AFAIK we don't do that at present. Does it
> > add too much complexity to the hooks?
> 
> Actually, the consistency checks could be done entirely by the
> addrev script, which already uses the GitHub API.

Incidently, I'm looking at a rewrite of addrev to make it a
self-contained script that doesn't need the assistance of gitaddrev,
and that also computes everything in the preparation sweep, so the
filter part just needs to do the editing (i.e. should work *much*
faster).

> git commit hook
> =
> 
> Accept pull request if and only if the CLA is on file or *all*
> commits have the "CLA: trivial" annotation.
> 
> (The git commit hook would need only a minimal change, if it does
> not check *all* commits yet.)

git education: It's the post-receive or update hooks, not the commit
hook.  There are commit hooks, and they are all client side, so
talking about a commit hook in this context is *really* confusing,
especially for those who aren't qutie aware.
Ref: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

That being said, the update hook does what you say, on a per commit
basis.  In other words, for each commit, it will do the appropriate
checks.  This is the leading comment of that script:

  # For any revision between oldrev and newrev, print VREF/UNREVIEWED/{sha}
  # if it hasn't been reviewed enough.
  #
  # The rules for being reviewed enough are:
  #
  # - UNLESS there's a 'CLA: Trivial' line:
  #   - at least one of the author and the reviewers MUST have a CLA and MUST
  # be member of the 'omc' group.
  #   - at least two of the author and the reviewers MUST have a CLA and MUST
  # be member of the 'commit' group.
  # - IF there's a 'CLA: Trivial' line:
  #   - at least one of the reviewers MUST have a CLA and MUST be member of the
  # 'omc' group.
  #   - at least two of the reviewers MUST have a CLA and MUST be member of the
  # 'commit' group.

(gitolite works by having a denial on any output VREF/UNREVIEWED/ line)

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Check NULL pointers or not...

2019-11-28 Thread Richard Levitte
On Wed, 27 Nov 2019 11:53:01 +0100,
Dr. Matthias St. Pierre wrote:
> 
> FYI: there was a related discussion a year ago on GitHub about an 
> ossl_is_null() macro,
> 
> https://github.com/openssl/openssl/pull/7218
> 
> which ended up undecided.

Thank you.  I had totally forgotten that one, so reminder appreciated.

Looking at the diff shows me that there's a precedent for what I
proposed (using ossl_assert).

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Check NULL pointers or not...

2019-11-27 Thread Richard Levitte
Depending on who you're asking, you get different responses.

The topic is: should we check pointers coming from applications before
using them or not?

There are two positions on this:

1. Yes we should so we don't crash processes unnecessarily, and we
   should return an error if pointers are NULL (unless that's valid
   input, such as for the freeing functions).  There is a generic
   error reason code for it, ERR_R_PASSED_NULL_PARAMETER.

2. No we should not, as we don't cater for user programming errors.
   Also, it allows us to catch our own errors early.

For a while, our pendulum was going with option 2, but lately, we seem
to go back to option 1.

Both options have their pros and cons:

1. cons: there's the very real possibility that users don't check for
 errors as they should, and things go wrong further on,
 sometimes in ways where the actual cause is very hard to
 figure out.
   pros: programs will not spuriously crash because we didn't catch an
 internal corner case in our tests.

2. cons: programs may crash, sometimes due to our own mistakes,
 sometimes due to user errors.
   pros: some very subtle bugs are found, either quicker or at all.
 An actual case is PR 7630 [1], which was uncovered because
 there was a crash due to a NULL being used (this was
 ultimately due to users not checking errors).

As a middle ground, and perhaps a way to satify both camps, we could
use ossl_assert() in our checks for input NULL, as follows:

if (!ossl_assert(ptr != NULL)) {
ERR_raise(ERR_LIB_WHATEVER, ERR_R_PASSED_NULL_PARAMETER);
return 0;
}

This means that in a debug build, there may be crashes due to pointers
being NULL, while in a release build, the caller gets an error.

Thoughts?

Cheers,
Richard

[1] https://github.com/openssl/openssl/pull/7630

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Re-requesting reviews on GitHub

2019-10-30 Thread Richard Levitte
This is a good idea, and also a detectable event for a bot to listen to.

Cheers
Richard 

"Dr. Matthias St. Pierre"  skrev: (30 oktober 
2019 09:30:11 CET)
>> Independently of the new 'approval: *' state labelling I was
>wondering whether it wouldn't
>> be a good idea to adopt the habit of explicitly requesting a
>re-review from the other reviewers
>> after significant changes, using the mechanism provided by GitHub
>(i.e. the button with the two
>
>To be more precise: not all reviews need to be invalidated by
>requesting a re-review, only the approvals.

-- 
Skickat från min Android-enhet med K-9 Mail. Ursäkta min fåordighet.


Re: Build failures on master

2019-09-29 Thread Richard Levitte
This keeps happening because the standard tests don't include no-fips
or no-pic or no-module (the latter two will disable fips too), so we
keep not hitting this hurdle in PRs unless we have extended tests run
(done by having '[extended tests]' in the commit message).  We should
add a configuration with 'no-fips' among our standard tests, so this
becomes automatically visible in PRs.

The issue with the tests?  Doing FIPS tests without checking if 'fips'
is disabled or not.

Cheers,
Richard

On Sat, 28 Sep 2019 22:25:04 +0200,
Dr. Matthias St. Pierre wrote:
> 
> Hi,
> 
> since my last commit on master did not build successfully, I checked the CIs.
> It turned out that a lot of tests are failing, and they have been for quite a 
> while now.
> 
> https://travis-ci.org/openssl/openssl/builds/590874649?utm_source=github_status_medium=notification
> 
> Apart from the well known trio boringssl, krb5, and pyca, there are some new 
> failing tests.
> It would be nice if we could put some effort into fixing them in order to get 
> back to having
> stable successful builds on master again. The build failures on master also 
> affect the
> pull request CI builds and if the failure persists, people stop checking 
> their red crosses.
> 
> Matthias
> 
> 
> --
> 
> https://travis-ci.org/openssl/openssl/jobs/590874663
> 
> 80-test_ssl_old.t(Wstat: 256 Tests: 6 Failed: 1)
>   Failed test:  3
>   Non-zero exit status: 1
> 
> 
> https://travis-ci.org/openssl/openssl/jobs/590874664
> 
> Test Summary Report
> ---
> 95-test_external_boringssl.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 95-test_external_krb5.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 95-test_external_pyca.t (Wstat: 256 Tests: 1 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 
> https://travis-ci.org/openssl/openssl/jobs/590874669
> 
> Test Summary Report
> ---
> 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
>   Failed tests:  1, 9-18, 25-34
>   Non-zero exit status: 21
> 
> https://travis-ci.org/openssl/openssl/jobs/590874670
> 
> Test Summary Report
> ---
> 30-test_evp.t(Wstat: 256 Tests: 19 Failed: 1)
>   Failed test:  1
>   Non-zero exit status: 1
> 30-test_evp_fetch_prov.t (Wstat: 5376 Tests: 34 Failed: 21)
>   Failed tests:  1, 9-18, 25-34
>   Non-zero exit status: 21
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Being socially aware

2019-09-17 Thread Richard Levitte
You forget that this is done in public, i.e. others are looking at how
we treat those who do come forward and submit code to some bite size
issue.  So while @agnosticdev may have enough skin on his nose,
onlookers that are considering whether they should contribute may not.

On Tue, 17 Sep 2019 09:04:21 +0200,
Dr. Matthias St. Pierre wrote:
> 
> I appreciate your concerns Richard, but I believe they are unwarranted in this
> case fortunately.
> 
> First, my impression is that the discussion between was objective all the time
> and far from being heated up with emotions.
> 
> Second, looking at the profile of the contributor, one can assert that he 
> might
> be relatively new to our project, but he is certainly experienced in Open 
> Source
> development. So I wouldn't worry too much about his feelings 
> 
> Regards,
> Matthias
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Being socially aware

2019-09-16 Thread Richard Levitte
...  or how our usual style of dispute can sometimes deter help from
the community.

https://github.com/openssl/openssl/pull/9912

In this PR, there's a lot of discussions going on, a bit back an
forth, about the right way to do things, and what does what like what
other thing and so on.

This is often our modus operandi, and it has often given us pretty
good quality code (or at least, so we hope), even though it can be a
bit exhausting at times.

I would like to emphasise that I think this is absolutely fine...
when it's among us who have been along for a bit, perhaps have come to
know each other a bit more and have some kind of sense of our
respective strengths and weaknesses, or even with someone that has
shown they can handle this type of discussion.

However, in pull requests like the one cited, where the associated
issue is marked "good first issue", and the author has done quite well
what was asked from the issue, it can be quite unexpected, not to say
overwhelming to be met with this discussion style that could be seen
as getting out of proportion.  A "good first issue" is supposed to be
a bite size thing after all, and I fear that if this is how we welcome
new contributors, they might more often than not choose to step away
from it all.

So maybe let's be a little more careful with contributions for "good
first issue" and potential newcomers, yeah?

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Deprecation of stuff

2019-09-02 Thread Richard Levitte
The dispute in PR https://github.com/openssl/openssl/pull/7853 has
made it quote obvious that we have some very different ideas on when
and why we should or shouldn't deprecate stuff.

What does deprecation mean?  Essentially, it's a warning that at some
point in the future, the deprecated functionality will be removed.  I
believe that part of the issue surrounding this is uncertainty about
when that removal will happen, so let me just remind you what's
written in our release strategy document:

  * No existing public interface can be removed until its replacement
has been in place in an LTS stable release. The original interface
must also have been documented as deprecated for at least 5 years.
A public interface is any function, structure or macro declared in
a public header file.

Ref: https://www.openssl.org/policies/releasestrat.html

I'd like to get this thread started with the following four questions,
for as many of us to answer as possible:

1. Why should we deprecate stuff

2. Why should we not deprecate stuff

3. When should we deprecate stuff

4. When should we not deprecate stuff

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: GitHub issue template for general questions

2019-09-01 Thread Richard Levitte
I don't remember why the generic template was made so small...  So I
think I would welcome a PR that does what you describe.

Cheers,
Richard

On Sun, 01 Sep 2019 14:10:42 +0200,
Dr. Matthias St. Pierre wrote:
> 
> Hi,
> 
> I noticed that a lot of issues on GitHub containing general questions get 
> tagged as a [bug],
> because the template list only gives the choice between two options, a *Bug 
> Report* and
> a *Feature request*. (There is a third choice in tiny letters, which usually 
> gets overlooked.)
> A recent example is https://github.com/openssl/openssl/issues/9743.
> 
> For that reason, I would like to suggest adding a third template, which 
> creates issues tagged
> as  [question]:
> 
>   *Bug report*
> Report a defect in the software
>   *Feature request*
> Propose a feature you would like to see added in the software
>   *Question*
> Ask a general question about the software
> 
> I am aware that this third category was probably omitted on purpose, because 
> we would rather
> see those questions asked on the openssl-users mailing list, but the practice 
> shows that user
> interaction on the mailing lists is fading and more and more user questions 
> are being asked on
> GitHub. What do you think about the suggestion?
> 
> Matthias
> 
> 
> 
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Monthly Status Report (July 2019)

2019-08-14 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, etc., key activities this month:

* Development

  - Re-implemente error reporting for providers and adapted the FIPS
module.
(PR openssl/openssl#9174)
  - Adapted provider cipher implementations to give back diverse
parameters in form of OSSL_PARAM instead of specialized functions.
(PR openssl/openssl#9328)
  - Corrected some OSSL_PARAM documentation
(PR openssl/openssl#9408)
  - Enable the use of Dl_info and dladdr() on Cygwin
(PR openssl/openssl#9402)
  - Added basic EVP_KEYMGMT API and libcrypto <-> provider interface,
and an export/import mechanism in the EVP sub-system to allow keys
to be passed between providers, insofar that the providers allow it.
(PR openssl/openssl#9312)
  - Added documentation to describe providers and the libcrypto <->
provider interface, provider(7), and provider-base(7) that
describing the base functions
(PR openssl/openssl#9409)
  - Added documentation of the KEYMGMT interface, provider-keymgmt(7)
(PR openssl/openssl#9429)
  - Re-implemented the cipher and digest listings for 'openssl list'
to be able to display implementations by providers alongside the
legacy built in one.  This included reworking the functionality to
walk through all available implemented algorithms, and diverse
added EVP information functionality.
(PR openssl/openssl#9356)
  - Documented OSSL_PARAM as a parameter descriptor, and replaced all
uses of OSSL_ITEM with OSSL_PARAM as parameter descriptor,
everywhere
(PR openssl/openssl#9346)
  - [draft] Started work on adapting OSSL_STORE for providers
(PR openssl/openssl#9389)
  - [not yet merged] Started the same work I did for ciphers (PR
9328), but for hash implementations
(PR openssl/openssl#9391)
  - Adapted DH to use with KEYMGMT
(PR openssl/openssl#9394)
  - Added functions to see if a provider is available for use, and
modify test/evp_test.c to check if the legacy provider is
available for the algorithms that are implemented there.
(PR openssl/openssl#9398)
  - [1.1.1 and 1.1.0] CVE-2019-1552 Fixed mingw installation paths
(PRs openssl/openssl#9400 and openssl/openssl#9460)
  - [1.0.2 only] CVE-2019-1552 Document issues with default
installation path
(PR openssl/openssl#9456)
  - Implemented ERR_raise() and ERR_raise_data() for more flexible
error reporting, and refactored all the XXXerr() macros to use
them.  Also refactored the provider error reporting support and
adapted the FIPS provider to use the new functionality.
(PR openssl/openssl#9452)
  - [not yet merged] Continued work to move all MAC implementations to
the providers
(PR openssl/openssl#8877)

* Web

  - CVE-2019-1552 Added security advisory
(PR openssl/web#134)

* System administration

  - Added CAA records for our main domains
  - Moved our VMs to larger space by creating a LLVM volume for them
on an unused partition, moving them there, then added the old
partition to that volume.

* Internal

  - Better logging of gitolite triggers

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (June 2019)

2019-08-14 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, etc., key activities this month:

* Development

  - [not yet merged] Moved BLAKE2 and MD5_SHA1 digests completely to
default provider
(PRs openssl/openssl#9075 and openssl/openssl#9076)
  - [not yet merged] Continued work to move all MAC implementations to
the providers
(PR openssl/openssl#8877)
  - For EVP fetching, made operation_id part of the method identity
(PR openssl/openssl#9109)
  - Added support for variables in build.info files and used them
where it's worth the while
(PR openssl/openssl#9144)
  - Added a core upcall to get the provider object's library context,
adapted all our providers to use it
(PR openssl/openssl#9160)
  - Allowed build.info conditions and variable values to have variable
references, and using that, moved all the asm and aux source specs
to the build.info files
(PR openssl/openssl#9166)
  - Added tracing capability in test utilities
(PR openssl/openssl#9191)
  - Enhanced and update the docs of the internal ossl_provider API
(PR openssl/openssl#9200)
  - [merged] Added support for multiple names per algorithm
(PR openssl/openssl#8967)
  - [1.1.1 only] Backported the modification to only output DER with
SPKAC input and when -out is chosen
(PR openssl/openssl#8368)

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (May 2019)

2019-08-14 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, etc., key activities this month:

* Development

  - [not yet merged] Created a FIPS module checksum script and
modified the Makefile rules to use it
(PR openssl/openssl#8871)
  - [not yet merged] Started work to move all MAC implementations to
the providers
(PR openssl/openssl#8877)
  - Created internal dynamic id number<->name mapping API, to replace
the legacy NID<->name database, and made the generic EVP fetching
mechanism use it
(PR openssl/openssl#8878)
  - [not yet merged] Added a .pragma directive for configuration files
and added a pragma that allows '$' to be considered a symbol
character unless followed by a brace.
(PR openssl/openssl#8882)
  - [not yet merged] Modify the VMS entropy gathering unit to use the
upcoming system servive SYS$GET_ENTROPY
(PR openssl/openssl#8926)
  - [not yet merged] Added new X509_LOOKUP method that works with
OSSL_STORE and a new command line options -CAstore that takes an
OSSL_STORE URI
(PR openssl/openssl#8442)
  - [not yet merged] Modified the internal dynamic id number<->name
map to handle multiple names for the same id number
(PR openssl/openssl#8967)
  - [not yet merged] Provided a couple of alternatives for populating
the internal dynamic id number<->name map with aliases
(PR openssl/openssl#8984 pre-populates the map with hard-coded
aliases within the EVP sub-system)
(PR openssl/openssl#8985 makes it possible for providers to
provide aliases)
  - Attended the OMC f2f in Vancouver
  - Attended the ICMC conference in Vancouver
  - Clarified the requirements for X509_LOOKUP methods as to what side
effects are expected when looking up diverse objects
(PR openssl/openssl#8755)
  - Join the x509 and x509v3 directories
(PR openssl/openssl#8925)
  - Constify OSSL_PROVIDER getter input parameters
(PR openssl/openssl#9054)
  - Released OpenSSL 1.1.1c, 1.1.0k and 1.0.2s
  - Backport making make C++ build tests optional and configurable to
1.1.1
(PRs openssl/openssl#8370 and openssl/openssl#9016)

* Internal

  - Recorded a number of committers that accepted their invitations
while in Vancouver

* System administration

  - Worked on our Apache configs to remove some kinks

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Late Monthly Status Report (April 2019)

2019-08-14 Thread Richard Levitte
Apart from normal business, such as normal reviews, OMC business,
normal system administration tasks, etc., key activities this month:

* Development

  - Restore the "heartbeats" configuration option in Configure among
the deprecated
(PR openssl/openssl#8632)
  - Made it possible to check module availability and to disable
module building
(PRs openssl/openssl#8623, openssl/openssl#8665)
  - Added a provider configuration module
(PR openssl/openssl#8549)
  - Renamed the PROVIDER_CONF and ENGINE_CONF trace classes to CONF
(PR openssl/openssl#8680)
  - Added EVP_set_default_properties() to set global properties and a
corresponding configuration command
(PR openssl/openssl#8681)
  - Changed the params API to require sizes for {utf8,octet}_ptr
(PR openssl/openssl#8703)
  - Fixed the generic EVP algorithm fetch to actually cache them
(PR openssl/openssl#8781)
  - Reworked the processing of %user and %useradd in Configure so they
are merged into %config earlier, and make disabling stuff easier
and safer
(PR openssl/openssl#8812)
  - Added the possibility to display and use MODULESDIR
(PR openssl/openssl#8709)
  - Added a new function OPENSSL_info() as a way for application to
get built-in OpenSSL data, as well as a corresponding command
'openssl info' for scripting purposes
(PR openssl/openssl#8709)
  - Made Configure recognise clang -fsanitize options and translate
them to our disabling options
(PR openssl/openssl#8778)
  - Configure: process shared-info.pl later
(PR openssl/openssl#8846)
  - Made the oneshot proider cipher function like the others
(PR openssl/openssl#8849)
  - Gave the possibility for the provider to create a context
(PR openssl/openssl#8848)
  - Submitted and kept working on previous work to switch apps to use
OSSL_STORE to find certs, keys and CRLs
(PR openssl/openssl#7390)
  - [unpublished] Continued work on flexible installation commands for
Makefiles
  - [1.1.1 only] Reworked DSO API conditions and configuration option
(PR openssl/openssl#8622)

* System administration

  - Worked on our bind configuration for better templating of repeated
data.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Machine downtime tomorrow morning

2019-07-26 Thread Richard Levitte
Now done.  It all went smoothly.

Cheers,
Richard ( your friendly sysadmin of the day )

On Thu, 25 Jul 2019 22:10:08 +0200,
Richard Levitte wrote:
> 
> I'm in the process of moving our VMs to a new volume with more space,
> which as running short in the current VM partition.
> 
> Most of them are quite unremarkable, running services such as DNS and
> similar that's cached or not very visible to the outside world...
> those have already moved to the new volume today.
> 
> However, two of the VMs are more noticable: our main git server / web
> server / ftp server and our mail server.  I will move them to the new
> volume tomorrow morning anywhere between 08:00 and 10:00, Swedish
> time.  The downtime will be somewhere around 30 minutes for each VM.
> 
> Cheers,
> Richard
> 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project     http://www.openssl.org/~levitte/
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Planning for future downtime

2019-07-25 Thread Richard Levitte
Our main server, which hosts all our VMs, will need some maintenance
some time soon.  I plan to do this in the second half of August.  Flag
day will be announced after my time off.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Machine downtime tomorrow morning

2019-07-25 Thread Richard Levitte
I'm in the process of moving our VMs to a new volume with more space,
which as running short in the current VM partition.

Most of them are quite unremarkable, running services such as DNS and
similar that's cached or not very visible to the outside world...
those have already moved to the new volume today.

However, two of the VMs are more noticable: our main git server / web
server / ftp server and our mail server.  I will move them to the new
volume tomorrow morning anywhere between 08:00 and 10:00, Swedish
time.  The downtime will be somewhere around 30 minutes for each VM.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Time off the next two weeks

2019-07-25 Thread Richard Levitte
I will take partial time off from Monday July 29th to the weekend
after and full time off from Sunday August 4th to Thursday August 9th.

Partial time off means that I might still spend a couple of hours
daily on OpenSSL, as time allows.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Taking some rest

2019-07-18 Thread Richard Levitte
I've more or less been typing around the clock since tuesday and
cranking out one commit or PR after the other at high pace...  with
very little reaction, so I think it's time I take a bit of rest and
let everyone catch up, comment (boo?) etc as you please.

Apart from a phone call with Matt tomorrow, I plan on being fairly
minimally present until Monday (just in case you find my responses
slow).

Cheers,
Richard ( feels like a balloon that lost all its air )

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Do we really want to have the legacy provider as opt-in only?

2019-07-15 Thread Richard Levitte
On Mon, 15 Jul 2019 16:15:01 +0200,
Tomas Mraz wrote:
> 
> So saying this is "just recompliation and configuration change" is
> at least somewhat oversimplified.
>
> But I am OK with that. I'm just saying it should be better advertised
> and that internally openssl should not use the "load legacy provider by
> having it in default config file" to actively encourage the "load
> legacy provider only if you *really* need it" behavior.

I'm a little confused.  "configuration changes" is about "having it in
the config file", so I don't quite understand "oversimplified".

Regardless of where this discussion gets us, it has always been the
aim that this will be configurable with the config file.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Vote FYI: Remove function codes from error reporting functions as per PR#9058

2019-07-10 Thread Richard Levitte
The vote closed today, with the following result:

+1: 4
 0: 3 (where one is a -0)
-1: 0

The vote passes.

Cheers,
Richard

On Thu, 04 Jul 2019 18:01:59 +0200,
Richard Levitte wrote:
> 
> topic: Remove function codes from error reporting functions as per PR#9058.
> comment: Discussed in
>  
> https://mta.openssl.org/pipermail/openssl-project/2019-June/001426.html
> Proposed by Richard Levitte
> opened: 2019-07-04
> closed: -mm-dd
> 
> I will follow up with the tally when the vote is concluded.
> 
> Cheers,
> Richard
> 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project     http://www.openssl.org/~levitte/
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Cleaning up include file inconsistencies

2019-07-06 Thread Richard Levitte
On Sat, 06 Jul 2019 19:03:27 +0200,
Dr. Matthias St. Pierre wrote:
> 
> > For things that are private to that sub-system (sorry, "package" doesn't 
> > sound right)
> 
> Neither does it to me, apologies. I was looking for the right word, but 
> nothing except
> "package" came to my mind. And I was too lazy to search for it in the docs.

It's not in the docs.  The original term is "library", and that's
because SSLeay and early OpenSSL were designed to make it possible to
build a separate library from any of those subdirectories.
Traces of this is still seen in the error codes.

"sub-system" is a term that some of us have started using, mostly on
github.  I find it catchy enough that I've started using it
regularly.

> > Me, I'm wondering if it wouldn't be clearer if we renamed
> > crypto/include/internal -> crypto/include/crypto, and thereby did
> > this:
> > 
> > #include "crypto/evp.h"
> > 
> > That, to me, is much clearer than the "_int" suffix.
> 
> This sounds like an excellent idea to me.

"Someone" should create a PR...

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Cleaning up include file inconsistencies

2019-07-06 Thread Richard Levitte
On Sat, 06 Jul 2019 12:20:11 +0200,
Dr. Matthias St. Pierre wrote:
> 
> > Having such a finegrained distinction is not the problem, but (at least to 
> > me)
> > it is not entirely clear which include file goes into which directory.
> 
> Note: the high score seems to lie at four different header files for the same 
> package,
> not counting the generated error header file:
> 
>   ~/src/openssl$ find -name 'store*.h'
> 
...
>   ./crypto/include/internal/store.h
>   ./crypto/include/internal/store_int.h
...

I have *no* idea why there are two header files.  I must have
forgotten about one of them when creating the other...

They should be merged into one.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: AW: Cleaning up include file inconsistencies

2019-07-06 Thread Richard Levitte
On Sat, 06 Jul 2019 11:50:48 +0200,
Dr. Matthias St. Pierre wrote:
> 
> There are more oddities in the organization of the internal header files.
> 
> 1) It appears to me that there are three different levels of internal header 
> files
> 
>  - headers in `include/internal`

For internal things that we want to make available for both libcrypto
and libssl (i.e. they are exported in libcrypto.so)

>  - headers in `crypto/include/internal`

For internal things that we only want available inside libcrypto
(i.e. they are NOT exported in libcrypto.so)

>  - headers in `crypto/` along with the source files

For things that are private to that sub-system (sorry, "package"
doesn't sound right)

> Having such a finegrained distinction is not the problem, but (at least to me)
> it is not entirely clear which include file goes into which directory.
> 
> 2) Some of the header files carry an `_int.h` suffix while others not,
> for seemingly no particular reason.
> 
>   ~/src/openssl$ find -name '*_int.h'
>   ./crypto/crmf/crmf_int.h

Should have been named crmf_locl.h

>   ./crypto/include/internal/err_int.h
>   ./crypto/include/internal/ess_int.h
>   ./crypto/include/internal/ec_int.h
>   ./crypto/include/internal/rand_int.h
>   ./crypto/include/internal/store_int.h
>   ./crypto/include/internal/asn1_int.h
>   ./crypto/include/internal/modes_int.h
>   ./crypto/include/internal/cryptlib_int.h
>   ./crypto/include/internal/bn_int.h
>   ./crypto/include/internal/evp_int.h
>   ./crypto/include/internal/x509_int.h

The reason for this is to avoid confusion between headers in
crypto/include/internal and headers in include/internal.  It's not
quite as necessary as all these names seem to suggest, but we do have
these as an example:

: ; find . -name 'err*.h' | grep /include/internal/
./include/internal/err.h
./crypto/include/internal/err_int.h

>   ./crypto/cmp/cmp_int.h

Should have been named cmp_locl.h

>   ./crypto/x509/pcy_int.h

Should have been named pcy_locl.h

> In particular for the headers the `include/internal` folder this suffix
> is superfluous, because these files are (or should be) included via
> relative paths. So instead of
> 
>   #include 
>   #include "internal/evp_int.h"
> 
> It could just be
> 
>   #include 
>   #include "internal/evp.h"

In most cases, you're right.  I suspect "_int" was habitually added to
ensure we know exactly where that header comes from.

Me, I'm wondering if it wouldn't be clearer if we renamed
crypto/include/internal -> crypto/include/crypto, and thereby did
this:

#include "crypto/evp.h"

That, to me, is much clearer than the "_int" suffix.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Vote FYI: Remove function codes from error reporting functions as per PR#9058

2019-07-04 Thread Richard Levitte
topic: Remove function codes from error reporting functions as per PR#9058.
comment: Discussed in
 https://mta.openssl.org/pipermail/openssl-project/2019-June/001426.html
Proposed by Richard Levitte
opened: 2019-07-04
closed: -mm-dd

I will follow up with the tally when the vote is concluded.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: OSSL_PARAM thought

2019-07-02 Thread Richard Levitte
This was actually discussed in the design phase, and I think we
concluded that all arrays of this sort should be terminated rather
than counted.  I frankly cannot remember what the reasoning was behind
this, and suspect it came down to personal taste.

However, whatever way we choose should be a consistent pattern.  The
worst we can do is to have some arrays counted while having others
terminated with a special value.  That's a recipe for confusion.

(I'll note, though, that some arrays must be counted because there is
no special terminating value to be had.  As far as I can recall,
though, this is uniquely for arrays of bytes)

Cheers,
Richard

On Mon, 24 Jun 2019 03:04:21 +0200,
Dr Paul Dale wrote:
> 
> [1  ]
> [2  ]
> We’re only starting out, so there isn’t any issue yet.  I am wondering if 
> instead of terminating
> out OSSL_PARAM arrays with an empty element, would it make sense to pass a 
> size and the array?
> 
> I.e. changing this code sequence (from crypto/evp/digest.c):
> 
> params[i++] = OSSL_PARAM_construct_size_t(OSSL_DIGEST_PARAM_XOFLEN,
>   , NULL);
> params[i++] = OSSL_PARAM_construct_end();
> EVP_MD_CTX_set_params(ctx, params);
> 
> into:
> 
> params[i++] = OSSL_PARAM_construct_size_t(OSSL_DIGEST_PARAM_XOFLEN,
>   , NULL);
> EVP_MD_CTX_set_params(ctx, i, params);
> 
> For fixed arrays OSSL_NELEM would be used instead of the counter variable.
> 
> There are downsides with both approaches of course and neither jumps out as 
> being obviously
> superior.
> 
> To me, at least, it looks like we’re going to have a lot of END’s throughout 
> the codebase.  Saving
> one line many times seems like a win.
> 
> Pauli
> -- 
> Dr Paul Dale | Cryptographer | Network Security & Encryption 
> Phone +61 7 3031 7217
> Oracle Australia
> 
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-14 Thread Richard Levitte
On Fri, 14 Jun 2019 07:13:21 +0200,
Viktor Dukhovni wrote:
> > #define ERR_raise_error ERR_raise_error_internal(__FILE__, __LINE__, 
> > __FUNC__)
> 
> Well, __FUNC__ is entirely non-standard, and __func__ is C99.  Are
> we ready to abandon C89/C90?  If not, then __func__ (and variants)
> becomes compiler-specific.

We've had the argument about C language versions before, and it seems
we're currently staying with C90, 'cause we have some prominent users
that are stuck with that version.

> In test/testutil.h, we have some of the requisite gymnastics:
> 
> # if !defined(__STDC_VERSION__) || __STDC_VERSION__ < 199901L
> #  if defined(_MSC_VER)
> #   define TEST_CASE_NAME __FUNCTION__
> #  else
> #   define testutil_stringify_helper(s) #s
> #   define testutil_stringify(s) testutil_stringify_helper(s)
> #   define TEST_CASE_NAME __FILE__ ":" testutil_stringify(__LINE__)
> #  endif/* _MSC_VER */
> # else
> #  define TEST_CASE_NAME __func__
> # endif /* __STDC_VERSION__ */
> 
> While the GCC manual: 
> http://gcc.gnu.org/onlinedocs/gcc-4.8.1/gcc/Function-Names.html
> suggests:
> 
>  #if __STDC_VERSION__ < 199901L
>  # if __GNUC__ >= 2
>  #  define __func__ __FUNCTION__
>  # else
>  #  define __func__ ""
>  # endif
>  #endif
> 
> we would also need similar for any other pre-C99 supported compilers.

Yes.  We already have our own aliases for __FILE__ and __LINE__ (see
include/openssl/opensslconf.h.in), we can simply add an OPENSSL_FUNC
that's appropriately defined depending on what the compiler supports.

Side note: opensslconf.h.in isn't the best place for this kind of
macro, not even for OPENSSL_FILE and OPENSSL_LINE.  I'd rather move
all that to e_os2.h.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
On Thu, 13 Jun 2019 22:04:17 +0200,
Roumen Petrov wrote:
> 
> Richard Levitte wrote:
> > On Thu, 13 Jun 2019 20:23:05 +0200,
> > Roumen Petrov wrote:
> >> Hello,
> >> 
> >> First I did not understand what is wrong to use function or
> >> reasons. All of them are subject to particular "library".
> > We didn't say anything against reason codes.  As a matter of fact, I
> > do find those useful.  Function codes, on the other hand, are unstable
> > and thereby quite useless, at least for figuring out errors.
> I'm not aware of OpenSSL documentation that describes reason codes as
> stable (fixed).

Perhaps, but that's not saying they aren't...  The reason codes,
specifically, have been fairly stable over time.  However, we can get
better.  Even better, we could get better at documenting them.

> If code is rewritten not only functions are changed. More or less
> reasons are changed as well.

Sometimes, that's true, and especially as new functionality has come
up.  We could certainly do a better job there by getting better at
giving consistent and more well defined reason codes.

> >> To parse openssl error code in an application code is bad practice.
> > Is it?  Would you say it's bad practice to look at errno codes too?
> No as already in one of my post I wrote that  errsrt utility is very useful.

I'm not talking about display for the moment.

> Bad practice is to check that reason code encoded into 'error' code
> has value of A or B.

Why?  Why would it be bad practice to check if the reason for an error
is ERR_R_MALLOC_FAILURE, for example?

You can make the exact same analogy with errno and checking its
values programatically.

git grep 'errno *[!=]='

Considering the result of the above command, would you say that we're
having bad practice in OpenSSL code?

> >> But developer could  format "extra message" for instance :
> >>      NSSerr(NSS_F_RSA_SIGN, NSS_R_UNSUPPORTED_NID);
> >>      {/*add extra error message data*/
> >>      char msgstr[10];
> >>      BIO_snprintf(msgstr, sizeof(msgstr), "%d", dtype);
> >>      ERR_add_error_data(2, "NID=", msgstr);
> >>      }
> > I'll counter with a (yet fictitious):
> > 
> >  ERR_raise_error(NSS_R_UNSUPPORED_NID, "NID=%d", dtype);
> Reason code could be shared between functions :

Err, yeah?  Of course they can!

> > or if you want to add information that helps debugging:
> > 
> >  ERR_raise_error_debug(NSS_R_UNSUPPORED_NID,
> >__FILE__, __LINE__, __FUNC__, "$Format:%H",
> >"NID=%d", dtype);
> 
> This look like complete different solution. __FILE__ exposes some
> 'private'  information (build related) and is some cases is not
> acceptable .

Maybe you should look at the WHATEVERerr macros, here's one:

# define SYSerr(f,r)  
ERR_PUT_error(ERR_LIB_SYS,(f),(r),OPENSSL_FILE,OPENSSL_LINE)

OPENSSL_FILE and OPENSSL_LINE are macros that are aliases for
__FILE__ and __LINE__, if those are available.

So we already do expose that "private information".  How other
applications call ERR_PUT_error() is none of our business.

> __FUNC__ does not look portable  - __func__ vs __FUNCTION__ vs "not
> defined".

It was just an example, to give you an idea...  I didn't test it or
check it for correctness.

> Also going into this direction question is why to use "reason code".

It's to allow calling applications to react differently depending on
the reason for an error to occur.  How else would you have them know?

> Functionality similar to errno, sys_errlist  is also outdated.

Ok, so what kind of functionality do you want to offer?  Remember,
we're talking about a language that doesn't have much error handling
per se.  More advanced languages have an exception system, but even
there, the applicatin can catch them and react differently depending
on what the error is, so apart from the out-of-band nature of an
exception system (and automated unwinding), the difference isn't
really that big.

> > (since this is just an idea so far, it's perfectly possible to throw
> > in a library code as well, but like I said already, dynamic library
> > codes have their own issue)
> > 
> > I know which I find easier to write.
> > 
> >> There is no reason OpenSSL code to use ERR_add_error_data as usually
> >> library packs whole functionality.
> > That's not really true, there are places where OpenSSL code does use
> > ERR_add_error_data(), and for good reason.
> Hmm, you cut my note for externals like IO CAPI ;)

Because I wasn't arguing that.  I was arguing that what you were
saying about OpenSSL code is incorrect.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
On Thu, 13 Jun 2019 20:23:05 +0200,
Roumen Petrov wrote:
> 
> Hello,
> 
> First I did not understand what is wrong to use function or
> reasons. All of them are subject to particular "library".

We didn't say anything against reason codes.  As a matter of fact, I
do find those useful.  Function codes, on the other hand, are unstable
and thereby quite useless, at least for figuring out errors.

> To parse openssl error code in an application code is bad practice.

Is it?  Would you say it's bad practice to look at errno codes too?

Error *text* is a different matter.

> Richard Levitte wrote:
> > On Thu, 13 Jun 2019 12:01:46 +0200,
> > Matt Caswell wrote:
> >>>   The
> >>> additional information you're talking about is something we currently
> >>> provide the ERR_add_error_data() function for, and that together with
> >>> the reason text (derived from the reason code) is the data the end
> >>> user can reasonably get.  It's up to whoever writes the error raising
> >>> code to provide enough useful information.
> >> Yes, although in practice we don't currently do this (we very rarely add
> >> additional explanatory text). Not sure if that is a problem with the API, 
> >> our
> >> coding standards, or our culture.
> > This is partly historical...  ERR_add_error_data() has been around
> > since the beginning of time, and it looks to me like it was designed
> > in a time where varargs hadn't universally caught on yet (yes, there
> > was a time before varargs, and it's appropriate to call that "the
> > stone age").
> 
> But developer could  format "extra message" for instance :
>     NSSerr(NSS_F_RSA_SIGN, NSS_R_UNSUPPORTED_NID);
>     {/*add extra error message data*/
>     char msgstr[10];
>     BIO_snprintf(msgstr, sizeof(msgstr), "%d", dtype);
>     ERR_add_error_data(2, "NID=", msgstr);
>     }

I'll counter with a (yet fictitious):

ERR_raise_error(NSS_R_UNSUPPORED_NID, "NID=%d", dtype);

or if you want to add information that helps debugging:

ERR_raise_error_debug(NSS_R_UNSUPPORED_NID,
  __FILE__, __LINE__, __FUNC__, "$Format:%H",
  "NID=%d", dtype);

(since this is just an idea so far, it's perfectly possible to throw
in a library code as well, but like I said already, dynamic library
codes have their own issue)

I know which I find easier to write.

> There is no reason OpenSSL code to use ERR_add_error_data as usually
> library packs whole functionality.

That's not really true, there are places where OpenSSL code does use
ERR_add_error_data(), and for good reason.  BIO_lookup and friends add
the extra resolver error on error, the CONF code adds information on
exactly what value causes a configuration file parsing error, the DSO
code adds information on exactly what file it attempted to load (full
path if possible), etc etc etc...  those are things that can't be part
of the error code.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
On Thu, 13 Jun 2019 18:26:41 +0200,
Salz, Rich wrote:
> 
> >The proper way to handle this, in my experience, is *DO NOT REUSE ERROR 
> > CODES.* Each error code appears in exactly one place, and we could 
> > eventually build up documentation explaining what they mean, the causes, 
> > and how to address this. This means, we don't use ERR_R_MALLOC when trying 
> > to create an RSA key, for example, but rather a handful of new errors for 
> > ERR_R_RSA_CANT_CREATE_D, ...CANT_CREATE_N, etc.  That is a big job, albeit 
> > mostly a tedious one.
>  
> I got some feedback on- and off-list about this. Most of it was
> "surely you can't be serious."  I am, and stop calling me
> Shirley. :) Let me add some details.  First, recall that OpenSSL has
> an error stack, and that as errors are "unwound" each function can
> add its own error code to that stack. This naturally leads to the
> point where the first entry has the most detailed error, "malloc
> failed" and the last entry has the most application-oriented error
> "Could not create RSA key"; along the way are "Could not create d"
> and "Could not create secure bignum" errors.  I hope that makes more
> sense.
> 
> HOWEVER, this point (which got the most comments) was a side-note to
> the main point of my email, which gave some reasons why I think
> including the function code is a bad idea.

So basically, what you're actually saying is that we should add
additional errors up the call stack...  so if some function called
OPENSSL_zalloc(), it should be perfectly OK to raise a
ERR_R_MALLOC_FAILURE, but functions above should *add* things like
WHATEVER_R_KEY_CREATE_FAILURE, etc etc etc, thereby creating that
backtrace that Tim talks about.

A point here is that the application may want to find out if there was
an allocation error -- at which point it might choose to simply bail --
or some other error that prevented the key to be created (insecure
amount of bits, say?) -- at which point it might choose to try and
mitigate.  The question is what's easier for the application, getting
the wanted information as a top error or having to walk to the bottom
of the error stack to figure things out.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
On Thu, 13 Jun 2019 12:01:46 +0200,
Matt Caswell wrote:
> >  The
> > additional information you're talking about is something we currently
> > provide the ERR_add_error_data() function for, and that together with
> > the reason text (derived from the reason code) is the data the end
> > user can reasonably get.  It's up to whoever writes the error raising
> > code to provide enough useful information.
> 
> Yes, although in practice we don't currently do this (we very rarely add
> additional explanatory text). Not sure if that is a problem with the API, our
> coding standards, or our culture.

This is partly historical...  ERR_add_error_data() has been around
since the beginning of time, and it looks to me like it was designed
in a time where varargs hadn't universally caught on yet (yes, there
was a time before varargs, and it's appropriate to call that "the
stone age").

In today's coding practices, I personally find ERR_add_error_data()
clumsy to deal with, so I seldom use it.  Also, being a separate
function, it's easy to forget that it's there and useful.  That's a
reason to think that having all integrated in one function call that
includes the flexibility of BIO_printf() probably would encourage
producing better information.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
On Thu, 13 Jun 2019 11:16:38 +0200,
Matt Caswell wrote:
> 
> I agree with everything Richard just said. I just have some additional 
> thoughts
> inserted below.
> 
> On 13/06/2019 10:00, Richard Levitte wrote:
> > If we look at it from the perspective of the application author,
> > what's most often needed are reliable error/reason codes (to check and
> > to react appropriatly to) and associated reason strings (for display).
> > For the application author, it would normally be completely
> > uninteresting exactly which function the error happened in (think
> > about it, when's the last time you wanted to know exactly which
> > internal function in libc raised the EAGAIN you just got?).  For the
> > application author, the interesting bit is usually "what went wrong?"
> > (reason) and "what did I call?" (they already know that).
> > 
> > 
> > If we look at *our* needs as library writers, then the exact location
> > that raised the error is interesting, of course!  But then, is it
> > interesting in the form of codes, or are we rather interested in the
> > text form?  I dunno about you, but I don't give a damn about the
> > actual library and function codes, I look at the displayed library
> > names and function names.  (and yet, they aren't necessarily enough
> > without knowing the OpenSSL version).
> > 
> > 
> > Finally, if we look at it from a provider author's perspective,
> > dealing with all these different codes is quite hard.  Ideally, a
> > provider author should just need to deal with their own reason codes
> > and associated strings, and that's about it.
> 
> I think you missed another important perspective, i.e. that of the end user.
> 
> Note that this is slightly different to the needs of the application author. 
> An
> application author will typically do one of two things if an error occurs:
> 
> 1) Try and figure out the cause of the error and recover from it, e.g. if I 
> get
> reason code x then I'll retry the operation with slightly different parameters
> but if I get y then I'll just ignore this failure and carry on anyway.
> 
> or
> 
> 2) If its an error they don't know how to handle then give up and display
> whatever error details are available to the end user (or log them, or 
> whatever)
> 
> So the application author is going to want a small number of reason codes that
> they can handle. If its not in the small set they know about, then give up.
> 
> The end user OTOH doesn't care about reason codes at all and wants as detailed
> an explanation of the error (in text form) as is possible. Potentially that
> might mean different text for every point in the code where we raise an error.
> Ideally they might want information about the context that led up to the 
> error,
> i.e. what particular operation was being attempted at the time. I'm thinking
> that there is some relationship here between the information available from 
> the
> trace API, and the error API.

Good point, but note that there is no conflict with what I wrote.  The
additional information you're talking about is something we currently
provide the ERR_add_error_data() function for, and that together with
the reason text (derived from the reason code) is the data the end
user can reasonably get.  It's up to whoever writes the error raising
code to provide enough useful information.

Incidently, I did mention thinking about a new (better, I hope) error
raising function on github, and might as well repeat it here:

int ERR_raise_error(int reason, const char *fmt, ...);

To allow for debugging information (like I mentioned, some sensible
combination of __FILE__, __LINE__, __FUNC__, and possibly git commit
id), one might imagine this:

int ERR_add_debug_info(const char *file, size_t line,
   const char *func, const char *commitid);

or a combo of both:

int ERR_raise_error_debug(int reason,
  const char *file, size_t line,
  const char *func, const char *commitid,
  const char *fmt, ...);

Note that this doesn't have to conflict with the current error raising
code, all we need to do is to convert whatever input ERR_put_error()
gets into the new form.

Cheers,
Richard

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


Re: Removing function names from errors (PR 9058)

2019-06-13 Thread Richard Levitte
The discussion so far has touched on a number of items that aren't
necessarily the same thing:

- the instability of function codes.
- debugging information
- display of errors

and then, the point that no one talks about but that's the main reason
error codes in general exist at all:

- an application's need to figure out what went wrong and how to react


The current error codes have these items added together:

- library code (I'll touch on that further down)
- function code (debugging information)
- reason code


If we look at it from the perspective of the application author,
what's most often needed are reliable error/reason codes (to check and
to react appropriatly to) and associated reason strings (for display).
For the application author, it would normally be completely
uninteresting exactly which function the error happened in (think
about it, when's the last time you wanted to know exactly which
internal function in libc raised the EAGAIN you just got?).  For the
application author, the interesting bit is usually "what went wrong?"
(reason) and "what did I call?" (they already know that).


If we look at *our* needs as library writers, then the exact location
that raised the error is interesting, of course!  But then, is it
interesting in the form of codes, or are we rather interested in the
text form?  I dunno about you, but I don't give a damn about the
actual library and function codes, I look at the displayed library
names and function names.  (and yet, they aren't necessarily enough
without knowing the OpenSSL version).


Finally, if we look at it from a provider author's perspective,
dealing with all these different codes is quite hard.  Ideally, a
provider author should just need to deal with their own reason codes
and associated strings, and that's about it.


What I would like to see is a simplified error system that delivers
reason codes, and attached extra text information.  That text
information could have two parts, where one is debugging information
(__FILE__, __LINE__, __FUNC__, and possibly exact commit hash
generated with 'git archive', see "export-subst" in gitattributes(5)),
and the other is the extra data added with the likes of
ERR_add_error_data().
We certainly already have the infrastructure to allow this, the
modifications needed wouldn't be that big.


On the subject of 'openssl errstr', it's abssolutely fine for errors
raised within our libraries, but outside of that, it's useless.  This
is an issue with engines that no one has spoken about but that has
been there all along.  See, engines, if they use our error reporting
system at all, usually allocate a new library code dynamically, by
calling ERR_get_next_error_library().  Since the library code may
differ from one time to the next, the complete error codes raised will
differ, giving 'openssl errstr' an impossible job, even if the engine
is loaded.
This issue is exactly the same for provider modules.  This makes
library codes less than useful outside of our libraries.


I digress..  the main point is, however, that we need to keep the
application author's needs from our needs.  That affects how we look
at error codes.


Cheers,
Richard


On Wed, 12 Jun 2019 05:51:44 +0200,
Richard Levitte wrote:
> 
> Many of us, both past and present, have looked at the error reporting
> code and wante to change it somehow.  There's currently a PR, 9058,
> which covers one aspect, the function name indicator.
> 
> A -1 was raised early on for the purpose of starting a discussion, and
> eventually (possibly?) a vote.
> 
> A discussion point in that PR is whether it's still interesting to
> keep information on the system function/callback that was called when
> we're reporting a system error, i.e. this type of error report:
> 
> SYSerr(SYS_F_FSTAT, errno);
> 
> (incidently, there's another PR, 9072, which changes those to
> 'SYSerr("fstat", errno)')
> 
> So, the main points of discussion seem to be:
> 
> - should we remove function indicators in our error reporting?
> - should we make an exception for system errors (SYSerr())?
> 
> Cheers,
> Richard
> 
> P.S. I'm personally not entirely sure that we need an actual vote, if
> this discussion shows that we all agree.  Perhaps to confirm and make
> the decision formally solid?
> 
> -- 
> Richard Levitte levi...@openssl.org
> OpenSSL Project http://www.openssl.org/~levitte/
> 
-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/


  1   2   3   >