Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[Willy Tarreau]

On Tue, Jul 18, 2023 at 10:43:57AM +0200, William Lallemand wrote:
> On Tue, Jul 18, 2023 at 09:11:33AM +0200, Willy Tarreau wrote:
> > I'll let the SSL maintainers check all this, but my sentiment is that in
> > general if there are differences between the libs, it would be better if
> > we have a special define for this one as well. It's easier to write and
> > maintain "#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC)"
> > than making it appear sometimes as one of them, sometimes as the other.
> > That's what we had a long time ago and it was a real pain, every single
> > move in any lib would cause breakage somewhere. Being able to reliably
> > identify a library and handle its special cases is much better.
>  
> I agree, we could even add a build option OPENSSL_AWSLC=1 like we've
> done with wolfssl, since this is a variant of the Openssl API. Then
> every supported features could be activated with the HAVE_SSL_* defines
> in openssl-compat.h. Discovering the features with libreSSL and
> boringSSL version defines was a real mess, we are probably going to end
> up with a matrix of features supported by different libraries.

Very good point, and I totally agree (with this and the rest of your
proposals).

Willy

Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[William Lallemand]

On Tue, Jul 18, 2023 at 09:11:33AM +0200, Willy Tarreau wrote:
> I'll let the SSL maintainers check all this, but my sentiment is that in
> general if there are differences between the libs, it would be better if
> we have a special define for this one as well. It's easier to write and
> maintain "#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC)"
> than making it appear sometimes as one of them, sometimes as the other.
> That's what we had a long time ago and it was a real pain, every single
> move in any lib would cause breakage somewhere. Being able to reliably
> identify a library and handle its special cases is much better.
 
I agree, we could even add a build option OPENSSL_AWSLC=1 like we've
done with wolfssl, since this is a variant of the Openssl API. Then
every supported features could be activated with the HAVE_SSL_* defines
in openssl-compat.h. Discovering the features with libreSSL and
boringSSL version defines was a real mess, we are probably going to end
up with a matrix of features supported by different libraries.

I'm seeing multiple defines that can be useful in haproxy:

- OPENSSL_IS_AWSLC could be used as Willy said, that could enough and we
  maybe won't need the build option.

- OPENSSL_VERSION_NUMBER it seems to be set to 0x1010107f but is this
  100% compatible with the openssl 1.1.1 API?

- AWSLC_VERSION_NUMBER_STRING It seems to be the OPENSSL_VERSION_TEXT
  counterpart but I don't see the equivalent as a number, in
  OpenSSL there is OPENSSL_VERSION_NUMBER which is used for doing #if
  (OPENSSL_VERSION_NUMBER >= 0x1010107f) in the code for example, this
  is really important for maintenance if we want to support multiple
  versions of aws-lc.

- AWSLC_API_VERSION maybe this would be enough instead of the
  VERSION_NUMBER. We could activate the HAVE_SSL_* defines using
  OPENSSL_VERSION_NUMBER and this.

> > To Alex's concern on API compatibility: yes AWS-LC is aiming to provide a
> > more stable API. We already run integration tests with 6 other projects [2]
> > including HAProxy. This will help ensure API compatibility going forward.
> > What is your specific concern with ABI compatibility? Are you looking to 
> > take
> > the haproxy executable built with OpenSSL libcrypto/libssl and drop in 
> > AWS-LC
> > without recompiling haproxy? Or do that between AWS-LC libcrypto/libssl
> > versions?
> 
> I personally have no interest in cross-libs ABI compatibility because
> that does not make much sense, particularly when considering that Openssl
> does not support QUIC so by definition there will be many symbol-level
> differences. Regarding aws-lc's libs over time, yes for the users it
> would be desirable that within a stable branch it's possible to update
> the library or the application in any order without having to rebuild
> the application. We all know that it's something that only becomes
> possible once the lib stabilizes enough to avoid invasive backports in
> stable branches. I don't know what the current status is for aws-lc's
> stable branches at the moment.
> 

Agreed, cross-libs ABI is not useful, but the ABI should remain stable
between minor releases so the library package could be updated without
rebuilding every software that depends on it.

Regards,


-- 
William Lallemand

Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[Илья Шипицин]

вт, 18 июл. 2023 г. в 09:14, Willy Tarreau :

> Hi Andrew,
>
> On Tue, Jul 18, 2023 at 06:26:45AM +, Hopkins, Andrew wrote:
> > Willy you're correct. AWS-LC does have support for the QUIC primitives
> > HAProxy  needs, we just need to fix some of the names [1] in either
> HAProxy's
> > code or AWS-LC in a follow up change.
>
> OK, thanks for confirming :-)
>
> I'll let the SSL maintainers check all this, but my sentiment is that in
> general if there are differences between the libs, it would be better if
> we have a special define for this one as well. It's easier to write and
> maintain "#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC)"
>

AWSLC_VERSION_NAME ?

aws-lc/include/openssl/base.h at 1dd5cf92e96edd4092bc307b14969dae5eaaa507 ·
aws/aws-lc (github.com)



> than making it appear sometimes as one of them, sometimes as the other.
> That's what we had a long time ago and it was a real pain, every single
> move in any lib would cause breakage somewhere. Being able to reliably
> identify a library and handle its special cases is much better.
>
> > To Alex's concern on API compatibility: yes AWS-LC is aiming to provide a
> > more stable API. We already run integration tests with 6 other projects
> [2]
> > including HAProxy. This will help ensure API compatibility going forward.
> > What is your specific concern with ABI compatibility? Are you looking to
> take
> > the haproxy executable built with OpenSSL libcrypto/libssl and drop in
> AWS-LC
> > without recompiling haproxy? Or do that between AWS-LC libcrypto/libssl
> > versions?
>
> I personally have no interest in cross-libs ABI compatibility because
> that does not make much sense, particularly when considering that Openssl
> does not support QUIC so by definition there will be many symbol-level
> differences. Regarding aws-lc's libs over time, yes for the users it
> would be desirable that within a stable branch it's possible to update
> the library or the application in any order without having to rebuild
> the application. We all know that it's something that only becomes
> possible once the lib stabilizes enough to avoid invasive backports in
> stable branches. I don't know what the current status is for aws-lc's
> stable branches at the moment.
>
> > Willy that's an interesting idea for library name conflict avoidance. I
> did
> > not know that's how quictls solved this problem, it seems like it would
> work
> > for simple applications with only one dependency on libcrypto (such as
> > HAProxy). However, I don't think it would solve the issue of transitive
> > dependencies mixing libcrypto implementations, e.g. an application
> depends on
> > 2 libraries that each depend on libcrypto, if one library moves to
> AWS-LC the
> > application will get symbol conflicts.
>
> Sure, but that application couldn't work otherwise anyway if some of
> its dependencies rely on openssl's libcrypto, because if you replace
> it with yours it'll get a different one than the one that was expected
> by all the macros, struct definitions and inline funtions anyway. That's
> already the problem we've faced with quictls: some applications with many
> dependencies (e.g. curl) need to have all these dependencies rebuilt
> against it anyway, so the application and its dependencies need to be
> explicitly built against the new lib. And at least with a distinct
> version number you don't need to rebuild the whole system, only the
> components that directly depend on aws-lc and their dependencies. This
> also allows to know better what depends on what.
>
> Just my two cents,
> Willy
>
>

Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[Willy Tarreau]

Hi Andrew,

On Tue, Jul 18, 2023 at 06:26:45AM +, Hopkins, Andrew wrote:
> Willy you're correct. AWS-LC does have support for the QUIC primitives
> HAProxy  needs, we just need to fix some of the names [1] in either HAProxy's
> code or AWS-LC in a follow up change. 

OK, thanks for confirming :-)

I'll let the SSL maintainers check all this, but my sentiment is that in
general if there are differences between the libs, it would be better if
we have a special define for this one as well. It's easier to write and
maintain "#if defined(OPENSSL_IS_BORINGSSL) || defined(OPENSSL_IS_AWSLC)"
than making it appear sometimes as one of them, sometimes as the other.
That's what we had a long time ago and it was a real pain, every single
move in any lib would cause breakage somewhere. Being able to reliably
identify a library and handle its special cases is much better.

> To Alex's concern on API compatibility: yes AWS-LC is aiming to provide a
> more stable API. We already run integration tests with 6 other projects [2]
> including HAProxy. This will help ensure API compatibility going forward.
> What is your specific concern with ABI compatibility? Are you looking to take
> the haproxy executable built with OpenSSL libcrypto/libssl and drop in AWS-LC
> without recompiling haproxy? Or do that between AWS-LC libcrypto/libssl
> versions?

I personally have no interest in cross-libs ABI compatibility because
that does not make much sense, particularly when considering that Openssl
does not support QUIC so by definition there will be many symbol-level
differences. Regarding aws-lc's libs over time, yes for the users it
would be desirable that within a stable branch it's possible to update
the library or the application in any order without having to rebuild
the application. We all know that it's something that only becomes
possible once the lib stabilizes enough to avoid invasive backports in
stable branches. I don't know what the current status is for aws-lc's
stable branches at the moment.

> Willy that's an interesting idea for library name conflict avoidance. I did
> not know that's how quictls solved this problem, it seems like it would work
> for simple applications with only one dependency on libcrypto (such as
> HAProxy). However, I don't think it would solve the issue of transitive
> dependencies mixing libcrypto implementations, e.g. an application depends on
> 2 libraries that each depend on libcrypto, if one library moves to AWS-LC the
> application will get symbol conflicts.

Sure, but that application couldn't work otherwise anyway if some of
its dependencies rely on openssl's libcrypto, because if you replace
it with yours it'll get a different one than the one that was expected
by all the macros, struct definitions and inline funtions anyway. That's
already the problem we've faced with quictls: some applications with many
dependencies (e.g. curl) need to have all these dependencies rebuilt
against it anyway, so the application and its dependencies need to be
explicitly built against the new lib. And at least with a distinct
version number you don't need to rebuild the whole system, only the
components that directly depend on aws-lc and their dependencies. This
also allows to know better what depends on what.

Just my two cents,
Willy

Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[Hopkins, Andrew]

Willy you're correct. AWS-LC does have support for the QUIC primitives HAProxy  
needs, we just need to fix some of the names [1] in either HAProxy's code or 
AWS-LC in a follow up change. 

To Alex's concern on API compatibility: yes AWS-LC is aiming to provide a more 
stable API. We already run integration tests with 6 other projects [2] 
including HAProxy. This will help ensure API compatibility going forward.  What 
is your specific concern with ABI compatibility? Are you looking to take the 
haproxy executable built with OpenSSL libcrypto/libssl and drop in AWS-LC 
without recompiling haproxy? Or do that between AWS-LC libcrypto/libssl 
versions?

Willy that's an interesting idea for library name conflict avoidance. I did not 
know that's how quictls solved this problem, it seems like it would work for 
simple applications with only one dependency on libcrypto (such as HAProxy). 
However, I don't think it would solve the issue of transitive dependencies 
mixing libcrypto implementations, e.g. an application depends on 2 libraries 
that each depend on libcrypto, if one library moves to AWS-LC the application 
will get symbol conflicts.

[1] 
https://github.com/haproxy/haproxy/blob/master/include/haproxy/quic_tls-t.h#L28
[2] https://github.com/aws/aws-lc/tree/main/tests/ci/integration


On 7/15/23, 1:41 AM, "Willy Tarreau" mailto:w...@1wt.eu>> wrote:


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.






Hi Alex, Andrew,


On Thu, Jul 13, 2023 at 11:54:44AM +0200, Aleksandar Lazic wrote:
> On 2023-07-13 (Do.) 08:22, Hopkins, Andrew wrote:
> > * Do you plan to add quic (Server part) faster then OpenSSL?
> >
> > I have not looked into quic benchmarks but it uses the same
> > cryptographic primitives as TLS so I imagine we'd be faster for a lot of
> > the algorithms. It might not be useful for HAProxy which is all C, but
> > AWS also launched s2n-quic [1] which does have extensive testing for
> > correctness and performance. s2n-quic evenuses AWS-LC's libcrypto for
> > all of the cryptographic operations [2] though our rust bindings
> > aws-lc-rs [3].
>
> Hm, this implies a dependency for rust which increases the complexity to
> build HAProxy. From my point of view isn't this very helpfull to bring the
> library into haproxy.


I think there was a misunderstanding between you two above. I understand
Alex' question as "will you merge quic support soon", but my understanding
last time I had the opportunity to discuss about AWS-LC was that it *is*
already there. Alex, what Andrew mentioned above is that their own QUIC
implementation, s2n-quic, uses their rust bindings, but we don't need
s2n-quic (since we have our own stack) and this will not add any
dependency.


> > * Will be there some packages for debian/ubuntu/RHEL/... so that the
> > users of HAProxy can "just install and run" HAProxy with that SSL Lib?
> >
> > In the near future no. Currently AWS-LC does not support enough packages
> > to fully replace libcrypto for the entire operating system, and
> > balancing different programs using different library paths and libcrypto
> > implementations is tricky. Eventually distributing static archives and
> > shared libraries once we have more support makes sense. There is more
> > context/history in this issue [4].
>
> Uh that's a show stopper, at least from my point of view. This implies the
> same work as HAProxy team have for wolfssl, BoringSSL and quictls and that's
> a lot of work.


I think that what would be important would be to find a package maintainer
willing to maintain this into a distro for the whole maintenance life cycle
of that distro. It also helps a lot in designing stable and durable APIs
that boost adoption, because the early maintenance trouble are quickly
learned during backports and everyone is more careful over time.


One thing that should really be avoided would be to name the library as
the regular openssl one, because while it helps with *early* adoption,
it complicates everything over time and for all packages. For example,
quictls adopted the .81 suffix (81 for "Q") and as such there's no
conflict on the shared lib. For building, it's as simple as passing the
SSL_INC/SSL_LIB during the build (and basically all software supporting
openssl support this), so there's no trouble either, and all libs can
coexist on the same distro. But yeah, it's really important to find
someone willing to maintain such a package long enough for a distro,
that fills a hole that the whole world is waiting to be plugged (since
we know it will not come from openssl anyway).


The partial support from various software is not a problem as long as
openssl and aws-lc can coexist on the system. Some will just be built
with one and others with the other. This has always existed and I would
be surprised if there's no software linked against GNUtls on distros
that ship it for example. So 

Re: [PATCH] BUILD: ssl: Build with new cryptographic library AWS-LC

[Hopkins, Andrew]

This is currently expected. AWS-LC inherited the different names from BoringSSL 
and AWS-LC doesn’t define OPENSSL_IS_BORINGSSL. So, in the HAProxy QUIC code 
here [1] we fall into the OpenSSL branch and the names don’t line up. In a 
follow up change we could tweak the if/def or AWS-LC could streamline the 
names. This also came up in the AWS-LC CI PR [2].

[1] 
https://github.com/haproxy/haproxy/blob/master/include/haproxy/quic_tls-t.h#L28
[2] https://github.com/aws/aws-lc/pull/1097#discussion_r1265953494


From: Илья Шипицин 
Date: Monday, July 17, 2023 at 3:09 PM
To: Willy Tarreau 
Cc: Aleksandar Lazic , "Hopkins, Andrew" 
, "haproxy@formilux.org" 
Subject: RE: [EXTERNAL][PATCH] BUILD: ssl: Build with new cryptographic library 
AWS-LC


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.




сб, 15 июл. 2023 г. в 10:44, Willy Tarreau mailto:w...@1wt.eu>>:
Hi Alex, Andrew,

On Thu, Jul 13, 2023 at 11:54:44AM +0200, Aleksandar Lazic wrote:
> On 2023-07-13 (Do.) 08:22, Hopkins, Andrew wrote:
> > * Do you plan to add quic (Server part) faster then OpenSSL?
> >
> > I have not looked into quic benchmarks but it uses the same
> > cryptographic primitives as TLS so I imagine we'd be faster for a lot of
> > the algorithms. It might not be useful for HAProxy which is all C, but
> > AWS also launched s2n-quic [1] which does have extensive testing for
> > correctness and performance. s2n-quic evenuses AWS-LC's libcrypto for
> > all of the cryptographic operations [2] though our rust bindings
> > aws-lc-rs [3].
>
> Hm, this implies a dependency for rust which increases the complexity to
> build HAProxy. From my point of view isn't this very helpfull to bring the
> library into haproxy.

I think there was a misunderstanding between you two above. I understand
Alex' question as "will you merge quic support soon", but my understanding
last time I had the opportunity to discuss about AWS-LC was that it *is*
already there. Alex, what Andrew mentioned above is that their own QUIC
implementation, s2n-quic, uses their rust bindings, but we don't need
s2n-quic (since we have our own stack) and this will not add any
dependency.

I tried to build with USE_QUIC=1:

[ilia@fedora haproxy]$ sh go.sh
  CC  src/quic_conn.o
  CC  src/h3.o
  CC  src/xprt_quic.o
  CC  src/quic_frame.o
  CC  src/quic_tls.o
In file included from src/quic_conn.c:60:
include/haproxy/quic_tls.h: In function ‘tls_aead’:
include/haproxy/quic_tls.h:124:14: error: ‘TLS1_3_CK_AES_128_GCM_SHA256’ 
undeclared (first use in this function); did you mean 
‘TLS1_CK_AES_128_GCM_SHA256’?
  124 | case TLS1_3_CK_AES_128_GCM_SHA256:
  |  ^~~~
  |  TLS1_CK_AES_128_GCM_SHA256
compilation terminated due to -Wfatal-errors.
make: *** [Makefile:998: src/quic_conn.o] Error 1
make: *** Waiting for unfinished jobs
[ilia@fedora haproxy]$


> > * Will be there some packages for debian/ubuntu/RHEL/... so that the
> > users of HAProxy can "just install and run" HAProxy with that SSL Lib?
> >
> > In the near future no. Currently AWS-LC does not support enough packages
> > to fully replace libcrypto for the entire operating system, and
> > balancing different programs using different library paths and libcrypto
> > implementations is tricky. Eventually distributing static archives and
> > shared libraries once we have more support makes sense. There is more
> > context/history in this issue [4].
>
> Uh that's a show stopper, at least from my point of view. This implies the
> same work as HAProxy team have for wolfssl, BoringSSL and quictls and that's
> a lot of work.

I think that what would be important would be to find a package maintainer
willing to maintain this into a distro for the whole maintenance life cycle
of that distro. It also helps a lot in designing stable and durable APIs
that boost adoption, because the early maintenance trouble are quickly
learned during backports and everyone is more careful over time.

One thing that should really be avoided would be to name the library as
the regular openssl one, because while it helps with *early* adoption,
it complicates everything over time and for all packages. For example,
quictls adopted the .81 suffix (81 for "Q") and as such there's no
conflict on the shared lib. For building, it's as simple as passing the
SSL_INC/SSL_LIB during the build (and basically all software supporting
openssl support this), so there's no trouble either, and all libs can
coexist on the same distro. But yeah, it's really important to find
someone willing to maintain such a package long enough for a distro,
that fills a hole that the whole world is waiting to be plugged (since
we know it will not come from openssl anyway).

The partial support from various software is not a problem as long as
openssl and aws-lc can coexist on the system. Some will