Re: [squid-dev] OpenSSL 1.1 regression
On 19/05/2017 07:19 μμ, Christos Tsantilas wrote: The t4 patch I committed this patch to squid-5 as r15152. On 19/05/2017 12:27 πμ, Amos Jeffries wrote: On 19/05/17 04:04, Christos Tsantilas wrote: On 18/05/2017 03:40 μμ, Amos Jeffries wrote: On 18/05/17 23:12, Christos Tsantilas wrote: +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + This bit seems to be correct. Given the .cc file sequence of macro tests I think we can speed up ./configure a bit by moving the use of SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. eg. AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) ],[ # check for bugs and hacks in the old OpenSSL API SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS ]) I am attaching a new patch. In this patch I moved the SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS as you suggested. But also my last patch was buggy, the AC_CHECK_LIB did not search at the correct directories for libssl library. In this patch I moved the "SQUID_STATE_ROLLBACK(squid_openssl_state)" line some lines down to have the correct libraries search path. Is it ok, or it is better to open a new SQUID_STATE_SAVE/ROLLBACK just for AC_CHECK_LIB? Ah. Either moving the check which alters compiler environment above the existign ROLLBACK, or a new one. It is important the CXXFLAGS and SSLLIB lines directly above where your patch placed it do not get rolled back. PS. Finally, this easy to fix issue, is one more prove that it is better to not start fixing files involved with this satanic tool called autoconf! :-P Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 21/05/17 11:03, Eliezer Croitoru wrote: I am missing coupe things about the subject and I want to verify it with your all. From my point of view when maintaining the RPM's for couple distributions I am looking at: Would a specific OpenSSL library hit the distributions I maintain or not or just in a couple years? But I am not sure about the concern of the developers since I read something about gcc 6 which is the cutting edge version of gcc to my knowledge. GCC-7 is now the latest. GCC-6 is old enough for some "stable" distributions to be including it. And I want to understand: What is the aim of the Squid-Cache software development team for Versions 3.5, 4.X, 5.X? Each member of the team has different goals/aims. What actually happens is somewhat the intersection of those ideas. As maintainer my goals are to get the last bugs of v4 closed and release it, dropping formal support for v3.5 before April 2024. Also, Do we expect the main line linux distributions to use the cutting edge gcc or OpenSSL or we are just in the stage which we try to patch up things before the actual integration of these tools will be done?(can take even couple years..) Officially we support as many OS as possible for their latest stable distribution release, and pre-stable releases if we can. Older releases and LTS are left to the OS vendors themselves for maintenance or sponsored work. As for those specific mentioned software; * Squid-3.5 are expected to be compatible with GCC-4.*, OpenSSL-0.9.* and OpenSSL-1.0.* (the 'old' libssl API). * Squid-4+ are expected to be compatible with GCC-5+ and OpenSSL-1.*. Of course respectively newer or older versions may work, but that is more luck than intention. Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
I am missing coupe things about the subject and I want to verify it with your all. From my point of view when maintaining the RPM's for couple distributions I am looking at: Would a specific OpenSSL library hit the distributions I maintain or not or just in a couple years? But I am not sure about the concern of the developers since I read something about gcc 6 which is the cutting edge version of gcc to my knowledge. And I want to understand: What is the aim of the Squid-Cache software development team for Versions 3.5, 4.X, 5.X? Also, Do we expect the main line linux distributions to use the cutting edge gcc or OpenSSL or we are just in the stage which we try to patch up things before the actual integration of these tools will be done?(can take even couple years..) Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-dev [mailto:squid-dev-boun...@lists.squid-cache.org] On Behalf Of Christos Tsantilas Sent: Friday, May 19, 2017 7:20 PM To: squid-dev@lists.squid-cache.org Subject: Re: [squid-dev] OpenSSL 1.1 regression The t4 patch On 19/05/2017 12:27 πμ, Amos Jeffries wrote: > On 19/05/17 04:04, Christos Tsantilas wrote: >> On 18/05/2017 03:40 μμ, Amos Jeffries wrote: >>> On 18/05/17 23:12, Christos Tsantilas wrote: >>>> +# check for API functions >>>> +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, >>>> [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate >>>> is available])], []) >>>> + >>> >>> This bit seems to be correct. >>> >>> Given the .cc file sequence of macro tests I think we can speed up >>> ./configure a bit by moving the use of >>> SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. >>> >>> eg. >>> >>> AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ >>> AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate >>> is available]) >>> ],[ >>> # check for bugs and hacks in the old OpenSSL API >>> SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS >>> ]) >> >> I am attaching a new patch. >> In this patch I moved the SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS as >> you suggested. >> >> But also my last patch was buggy, the AC_CHECK_LIB did not search at >> the correct directories for libssl library. >> >> In this patch I moved the "SQUID_STATE_ROLLBACK(squid_openssl_state)" >> line some lines down to have the correct libraries search path. >> Is it ok, or it is better to open a new SQUID_STATE_SAVE/ROLLBACK just >> for AC_CHECK_LIB? > > Ah. Either moving the check which alters compiler environment above the > existign ROLLBACK, or a new one. It is important the CXXFLAGS and SSLLIB > lines directly above where your patch placed it do not get rolled back. > > >> >> >> PS. Finally, this easy to fix issue, is one more prove that it is >> better to not start fixing files involved with this satanic tool >> called autoconf! >> > > :-P > > Amos > > ___ > squid-dev mailing list > squid-dev@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-dev ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
The t4 patch On 19/05/2017 12:27 πμ, Amos Jeffries wrote: On 19/05/17 04:04, Christos Tsantilas wrote: On 18/05/2017 03:40 μμ, Amos Jeffries wrote: On 18/05/17 23:12, Christos Tsantilas wrote: +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + This bit seems to be correct. Given the .cc file sequence of macro tests I think we can speed up ./configure a bit by moving the use of SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. eg. AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) ],[ # check for bugs and hacks in the old OpenSSL API SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS ]) I am attaching a new patch. In this patch I moved the SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS as you suggested. But also my last patch was buggy, the AC_CHECK_LIB did not search at the correct directories for libssl library. In this patch I moved the "SQUID_STATE_ROLLBACK(squid_openssl_state)" line some lines down to have the correct libraries search path. Is it ok, or it is better to open a new SQUID_STATE_SAVE/ROLLBACK just for AC_CHECK_LIB? Ah. Either moving the check which alters compiler environment above the existign ROLLBACK, or a new one. It is important the CXXFLAGS and SSLLIB lines directly above where your patch placed it do not get rolled back. PS. Finally, this easy to fix issue, is one more prove that it is better to not start fixing files involved with this satanic tool called autoconf! :-P Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev Squid crashes when ServerFirst bumping mode is used with openSSL-1.1.0 release When OpenSSL-1.1.0 or later is used: - The SQUID_USE_SSLGETCERTIFICATE_HACK configure test is false - The SQUID_SSLGETCERTIFICATE_BUGGY configure test is true - Squid hits an assert(0) inside Ssl::verifySslCertificate when trying to retrieve a generated certificate from cache. This is a Measurement Factory project === modified file 'configure.ac' --- configure.ac 2017-03-31 18:43:20 + +++ configure.ac 2017-05-19 16:17:18 + @@ -1307,42 +1307,54 @@ AC_MSG_ERROR([library 'crypto' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) AC_CHECK_LIB(ssl,[SSL_library_init],[LIBOPENSSL_LIBS="-lssl $LIBOPENSSL_LIBS"],[ AC_MSG_ERROR([library 'ssl' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) ]) # This is a workaround for RedHat 9 brain damage.. if test -d /usr/kerberos/include -a -f /usr/include/openssl/kssl.h; then AC_MSG_NOTICE([OpenSSL depends on Kerberos]) LIBOPENSSL_LIBS="-L/usr/kerberos/lib $LIBOPENSSL_LIBS" CPPFLAGS="$CPPFLAGS -I/usr/kerberos/include" fi SQUID_STATE_ROLLBACK(squid_openssl_state) #de-pollute LIBS if test "x$LIBOPENSSL_LIBS" != "x"; then CXXFLAGS="$LIBOPENSSL_CFLAGS $CXXFLAGS" SSLLIB="$LIBOPENSSL_PATH $LIBOPENSSL_LIBS $SSLLIB" AC_DEFINE(USE_OPENSSL,1,[OpenSSL support is available]) +# check for API functions +SQUID_STATE_SAVE(check_SSL_CTX_get0_certificate) +LIBS="$LIBS $SSLLIB" +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ + AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) + ], [ + missing_SSL_CTX_get0_certificate=yes + ]) +SQUID_STATE_ROLLBACK(check_SSL_CTX_get0_certificate) + # check for other specific broken implementations -SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS +if test "x$missing_SSL_CTX_get0_certificate" = "xyes"; then + SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS +fi SQUID_CHECK_OPENSSL_CONST_SSL_METHOD SQUID_CHECK_OPENSSL_TXTDB SQUID_CHECK_OPENSSL_HELLO_OVERWRITE_HACK fi if test "x$SSLLIB" = "x"; then AC_MSG_ERROR([Required OpenSSL library not found]) fi fi AC_MSG_NOTICE([OpenSSL library support: ${with_openssl:=no} ${LIBOPENSSL_PATH} ${LIBOPENSSL_LIBS}]) AM_CONDITIONAL(ENABLE_SSL,[ test "x$with_openssl" = "xyes" ]) AC_SUBST(SSLLIB) dnl User may specify MIT Kerberos is needed from a non-standard location AC_ARG_WITH(mit-krb5, AS_HELP_STRING([--without-mit-krb5], [Compile without MIT Kerberos support.]), [ case "$with_mit_krb5" in yes|no) : # Nothing special to do here ;; === modified file 'src/ssl/support.cc' --- src/ssl/support.cc 2017-04-29 16:19:15 + +++ src/ssl/support.cc 2017-05-18 17:34:46 + @@ -969,43 +969,45 @@ Security::CertPointer cert; Ssl::EVP_PKEY_Pointer pkey; if (!readCertAndPrivateKeyFromMemory(cert, pkey, data)) return false; if (!cert || !pkey) return false; if (!SSL_use_certificate(ssl, cert.get())) return false; if (!SSL_use_PrivateKey(ssl, pkey.get())) return false;
Re: [squid-dev] OpenSSL 1.1 regression
On 19/05/17 04:04, Christos Tsantilas wrote: On 18/05/2017 03:40 μμ, Amos Jeffries wrote: On 18/05/17 23:12, Christos Tsantilas wrote: +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + This bit seems to be correct. Given the .cc file sequence of macro tests I think we can speed up ./configure a bit by moving the use of SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. eg. AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) ],[ # check for bugs and hacks in the old OpenSSL API SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS ]) I am attaching a new patch. In this patch I moved the SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS as you suggested. But also my last patch was buggy, the AC_CHECK_LIB did not search at the correct directories for libssl library. In this patch I moved the "SQUID_STATE_ROLLBACK(squid_openssl_state)" line some lines down to have the correct libraries search path. Is it ok, or it is better to open a new SQUID_STATE_SAVE/ROLLBACK just for AC_CHECK_LIB? Ah. Either moving the check which alters compiler environment above the existign ROLLBACK, or a new one. It is important the CXXFLAGS and SSLLIB lines directly above where your patch placed it do not get rolled back. PS. Finally, this easy to fix issue, is one more prove that it is better to not start fixing files involved with this satanic tool called autoconf! :-P Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
Hi, Alex Rousskov wrote on Thu, May 18, 2017 at 10:48:15AM -0600: > On 05/18/2017 10:23 AM, Christos Tsantilas wrote: >> However is not easy to always use this method. The OpenSSL-1.1.0 put >> many API changes trying to hide structure members from user and replace >> with API calls instead. Also has many API changes, I suppose to make API >> more consistent. >> >> This is resulted to many changes and it will not be easy to check >> availability for each of the investigated functions and API changes. That is indeed very true. Basically, OpenSSL-1.0.x and OpenSSL-1.1.x are two quite different APIs, and it is impossible to support both in the same application program without either littering the code with lots of #ifdefs (no matter whether testing for version numbers or features) or defining large numbers of private wrappers. Both solutions are hard to maintain, cause a lot of work, and both are problematic from a security perspective because both substantially hinder code auditing. OpenSSL does not offer any kind of migration path from OpenSSL-1.0.x to OpenSSL-1.1.x but dumps the maintenance cost of having to support both APIs in parallel on each and every application software project. That is exactly why - just as an example - OpenSSH decided to not support the OpenSSL-1.1.x API at all, for now. Of course, i don't have the slightest idea whether that would be viable for squid. > Individual feature tests are not panacea. When dealing with a large > across-the-board change like the one you are describing above, [...] FTR, i fully agree that *individual* feature tests are not a good fit for *large-scale* API changes, and well-designed feature group tests can occasionally help. But even with those, supporting two very different APIs in the same program tends to get messy no matter what and will usually obfuscate the code. Yours, Ingo ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 05/18/2017 10:23 AM, Christos Tsantilas wrote: > On 18/05/2017 06:05 μμ, Alex Rousskov wrote: >> I suspect it would be cheaper, in the long term, to use feature tests. > However is not easy to always use this method. The OpenSSL-1.1.0 put > many API changes trying to hide structure members from user and replace > with API calls instead. Also has many API changes, I suppose to make API > more consistent. > > This is resulted to many changes and it will not be easy to check > availability for each of the investigated functions and API changes. Individual feature tests are not panacea. When dealing with a large across-the-board change like the one you are describing above, it is often better to have a single test and a single marker for all affected functions: #if OPENSSL_EXPOSES_MEMBERS ...old code here, possibly with other feature #if tests... #else ...new code here, possibly with other feature #if tests... #endif Also, we probably should wrap all OpenSSL calls that are affected by version variations. It is wrong to sprinkle regular code with numerous #ifs. Yes, that probably means 20+ wrappers, but it is better than the alternatives. Alex. ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/2017 06:05 μμ, Alex Rousskov wrote: On 05/18/2017 05:12 AM, Christos Tsantilas wrote: Agrr... Using the openSSL version was the faster/easier way. Touching autoconf may result to 2-3 full squid rebuilds to implement/test similar fixes. The alternative is to convince others that Squid will not support OpenSSL API implementations that lie about their OpenSSL API version. Judging by the time wasted on related discussions about API basics, I suspect it would be cheaper, in the long term, to use feature tests. Of course I agree with you. In many cases we used openSSL version just because it was easier and this is wrong. However is not easy to always use this method. The OpenSSL-1.1.0 put many API changes trying to hide structure members from user and replace with API calls instead. Also has many API changes, I suppose to make API more consistent. This is resulted to many changes and it will not be easy to check availability for each of the investigated functions and API changes. Alex. ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 05/18/2017 09:34 AM, Ingo Schwarze wrote: > Alex Rousskov wrote on Thu, May 18, 2017 at 09:05:29AM -0600: >> On 05/18/2017 05:12 AM, Christos Tsantilas wrote: >>> Agrr... Using the openSSL version was the faster/easier way. Touching >>> autoconf may result to 2-3 full squid rebuilds to implement/test similar >>> fixes. >> The alternative is to convince others that Squid will not support >> OpenSSL API implementations that lie about their OpenSSL API version. >> Judging by the time wasted on related discussions about API basics, I >> suspect it would be cheaper, in the long term, to use feature tests. > In general, using feature tests is also the cleaner and more > reliable way of dealing with API variations. Yes, of course. If we have to support same-API variations, then feature tests are the right solution. I hope there is consensus around that! Moreover, feature tests are often the right solution even when there are no same-API variations and all features _can_ be reliably detected by API version tests. The difficult question is: Given scarce resources, which is better: 1. spending many hours on supporting OpenSSL API variations or 2. spending those hours on other pressing Squid issues? I do not know the correct answer, but I know that * it is a difficult question without an algorithmic solution; * such questions often create expensive discussions that often end without reaching consensus; and * the Project lacks a mechanism to resolve consensus deadlocks. Alex. ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/2017 03:40 μμ, Amos Jeffries wrote: On 18/05/17 23:12, Christos Tsantilas wrote: +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + This bit seems to be correct. Given the .cc file sequence of macro tests I think we can speed up ./configure a bit by moving the use of SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. eg. AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) ],[ # check for bugs and hacks in the old OpenSSL API SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS ]) I am attaching a new patch. In this patch I moved the SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS as you suggested. But also my last patch was buggy, the AC_CHECK_LIB did not search at the correct directories for libssl library. In this patch I moved the "SQUID_STATE_ROLLBACK(squid_openssl_state)" line some lines down to have the correct libraries search path. Is it ok, or it is better to open a new SQUID_STATE_SAVE/ROLLBACK just for AC_CHECK_LIB? PS. Finally, this easy to fix issue, is one more prove that it is better to not start fixing files involved with this satanic tool called autoconf! Amos Squid crashes when ServerFirst bumping mode is used with openSSL-1.1.0 release When OpenSSL-1.1.0 or later is used: - The SQUID_USE_SSLGETCERTIFICATE_HACK configure test is false - The SQUID_SSLGETCERTIFICATE_BUGGY configure test is true - Squid hits an assert(0) inside Ssl::verifySslCertificate when trying to retrieve a generated certificate from cache. This is a Measurement Factory project === modified file 'configure.ac' --- configure.ac 2017-03-31 18:43:20 + +++ configure.ac 2017-05-18 15:44:32 + @@ -1300,53 +1300,62 @@ # Windows MinGW has some special libraries ... if test "x$squid_host_os" = "xmingw" ; then LIBOPENSSL_LIBS='-lssleay32 -leay32 -lgdi32 $LIBOPENSSL_LIBS' AC_MSG_NOTICE([Windows OpenSSL library support: yes -lssleay32 -leay32 -lgdi32]) fi AC_CHECK_LIB(crypto,[CRYPTO_new_ex_data],[LIBOPENSSL_LIBS="-lcrypto $LIBOPENSSL_LIBS"],[ AC_MSG_ERROR([library 'crypto' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) AC_CHECK_LIB(ssl,[SSL_library_init],[LIBOPENSSL_LIBS="-lssl $LIBOPENSSL_LIBS"],[ AC_MSG_ERROR([library 'ssl' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) ]) # This is a workaround for RedHat 9 brain damage.. if test -d /usr/kerberos/include -a -f /usr/include/openssl/kssl.h; then AC_MSG_NOTICE([OpenSSL depends on Kerberos]) LIBOPENSSL_LIBS="-L/usr/kerberos/lib $LIBOPENSSL_LIBS" CPPFLAGS="$CPPFLAGS -I/usr/kerberos/include" fi - SQUID_STATE_ROLLBACK(squid_openssl_state) #de-pollute LIBS if test "x$LIBOPENSSL_LIBS" != "x"; then CXXFLAGS="$LIBOPENSSL_CFLAGS $CXXFLAGS" SSLLIB="$LIBOPENSSL_PATH $LIBOPENSSL_LIBS $SSLLIB" AC_DEFINE(USE_OPENSSL,1,[OpenSSL support is available]) +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ + AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) + ], [ + # check for bugs and hacks in the old OpenSSL API + SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS + ]) + # check for other specific broken implementations -SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS SQUID_CHECK_OPENSSL_CONST_SSL_METHOD SQUID_CHECK_OPENSSL_TXTDB SQUID_CHECK_OPENSSL_HELLO_OVERWRITE_HACK fi + + SQUID_STATE_ROLLBACK(squid_openssl_state) #de-pollute LIBS + if test "x$SSLLIB" = "x"; then AC_MSG_ERROR([Required OpenSSL library not found]) fi fi AC_MSG_NOTICE([OpenSSL library support: ${with_openssl:=no} ${LIBOPENSSL_PATH} ${LIBOPENSSL_LIBS}]) AM_CONDITIONAL(ENABLE_SSL,[ test "x$with_openssl" = "xyes" ]) AC_SUBST(SSLLIB) dnl User may specify MIT Kerberos is needed from a non-standard location AC_ARG_WITH(mit-krb5, AS_HELP_STRING([--without-mit-krb5], [Compile without MIT Kerberos support.]), [ case "$with_mit_krb5" in yes|no) : # Nothing special to do here ;; *) if test ! -d "$withval" ; then AC_MSG_ERROR([--with-mit-krb5 path does not point to a directory]) fi === modified file 'src/ssl/support.cc' --- src/ssl/support.cc 2017-04-29 16:19:15 + +++ src/ssl/support.cc 2017-05-18 09:15:54 + @@ -969,43 +969,45 @@ Security::CertPointer cert; Ssl::EVP_PKEY_Pointer pkey; if (!readCertAndPrivateKeyFromMemory(cert, pkey, data)) return false; if (!cert || !pkey) return false; if (!SSL_use_certificate(ssl, cert.get())) return false; if (!SSL_use_PrivateKey(ssl, pkey.get())) return false; return true; } bool Ssl::verifySslCertificate(Security::ContextPointer &ctx, CertificateProperties const &properties)
Re: [squid-dev] OpenSSL 1.1 regression
Hi, Alex Rousskov wrote on Thu, May 18, 2017 at 09:05:29AM -0600: > On 05/18/2017 05:12 AM, Christos Tsantilas wrote: >> Agrr... Using the openSSL version was the faster/easier way. Touching >> autoconf may result to 2-3 full squid rebuilds to implement/test similar >> fixes. > The alternative is to convince others that Squid will not support > OpenSSL API implementations that lie about their OpenSSL API version. > Judging by the time wasted on related discussions about API basics, I > suspect it would be cheaper, in the long term, to use feature tests. In general, using feature tests is also the cleaner and more reliable way of dealing with API variations. By definition, inspecting library version numbers only works - for one specific implementation: If somebody else does an alternative implementation, they have the choice of (1) not defining the foreign version number at all - which will result in catastrophic breakage in application software inspecting it - or (2) defining a very low number - which will result in such software not using available features - or (3) defining a very high number - which will result in such software to try and use unavailable features - or (4) slavishly follow the API development path of the older implementation, which defeats part of the purpose of an alternative implementation, because that precludes all of the following: earlier addition of good interfaces that the older implementation only added later; removal of dangerous interfaces that you don't want to support; postponing the addition of new interfaces added in the older implementation that you don't have the time yet to implement properly, or that you consider less important. - as long as that implementation does not change the API again: If they do change the API back later, you have to touch all your application code again and adjust all related version number checks. So while checking version numbers may sometimes look like the easiest and quickest way to get an issue out of the way, it is not very sustainable in the long run, and avoiding it generally results in better and more portable software, and in safer and ever easier maintenance. Thanks for caring! Ingo P.S. For me, it's now "back to lurking"... ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 05/18/2017 05:12 AM, Christos Tsantilas wrote: > Agrr... Using the openSSL version was the faster/easier way. Touching > autoconf may result to 2-3 full squid rebuilds to implement/test similar > fixes. The alternative is to convince others that Squid will not support OpenSSL API implementations that lie about their OpenSSL API version. Judging by the time wasted on related discussions about API basics, I suspect it would be cheaper, in the long term, to use feature tests. Alex. ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/2017 03:12 μμ, Amos Jeffries wrote: On 18/05/17 23:12, Christos Tsantilas wrote: On 17/05/2017 07:56 μμ, Alex Rousskov wrote: On 05/17/2017 10:35 AM, Christos Tsantilas wrote: +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L) +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); If it is possible to replace this version check with a ./configure-time detection of SSL_CTX_get0_certificate() availability, please do that. Avoiding OPENSSL_VERSION_NUMBER macros in new code may help with future support for LibreSSL and/or other libraries that lie about OpenSSL API version they provide. For the t2 patch I am using the AC_CHECK_LIB autoconf macro to check for the function availability. http://bugs.squid-cache.org/show_bug.cgi?id=4662 Agrr... Using the openSSL version was the faster/easier way. Touching autoconf may result to 2-3 full squid rebuilds to implement/test similar fixes. The autoconf detection part can be designed and tested with just bootstrap.sh and ./configure execution - then check what got set in includes/autoconf.h and config.log. No need for the "make" parts of building. Well. Of course this is what someone must do. But you have always to build and test, to avoid typo errors for example or other simple mistakes. And after you are finished with this you are finding that it would be better to move a check into the if-not-found [] path to speed-up configure script! Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/17 23:12, Christos Tsantilas wrote: +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + This bit seems to be correct. Given the .cc file sequence of macro tests I think we can speed up ./configure a bit by moving the use of SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS into the if-not-found [] path. eg. AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [ AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available]) ],[ # check for bugs and hacks in the old OpenSSL API SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS ]) Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/17 23:12, Christos Tsantilas wrote: On 17/05/2017 07:56 μμ, Alex Rousskov wrote: On 05/17/2017 10:35 AM, Christos Tsantilas wrote: +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L) +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); If it is possible to replace this version check with a ./configure-time detection of SSL_CTX_get0_certificate() availability, please do that. Avoiding OPENSSL_VERSION_NUMBER macros in new code may help with future support for LibreSSL and/or other libraries that lie about OpenSSL API version they provide. For the t2 patch I am using the AC_CHECK_LIB autoconf macro to check for the function availability. http://bugs.squid-cache.org/show_bug.cgi?id=4662 Agrr... Using the openSSL version was the faster/easier way. Touching autoconf may result to 2-3 full squid rebuilds to implement/test similar fixes. The autoconf detection part can be designed and tested with just bootstrap.sh and ./configure execution - then check what got set in includes/autoconf.h and config.log. No need for the "make" parts of building. Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/2017 01:54 μμ, Amos Jeffries wrote: Sorry for wasting time. :-( there is not wasting time, because squid with server-first bumping mode crashes when openssl-1.1 is used. So we have to fix it. Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 17/05/2017 07:56 μμ, Alex Rousskov wrote: On 05/17/2017 10:35 AM, Christos Tsantilas wrote: +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L) +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); If it is possible to replace this version check with a ./configure-time detection of SSL_CTX_get0_certificate() availability, please do that. Avoiding OPENSSL_VERSION_NUMBER macros in new code may help with future support for LibreSSL and/or other libraries that lie about OpenSSL API version they provide. For the t2 patch I am using the AC_CHECK_LIB autoconf macro to check for the function availability. http://bugs.squid-cache.org/show_bug.cgi?id=4662 Agrr... Using the openSSL version was the faster/easier way. Touching autoconf may result to 2-3 full squid rebuilds to implement/test similar fixes. Thank you, Alex. Squid crashes when ServerFirst bumping mode is used with openSSL-1.1.0 release When OpenSSL-1.1.0 or later is used: - The SQUID_USE_SSLGETCERTIFICATE_HACK configure test is false - The SQUID_SSLGETCERTIFICATE_BUGGY configure test is true - Squid hits an assert(0) inside Ssl::verifySslCertificate when trying to retrieve a generated certificate from cache. This is a Measurement Factory project === modified file 'configure.ac' --- configure.ac 2017-03-31 18:43:20 + +++ configure.ac 2017-05-18 09:13:03 + @@ -1307,40 +1307,43 @@ AC_MSG_ERROR([library 'crypto' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) AC_CHECK_LIB(ssl,[SSL_library_init],[LIBOPENSSL_LIBS="-lssl $LIBOPENSSL_LIBS"],[ AC_MSG_ERROR([library 'ssl' is required for OpenSSL]) ],$LIBOPENSSL_LIBS) ]) # This is a workaround for RedHat 9 brain damage.. if test -d /usr/kerberos/include -a -f /usr/include/openssl/kssl.h; then AC_MSG_NOTICE([OpenSSL depends on Kerberos]) LIBOPENSSL_LIBS="-L/usr/kerberos/lib $LIBOPENSSL_LIBS" CPPFLAGS="$CPPFLAGS -I/usr/kerberos/include" fi SQUID_STATE_ROLLBACK(squid_openssl_state) #de-pollute LIBS if test "x$LIBOPENSSL_LIBS" != "x"; then CXXFLAGS="$LIBOPENSSL_CFLAGS $CXXFLAGS" SSLLIB="$LIBOPENSSL_PATH $LIBOPENSSL_LIBS $SSLLIB" AC_DEFINE(USE_OPENSSL,1,[OpenSSL support is available]) +# check for API functions +AC_CHECK_LIB(ssl, SSL_CTX_get0_certificate, [AC_DEFINE(HAVE_SSL_CTX_GET0_CERTIFICATE, 1, [SSL_CTX_get0_certificate is available])], []) + # check for other specific broken implementations SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS SQUID_CHECK_OPENSSL_CONST_SSL_METHOD SQUID_CHECK_OPENSSL_TXTDB SQUID_CHECK_OPENSSL_HELLO_OVERWRITE_HACK fi if test "x$SSLLIB" = "x"; then AC_MSG_ERROR([Required OpenSSL library not found]) fi fi AC_MSG_NOTICE([OpenSSL library support: ${with_openssl:=no} ${LIBOPENSSL_PATH} ${LIBOPENSSL_LIBS}]) AM_CONDITIONAL(ENABLE_SSL,[ test "x$with_openssl" = "xyes" ]) AC_SUBST(SSLLIB) dnl User may specify MIT Kerberos is needed from a non-standard location AC_ARG_WITH(mit-krb5, AS_HELP_STRING([--without-mit-krb5], [Compile without MIT Kerberos support.]), [ case "$with_mit_krb5" in yes|no) === modified file 'src/ssl/support.cc' --- src/ssl/support.cc 2017-04-29 16:19:15 + +++ src/ssl/support.cc 2017-05-18 09:15:54 + @@ -969,43 +969,45 @@ Security::CertPointer cert; Ssl::EVP_PKEY_Pointer pkey; if (!readCertAndPrivateKeyFromMemory(cert, pkey, data)) return false; if (!cert || !pkey) return false; if (!SSL_use_certificate(ssl, cert.get())) return false; if (!SSL_use_PrivateKey(ssl, pkey.get())) return false; return true; } bool Ssl::verifySslCertificate(Security::ContextPointer &ctx, CertificateProperties const &properties) { +#if HAVE_SSL_CTX_GET0_CERTIFICATE +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); +#elif SQUID_USE_SSLGETCERTIFICATE_HACK // SSL_get_certificate is buggy in openssl versions 1.0.1d and 1.0.1e // Try to retrieve certificate directly from Security::ContextPointer object -#if SQUID_USE_SSLGETCERTIFICATE_HACK X509 ***pCert = (X509 ***)ctx->cert; X509 * cert = pCert && *pCert ? **pCert : NULL; #elif SQUID_SSLGETCERTIFICATE_BUGGY X509 * cert = NULL; assert(0); #else // Temporary ssl for getting X509 certificate from SSL_CTX. Security::SessionPointer ssl(Security::NewSessionObject(ctx)); X509 * cert = SSL_get_certificate(ssl.get()); #endif if (!cert) return false; ASN1_TIME * time_notBefore = X509_get_notBefore(cert); ASN1_TIME * time_notAfter = X509_get_notAfter(cert); bool ret = (X509_cmp_current_time(time_notBefore) < 0 && X509_cmp_current_time(time_notAfter) > 0); if (!ret) return false; return certificateMatchesProperties(cert, properties); } ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 18/05/17 04:35, Christos Tsantilas wrote: On 16/05/2017 03:04 μμ, Amos Jeffries wrote: Building Squid-5 r15136 against the latest libssl 1.1.0e on Ubuntu. src/ssl/support.cc: In function ‘bool Ssl::verifySslCertificate(Security::ContextPointer&, const Ssl::CertificateProperties&)’: src/ssl/support.cc:995:34: error: invalid use of incomplete type ‘struct ssl_ctx_st’ X509 ***pCert = (X509 ***)ctx->cert; I am not getting this compile error when I am trying to use openSSL-1.1.0, but I am getting a crash when squid is running and uses server-first bumping mode. The crash is caused because the SQUID_USE_SSLGETCERTIFICATE_HACK is false and SQUID_SSLGETCERTIFICATE_BUGGY is true. GCC-6 went through another update for me today, and after re-bootstrapping the problem is gone. So I'm now thinking this may have been a fluke or timing mixup in my library juggling act between v5/v4 and v3.5 builds. I am attaching a patch which fixes this bug for squid-5. Should I just update this hack code to use the X509_STORE_CTX_get0_cert() getter ? or is this a sign of a deeper bug with the SQUID_USE_SSLGETCERTIFICATE_HACK autoconf test that needs to be fixed? In my tests no, there is not need to be fixed. Are you using an unmodified squid? Latest bzr checkout of Squid. But OpenSSL for me is ... well PITA is an understatement when it comes to Squid-3.5. I am beginning to think it was still setup for 3.5 when I built that v5. I will see if it happens again and reevaluate the patch then. Sorry for wasting time. :-( Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 05/17/2017 10:35 AM, Christos Tsantilas wrote: > +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L) > +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); If it is possible to replace this version check with a ./configure-time detection of SSL_CTX_get0_certificate() availability, please do that. Avoiding OPENSSL_VERSION_NUMBER macros in new code may help with future support for LibreSSL and/or other libraries that lie about OpenSSL API version they provide. http://bugs.squid-cache.org/show_bug.cgi?id=4662 Thank you, Alex. ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 16/05/2017 03:04 μμ, Amos Jeffries wrote: Building Squid-5 r15136 against the latest libssl 1.1.0e on Ubuntu. src/ssl/support.cc: In function ‘bool Ssl::verifySslCertificate(Security::ContextPointer&, const Ssl::CertificateProperties&)’: src/ssl/support.cc:995:34: error: invalid use of incomplete type ‘struct ssl_ctx_st’ X509 ***pCert = (X509 ***)ctx->cert; I am not getting this compile error when I am trying to use openSSL-1.1.0, but I am getting a crash when squid is running and uses server-first bumping mode. The crash is caused because the SQUID_USE_SSLGETCERTIFICATE_HACK is false and SQUID_SSLGETCERTIFICATE_BUGGY is true. I am attaching a patch which fixes this bug for squid-5. Should I just update this hack code to use the X509_STORE_CTX_get0_cert() getter ? or is this a sign of a deeper bug with the SQUID_USE_SSLGETCERTIFICATE_HACK autoconf test that needs to be fixed? In my tests no, there is not need to be fixed. Are you using an unmodified squid? Amos Squid crashes when ServerFirst bumping mode is used with openSSL-1.1.0 release When OpenSSL-1.1.0 or later is used: - The SQUID_USE_SSLGETCERTIFICATE_HACK configure test is false - The SQUID_SSLGETCERTIFICATE_BUGGY configure test is true - Squid hits an assert(0) inside Ssl::verifySslCertificate when trying to retrieve a generated certificate from cache. This is a Measurement Factory project === modified file 'src/ssl/support.cc' --- src/ssl/support.cc 2017-04-29 16:19:15 + +++ src/ssl/support.cc 2017-05-17 16:26:34 + @@ -969,43 +969,45 @@ Security::CertPointer cert; Ssl::EVP_PKEY_Pointer pkey; if (!readCertAndPrivateKeyFromMemory(cert, pkey, data)) return false; if (!cert || !pkey) return false; if (!SSL_use_certificate(ssl, cert.get())) return false; if (!SSL_use_PrivateKey(ssl, pkey.get())) return false; return true; } bool Ssl::verifySslCertificate(Security::ContextPointer &ctx, CertificateProperties const &properties) { +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L) +X509 * cert = SSL_CTX_get0_certificate(ctx.get()); +#elif SQUID_USE_SSLGETCERTIFICATE_HACK // SSL_get_certificate is buggy in openssl versions 1.0.1d and 1.0.1e // Try to retrieve certificate directly from Security::ContextPointer object -#if SQUID_USE_SSLGETCERTIFICATE_HACK X509 ***pCert = (X509 ***)ctx->cert; X509 * cert = pCert && *pCert ? **pCert : NULL; #elif SQUID_SSLGETCERTIFICATE_BUGGY X509 * cert = NULL; assert(0); #else // Temporary ssl for getting X509 certificate from SSL_CTX. Security::SessionPointer ssl(Security::NewSessionObject(ctx)); X509 * cert = SSL_get_certificate(ssl.get()); #endif if (!cert) return false; ASN1_TIME * time_notBefore = X509_get_notBefore(cert); ASN1_TIME * time_notAfter = X509_get_notAfter(cert); bool ret = (X509_cmp_current_time(time_notBefore) < 0 && X509_cmp_current_time(time_notAfter) > 0); if (!ret) return false; return certificateMatchesProperties(cert, properties); } ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] OpenSSL 1.1 regression
On 16/05/2017 03:04 μμ, Amos Jeffries wrote: Building Squid-5 r15136 against the latest libssl 1.1.0e on Ubuntu. src/ssl/support.cc: In function ‘bool Ssl::verifySslCertificate(Security::ContextPointer&, const Ssl::CertificateProperties&)’: src/ssl/support.cc:995:34: error: invalid use of incomplete type ‘struct ssl_ctx_st’ X509 ***pCert = (X509 ***)ctx->cert; Should I just update this hack code to use the X509_STORE_CTX_get0_cert() getter ? No we can not use this function here. But we can use the SSL_CTX_get0_certificate. But this is added after openssl-1.0.2 releases. or is this a sign of a deeper bug with the SQUID_USE_SSLGETCERTIFICATE_HACK autoconf test that needs to be fixed? Looks that SQUID_USE_SSLGETCERTIFICATE_HACK autoconf test does not work well. The workaround used when the SQUID_USE_SSLGETCERTIFICATE_HACK macro is false, which uses a termporary SSL object should work also. Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
[squid-dev] OpenSSL 1.1 regression
Building Squid-5 r15136 against the latest libssl 1.1.0e on Ubuntu. src/ssl/support.cc: In function ‘bool Ssl::verifySslCertificate(Security::ContextPointer&, const Ssl::CertificateProperties&)’: src/ssl/support.cc:995:34: error: invalid use of incomplete type ‘struct ssl_ctx_st’ X509 ***pCert = (X509 ***)ctx->cert; Should I just update this hack code to use the X509_STORE_CTX_get0_cert() getter ? or is this a sign of a deeper bug with the SQUID_USE_SSLGETCERTIFICATE_HACK autoconf test that needs to be fixed? Amos ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev