Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 19.08.2015, at 21:40, Viktor Dukhovni postfix-us...@dukhovni.org wrote: I've figured out what's going on. LibreSSL 2.2.2 appears to have disabled support for the SSLv2-compatible client HELLO. Servers that have not disabled SSLv2 are unable to complete an SSLv2-compatible TLS handshake with LibreSSL 2.2.2. Connections that use an SSLv2 hello fail. Also clients that use just SSLv3 (no extensions, ...) fail. JFTR: We have released LibreSSL 2.2.3, which will be arriving in the LibreSSL directory of your local OpenBSD mirror soon. This release is based on the stable OpenBSD 5.8 branch, fixing a bug that affects interoperability with some SSL clients. * LibreSSL 2.2.2 incorrectly handles ClientHello messages that do not include TLS extensions, resulting in such handshakes being aborted. This release corrects the handling of such messages. Thanks to Ligushka from github for reporting the issue. (see http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.2.3-relnotes.txt) I did test a pre-release patch and didn't see my reported issues with LibreSSL 2.2.2 any longer. Regards, Michael
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Wed, Aug 19, 2015 at 12:58:38PM -0700, Alice Wonder wrote: ``You also turn on thousands and thousands of lines of OpenSSL library code. Assuming that OpenSSL is written as carefully as Wietse's own code, every 1000 lines introduce one additional bug into Postfix.'' We now know OpenSSL has not been written as carefully as Postfix. LibreSSL removed a lot of needless code and has cleaned up a lot of what was left. Yes, but LibreSSL is just a fork, with mostly the same real issues. Real work is happening upstream to improve the internals, not just remove non-mainstream features. I don't see a compelling reason to use LibreSSL if you're not on OpenBSD. I see successful marketing with not much substance underneath. If they really wanted to make a difference, they'd send patches, not fork the project. I've seen very little by way of upstream contributions. -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 08/22/2015 06:08 AM, Viktor Dukhovni wrote: On Wed, Aug 19, 2015 at 12:58:38PM -0700, Alice Wonder wrote: ``You also turn on thousands and thousands of lines of OpenSSL library code. Assuming that OpenSSL is written as carefully as Wietse's own code, every 1000 lines introduce one additional bug into Postfix.'' We now know OpenSSL has not been written as carefully as Postfix. LibreSSL removed a lot of needless code and has cleaned up a lot of what was left. Yes, but LibreSSL is just a fork, with mostly the same real issues. Real work is happening upstream to improve the internals, not just remove non-mainstream features. I don't see a compelling reason to use LibreSSL if you're not on OpenBSD. I see successful marketing with not much substance underneath. If they really wanted to make a difference, they'd send patches, not fork the project. I've seen very little by way of upstream contributions. One of the reasons they forked is because there were issue WITH PATCHES in the OpenSSL bug database that were not addressed for several years.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Sat, Aug 22, 2015 at 07:37:47AM -0700, Alice Wonder wrote: If they really wanted to make a difference, they'd send patches, not fork the project. I've seen very little by way of upstream contributions. One of the reasons they forked is because there were issue WITH PATCHES in the OpenSSL bug database that were not addressed for several years. That was then, things are different now. We'll see how 1.1.0 is received. Though there'll still be lots of work to do for a while. Google's BoringSSL is also a fork, but they're also contributing to OpenSSL. Anyway, bottom-line is that for now LibreSSL is too bleeding edge for use with SMTP (and in particular Postfix). Speaking of OpenSSL 1.1.0, that'll come out early next year. In that version: *) SSLv2 support has been removed. It still supports receiving a SSLv2 compatible client hello. [Kurt Roeckx] Which solves the problem in a more compatible way. -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 19.08.2015, at 18:58, Viktor Dukhovni postfix-us...@dukhovni.org wrote: On Wed, Aug 19, 2015 at 06:30:43PM +0200, Michael Grimm wrote: This is postfix 3.0.2 and FreeBSD-10.2/STABLE. I switched from OpenSLL to LibreSSL some month ago. LibreSSL is not tested with Postfix, and so not officially supported. Understood. I will revert to OpenSLL, then. But see below. My relevant SSL/TLS settings for receiving mail didn't change ever since that time (postconf -n | grep tls | grep smtpd) smtpd_use_tls = yes Obsolete. Thanks and removed. Previous LibreSSL 2.2.1: *all* those servers delivered their mail as reported by logwatch; example: 16 Anonymous: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) 1 1.2.3.4xxx.xxx Well, LibreSSL 2.2.2 must have broken something. If you want more help, you'll need to disclose the IP address of your server. The servers in question must be doing something more exotic than you report (or I am testing the wrong server): $ posttls-finger -c -p TLSv1 -lsecure -Lsummary \ -o tls_medium_cipherlist=DHE-RSA-AES256-SHA \ odo.in-berlin.de mx1.enfer-du-nord.net[87.98.149.189]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) Yes, this is my receiving mailserver. One of the servers in question is one of the servers sending mail for this ML: Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: connect from russian-caravan.cloud9.net[2604:8d00:0:1::4] Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: SSL_accept error from russian-caravan.cloud9.net[2604:8d00:0:1::4]: lost connection Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: lost connection after STARTTLS from russian-caravan.cloud9.net[2604:8d00:0:1::4] Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: disconnect from russian-caravan.cloud9.net[2604:8d00:0:1::4] ehlo=1 starttls=0/1 commands=1/2 (JFTR: Those servers in question use IPv4 and IPv6) [Very informative information about SSL 3.0 ciphers removed. Thanks for that.] Sigh, I do have to admit that crypto configuration isn't well understood by myself, thus I feel lost here. But every hint is highly appreciated. Postfix default settings strive to free users of the burden of becoming experts at cryptography. Use largely default settings, or overrides recommended as sensible alternatives in the documentation. Plus the settings in my recent post on best practice TLS configuration. [quoting re-orderd] smtpd_tls_auth_only = yes smtpd_tls_security_level = may smtpd_tls_loglevel = 1 smtpd_tls_cert_file = /path-to-pem/my-server.pem smtpd_tls_key_file = /path-to-pem/my-server.pem smtpd_tls_security_level = may smtpd_tls_protocols = !SSLv2 !SSLv3 smtpd_tls_ciphers = medium smtpd_tls_mandatory_protocols = !SSLv2 !SSLv3 smtpd_tls_mandatory_ciphers = high smtpd_tls_dh1024_param_file = /path-to-pem/dh-2048.pem smtpd_tls_dh512_param_file = /path-to-pem/dh-512.pem Looks good. I always tried to stick to the default. Thus, are my settings reported too far off default? I will revert back to OpenSLL. If you won't to investigate LibreSSL's behavior with regard to russian-caravan.cloud9.net any further, I am willing to keep my secondary mx to LibreSSL for the time being. If not, please let me know. Might have been too early for that switch to LibreSSL ... Thanks for your input and with kind regards, Michael
SSL_accept errors after recent upgrade to LibreSSL 2.2.2
Hi — This is postfix 3.0.2 and FreeBSD-10.2/STABLE. I switched from OpenSLL to LibreSSL some month ago. My relevant SSL/TLS settings for receiving mail didn't change ever since that time (postconf -n | grep tls | grep smtpd) smtpd_use_tls = yes smtpd_tls_auth_only = yes smtpd_tls_security_level = may smtpd_tls_loglevel = 1 smtpd_tls_cert_file = /path-to-pem/my-server.pem smtpd_tls_key_file = /path-to-pem/my-server.pem smtpd_tls_security_level = may smtpd_tls_protocols = !SSLv2 !SSLv3 smtpd_tls_ciphers = medium smtpd_tls_mandatory_protocols = !SSLv2 !SSLv3 smtpd_tls_mandatory_ciphers = high smtpd_tls_dh1024_param_file = /path-to-pem/dh-2048.pem smtpd_tls_dh512_param_file = /path-to-pem/dh-512.pem After my recent upgrade of LibreSSL to 2.2.2 some servers fail to deliver mail. Example logfile entry: postfix/smtpd[111]: connect from xxx.xxx[1.2.3.4] postfix/smtpd[111]: SSL_accept error from xxx.xxx[1.2.3.4]: lost connection postfix/smtpd[111]: lost connection after STARTTLS from xxx.xxx[1.2.3.4]: postfix/smtpd[111]: disconnect from xxx.xxx[1.2.3.4]: ehlo=1 starttls=0/1 commands=1/2 Previous LibreSSL 2.2.1: *all* those servers delivered their mail as reported by logwatch; example: 16 Anonymous: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) 1 1.2.3.4xxx.xxx For the time being I am helping myself by discarding TLS for those servers by setting 1.2.3.4 STARTTLS in: smtpd_discard_ehlo_keyword_address_maps = cidr:/path-to-conf/smtpd_discard_ehlo_keyword_address_maps But, I do consider this approach somehow error prone. I could revert either to the previous LibreSSL version or back to OpenSSL, but I really would like to understand whether I do have an erroneous configuration of postfix, or if I am missing something else. In the release notes of LibreSSL 2.2.2 (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.2.2-relnotes.txt) I do find: * Removed SSLv3 support from openssl(1) But I do find SSLv3 protocol entries: mail openssl version LibreSSL 2.2.2 mail openssl ciphers -v | grep ^DHE-RSA-AES256-SHA DHE-RSA-AES256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256 DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1 Sigh, I do have to admit that crypto configuration isn't well understood by myself, thus I feel lost here. But every hint is highly appreciated. (BTW: is this off-topic for that list? If so, tell me then. I will move to a recommended ML.) With kind regards and thanks in advance, Michael
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Wed, Aug 19, 2015 at 06:30:43PM +0200, Michael Grimm wrote: This is postfix 3.0.2 and FreeBSD-10.2/STABLE. I switched from OpenSLL to LibreSSL some month ago. LibreSSL is not tested with Postfix, and so not officially supported. My relevant SSL/TLS settings for receiving mail didn't change ever since that time (postconf -n | grep tls | grep smtpd) smtpd_use_tls = yes Obsolete. smtpd_tls_auth_only = yes smtpd_tls_security_level = may smtpd_tls_loglevel = 1 smtpd_tls_cert_file = /path-to-pem/my-server.pem smtpd_tls_key_file = /path-to-pem/my-server.pem smtpd_tls_security_level = may smtpd_tls_protocols = !SSLv2 !SSLv3 smtpd_tls_ciphers = medium smtpd_tls_mandatory_protocols = !SSLv2 !SSLv3 smtpd_tls_mandatory_ciphers = high smtpd_tls_dh1024_param_file = /path-to-pem/dh-2048.pem smtpd_tls_dh512_param_file = /path-to-pem/dh-512.pem Looks good. After my recent upgrade of LibreSSL to 2.2.2 some servers fail to deliver mail. Check the LibreSSL release notes. Previous LibreSSL 2.2.1: *all* those servers delivered their mail as reported by logwatch; example: 16 Anonymous: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) 1 1.2.3.4xxx.xxx Well, LibreSSL 2.2.2 must have broken something. If you want more help, you'll need to disclose the IP address of your server. The servers in question must be doing something more exotic than you report (or I am testing the wrong server): $ posttls-finger -c -p TLSv1 -lsecure -Lsummary \ -o tls_medium_cipherlist=DHE-RSA-AES256-SHA \ odo.in-berlin.de mx1.enfer-du-nord.net[87.98.149.189]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) In the release notes of LibreSSL 2.2.2 (http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.2.2-relnotes.txt) I do find: * Removed SSLv3 support from openssl(1) But the previous connections where TLS 1.0, not SSL 3.0. And they did not remove the SSL 3.0 ciphers, that are needed for TLS 1.0 support. But I do find SSLv3 protocol entries: mail openssl version LibreSSL 2.2.2 mail openssl ciphers -v | grep ^DHE-RSA-AES256-SHA DHE-RSA-AES256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256 DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1 That's not SSL 3.0 protocol support. That's a cipher suite that was introduced with SSL 3.0 and is also used for TLS 1.0 through TLS 1.2. Sigh, I do have to admit that crypto configuration isn't well understood by myself, thus I feel lost here. But every hint is highly appreciated. Postfix default settings strive to free users of the burden of becoming experts at cryptography. Use largely default settings, or overrides recommended as sensible alternatives in the documentation. Plus the settings in my recent post on best practice TLS configuration. (BTW: is this off-topic for that list? If so, tell me then. I will move to a recommended ML.) No, this is on topic. -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 19.08.2015, at 20:02, Viktor Dukhovni postfix-us...@dukhovni.org wrote: On Wed, Aug 19, 2015 at 07:49:42PM +0200, Michael Grimm wrote: One of the servers in question is one of the servers sending mail for this ML: Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: connect from russian-caravan.cloud9.net[2604:8d00:0:1::4] Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: SSL_accept error from russian-caravan.cloud9.net[2604:8d00:0:1::4]: lost connection Works for me via IPv6 too: $ posttls-finger -o inet_protocols=ipv6 -c -p TLSv1 -lmay -Lsummary \ -o tls_medium_cipherlist=DHE-RSA-AES256-SHA \ odo.in-berlin.de posttls-finger: Untrusted TLS connection established to mx1.enfer-du-nord.net[2001:41d0:8:67d4:1:1:0:1]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) Just in case I couldn't make it clear: All sending servers from cloud9.net deliver mail via IPv4 or IPv6 to my servers without any issue and without entries in smtpd_discard_ehlo_keyword_address_maps. The *only exception* is russian-caravan.cloud9.net[2604:8d00:0:1::4]. To debug further, we'd need a tcpdump full packet capture: http://www.postfix.org/DEBUG_README.html#sniffer […] I will revert back to OpenSLL. If you won't to investigate LibreSSL's behavior with regard to russian-caravan.cloud9.net any further, I am willing to keep my secondary mx to LibreSSL for the time being. If not, please let me know. Might have been too early for that switch to LibreSSL I would not go out of my way to switch to LibreSSL at this time. Use it if you're using OpenBSD, but stick with OpenSSL for now on other platforms. Understood. That said, it might be helpful to others to find out what interoperability problem was introduced by LibreSSL 2.2.2. So get a packet capture or two before reverting to OpenSSL. I will revert to OpenSSL my primary mx, first. Then I will come back to this issue and provide you with tcpdump debugging info. Might take some days, though. Should I send them off-list or on-list? Thanks again and with kind regards, Michael
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Wed, Aug 19, 2015 at 07:49:42PM +0200, Michael Grimm wrote: mx1.enfer-du-nord.net[87.98.149.189]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) Yes, this is my receiving mailserver. One of the servers in question is one of the servers sending mail for this ML: Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: connect from russian-caravan.cloud9.net[2604:8d00:0:1::4] Aug 19 19:08:29 mail.info mail postfix/smtpd[94303]: SSL_accept error from russian-caravan.cloud9.net[2604:8d00:0:1::4]: lost connection Works for me via IPv6 too: $ posttls-finger -o inet_protocols=ipv6 -c -p TLSv1 -lmay -Lsummary \ -o tls_medium_cipherlist=DHE-RSA-AES256-SHA \ odo.in-berlin.de posttls-finger: Untrusted TLS connection established to mx1.enfer-du-nord.net[2001:41d0:8:67d4:1:1:0:1]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) To debug further, we'd need a tcpdump full packet capture: http://www.postfix.org/DEBUG_README.html#sniffer (replace example.com with a suitable name or address). smtpd_tls_auth_only = yes smtpd_tls_security_level = may smtpd_tls_loglevel = 1 smtpd_tls_cert_file = /path-to-pem/my-server.pem smtpd_tls_key_file = /path-to-pem/my-server.pem smtpd_tls_security_level = may smtpd_tls_protocols = !SSLv2 !SSLv3 smtpd_tls_ciphers = medium smtpd_tls_mandatory_protocols = !SSLv2 !SSLv3 smtpd_tls_mandatory_ciphers = high smtpd_tls_dh1024_param_file = /path-to-pem/dh-2048.pem smtpd_tls_dh512_param_file = /path-to-pem/dh-512.pem Looks good. I always tried to stick to the default. Thus, are my settings reported too far off default? No, they're just right. I will revert back to OpenSLL. If you won't to investigate LibreSSL's behavior with regard to russian-caravan.cloud9.net any further, I am willing to keep my secondary mx to LibreSSL for the time being. If not, please let me know. Might have been too early for that switch to LibreSSL I would not go out of my way to switch to LibreSSL at this time. Use it if you're using OpenBSD, but stick with OpenSSL for now on other platforms. That said, it might be helpful to others to find out what interoperability problem was introduced by LibreSSL 2.2.2. So get a packet capture or two before reverting to OpenSSL. -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Wed, Aug 19, 2015 at 09:11:16PM +0200, Michael Grimm wrote: On 19.08.2015, at 20:21, Michael Grimm trash...@odo.in-berlin.de wrote: I will revert to OpenSSL my primary mx, first. Done. BTW: LibreSSL 2.2.2 broke unbound 1.5.4 as well. Then I will come back to this issue and provide you with tcpdump debugging info. Now, my secondary is postfix/LibrSSL, only. I've figured out what's going on. LibreSSL 2.2.2 appears to have disabled support for the SSLv2-compatible client HELLO. Servers that have not disabled SSLv2 are unable to complete an SSLv2-compatible TLS handshake with LibreSSL 2.2.2. Connections that use an SSLv2 hello fail. Also clients that use just SSLv3 (no extensions, ...) fail. -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On Wed, Aug 19, 2015 at 09:54:01PM +0200, Michael Grimm wrote: If I do understand that correctly, it has been a good advice to revert back to OpenSSL running OS != OpenBSD. I stand by that advice. And, if I am not mistaken, there is no way to tell postfix to work around that disabled support for HELLO. Correct? Unless there's some new flag to SSL_CTX_set_options() that re-enables SSL2-compatible HELLO support. You can check the documentation for any hint of such a mechanism. If you are interested in tcpdumps of connections from russian-caravan.cloud9.net, please let me know. I do have one dump at hand, already. I've managed to reproduce failing connections to your (backup MX) machine with: openssl s_client -starttls smtp -connect host:25 and succeed with: openssl s_client -starttls smtp -no_ssl2 -connect host:25 -- Viktor.
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 19.08.2015, at 20:21, Michael Grimm trash...@odo.in-berlin.de wrote: I will revert to OpenSSL my primary mx, first. Done. BTW: LibreSSL 2.2.2 broke unbound 1.5.4 as well. Then I will come back to this issue and provide you with tcpdump debugging info. Now, my secondary is postfix/LibrSSL, only. Regards, Michael
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 19.08.2015, at 21:40, Viktor Dukhovni postfix-us...@dukhovni.org wrote: I've figured out what's going on. LibreSSL 2.2.2 appears to have disabled support for the SSLv2-compatible client HELLO. Servers that have not disabled SSLv2 are unable to complete an SSLv2-compatible TLS handshake with LibreSSL 2.2.2. Connections that use an SSLv2 hello fail. Also clients that use just SSLv3 (no extensions, ...) fail. If I do understand that correctly, it has been a good advice to revert back to OpenSSL running OS != OpenBSD. And, if I am not mistaken, there is no way to tell postfix to work around that disabled support for HELLO. Correct? If you are interested in tcpdumps of connections from russian-caravan.cloud9.net, please let me know. I do have one dump at hand, already. Thanks for your great help, Michael
Re: SSL_accept errors after recent upgrade to LibreSSL 2.2.2
On 08/19/2015 12:11 PM, Michael Grimm wrote: On 19.08.2015, at 20:21, Michael Grimm trash...@odo.in-berlin.de wrote: I will revert to OpenSSL my primary mx, first. Done. BTW: LibreSSL 2.2.2 broke unbound 1.5.4 as well. Already fixed in unbound upstream, they (unbound) were doing an improper version check if I recall instead of feature check. And the patch removed checks specific to them doing something different if libressl was found. That bug wasn't the fault of LibreSSL but of unbound. -=- It's kind of a chicken and egg problem, if LibreSSL isn't recommended because it isn't well tested then it will never be well tested. But if it isn't recommended because of problems with LibreSSL itself that's understandable, but I think at this point most of the bugs with projects building against LibreSSL are actually exposing flaws in the projects that weren't exposed with OpenSSL. If there is any specific testing I can do, I would be happy to, I'm running Postfix 2.11.6 built against LibreSSL 2.2.2 on CentOS 7 - but for less than 48 hours now ;) https://librelamp.com/#postfix From the Postfix page on TLS ``You also turn on thousands and thousands of lines of OpenSSL library code. Assuming that OpenSSL is written as carefully as Wietse's own code, every 1000 lines introduce one additional bug into Postfix.'' We now know OpenSSL has not been written as carefully as Postfix. LibreSSL removed a lot of needless code and has cleaned up a lot of what was left.
