Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
Hi Sam, Thanks for the thorough reply. On 25 Jun 2020 14:50:55, Sam Clippinger via spamdyke-users wrote: > If tls_read() encountered an error in the > OpenSSL library and didn't actually read any data, spamdyke's main > loop would see data waiting and call tls_read() again. That could > cause an infinite loop. I'll upgrade spamdyke, and hope the fix was done. > There are a few things you could try. As Bucky Carr pointed out, the > softlimit program causes all kinds of problems and leads to very > strange errors and crashes. What are your thought on the RCE in qmail that I mentioned my other email? I'll try raising the memory limit a bit. > You could also try upgrading OpenSSL. That's the latest version provide by the distro, which has an EOL Nov 30 of this year. Rebuilding the server on a new distro is on my todo list. Thanks again, Quinn ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
2.8M lines in 34 seconds? Yikes! Sounds like an infinite loop. It's been a while since I've looked at that code (and I apologize I don't have time to go through it in detail), but that error message is only printed from one place in spamdyke's code. It runs when a TLS/SSL session is active and data is waiting to be read from the network connection. If tls_read() encountered an error in the OpenSSL library and didn't actually read any data, spamdyke's main loop would see data waiting and call tls_read() again. That could cause an infinite loop. If I'm reading the error message correctly, SSL_get_error() must have returned SSL_ERROR_SSL, which is the catch-all code for a protocol or library failure. I think reason 255 is SSL_R_UNKNOWN_STATE, another catch-all error code. There are a few things you could try. As Bucky Carr pointed out, the softlimit program causes all kinds of problems and leads to very strange errors and crashes. If you can remove it, you should. If not, you could try increasing the memory limit (try doubling it) and see if that changes anything. Choosing a memory limit for softlimit is just a guessing game anyway; maybe you need to guess higher? You could also try upgrading OpenSSL. Your version is very old and this spamdyke error may be caused by a bug OpenSSL has already fixed. (Plus, OpenSSL 1.0.1e contains a huge number of serious CVEs which upgrading would fix.) And also you could try upgrading spamdyke. Between versions 4 and 5, I made a lot of changes and the changelog mentions tls_read() specifically. This could be a bug I've already fixed. I hope that helps, good luck! -- Sam Clippinger > On Jun 24, 2020, at 11:36 PM, Quinn Comendant via spamdyke-users > wrote: > > Hello all, > > Recently, I checked the smtp log files of my qmailtoaster server, and found > millions of the following error message written to the smtp log: > > spamdyke[4875]: ERROR: unable to read from SSL/TLS stream: A protocol or > library failure occurred, error:140800FF:lib(20):func(128):reason(255) > > I restarted the mail-related services (qmailctl stop && qmailctl start), and > the errors stopped. I thought it was a fluke, since I had never seen this in > the decade+ I've been managing this server. However, a few days later, I > found the same thing. Again, I restarted and the errors stopped. > > The really weird thing is that all the errors that were logged (~ 2.8 million > lines) occurred over 34 seconds (from 2020-06-25 03:47:58 to 2020-06-25 > 03:48:36)! I'd guess that only *one* error occurred (memory error? > buffer-overrun?) which somehow caused an infinite loop of logging. > > Although the last logged error was at 2020-06-25 03:48:36, I didn't discover > the issue until 2020-06-25 04:10, which means there was at least a 20 minute > delay between when the log line was time-stamped and when the line was > finally added to the log, perhaps caused by IO constraints). > > So there's a couple issues I'm worried about: > > 1. why did spamdyke get stuck in an infinite loop? > 2. what caused this error in the first place? > > Versions: > > OpenSSL 1.0.1e-fips 11 Feb 2013 > spamdyke 4.3.1+TLS+CONFIGTEST+DEBUG+EXCESSIVE > > Spamdyke is executed via the /var/qmail/supervise/smtp/run file: > > QMAILDUID=`id -u vpopmail` > NOFILESGID=`id -g vpopmail` > MAXSMTPD=`cat /var/qmail/control/concurrencyincoming` > SPAMDYKE="/usr/bin/spamdyke" > SPAMDYKE_CONF="/etc/spamdyke.conf" > SMTPD="/var/qmail/bin/qmail-smtpd" > TCP_CDB="/etc/tcprules.d/tcp.smtp.cdb" > HOSTNAME=`hostname` > VCHKPW="/home/vpopmail/bin/vchkpw" > REQUIRE_AUTH=0 > > exec /usr/bin/softlimit -m 9900 \ > /usr/bin/tcpserver -R -l $HOSTNAME -x $TCP_CDB -c "$MAXSMTPD" \ > -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \ > $SPAMDYKE --config-file $SPAMDYKE_CONF \ > $SMTPD $VCHKPW /bin/true 2>&1 > > Thanks! > > ___ > spamdyke-users mailing list > spamdyke-users@spamdyke.org > https://spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
On 24 Jun 2020 23:36:44, Quinn Comendant via spamdyke-users wrote: > I thought it was a fluke, since I had never seen this in the decade+ > I've been managing this server. Actually, there were a few single instances of this error outside of the 34 seconds when it blew up. I found four: 2020-06-25 06:11:16.154545500 spamdyke[20782]: ERROR: unable to start SSL/TLS connection: A protocol or library failure occurred, error:1408A0C1:lib(20):func(138):reason(193) […] 2020-06-25 11:56:18.666164500 spamdyke[14604]: ERROR: unable to start SSL/TLS connection: A protocol or library failure occurred, error:1408A0C1:lib(20):func(138):reason(193) […] 2020-06-25 15:19:02.951764500 spamdyke[18632]: ERROR: unable to start SSL/TLS connection: A protocol or library failure occurred, error:1408A0C1:lib(20):func(138):reason(193) […] 2020-06-25 15:19:13.346393500 spamdyke[18701]: ERROR: unable to start SSL/TLS connection: A protocol or library failure occurred, error:14094412:lib(20):func(148):reason(1042) I notice that the reason codes (`reason(193)` and `reason(1042)`) are different than the reason in the other 2.8 million lines logged (`reason(255)`). I'm guessing those are openssl error codes? Quinn ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
Hi Bucky, On 25 Jun 2020 07:12:27, Bucky Carr via spamdyke-users wrote: > Do you need to use 'softlimit'? Yes, using softlimit to restrict process memory limit is useful, and in fact necessary to prevent this remotely-exploitable vulnerability in qmail: https://www.qualys.com/2020/05/19/cve-2005-1513/remote-code-execution-qmail.txt “TLDR: In 2005, three vulnerabilities were discovered in qmail but were never fixed because they were believed to be unexploitable in a default installation. We recently re-discovered these vulnerabilities and were able to exploit one of them remotely in a default installation.” The RCE can be mitigated by: - using softlimit to restrict process memory limit, even on qmail-local - configure databytes to limit email message size. or by applying the patches included in the article linked above. Quinn ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
I know next to nothing, so this may not help. Do you need to use 'softlimit'? When I quit using it a bunch of troubles went away. On 6/24/2020 10:36 PM, Quinn Comendant via spamdyke-users wrote: exec /usr/bin/softlimit -m 9900 \ /usr/bin/tcpserver -R -l $HOSTNAME -x $TCP_CDB -c "$MAXSMTPD" \ -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \ $SPAMDYKE --config-file $SPAMDYKE_CONF \ $SMTPD $VCHKPW /bin/true 2>&1 ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream
Hello all, Recently, I checked the smtp log files of my qmailtoaster server, and found millions of the following error message written to the smtp log: spamdyke[4875]: ERROR: unable to read from SSL/TLS stream: A protocol or library failure occurred, error:140800FF:lib(20):func(128):reason(255) I restarted the mail-related services (qmailctl stop && qmailctl start), and the errors stopped. I thought it was a fluke, since I had never seen this in the decade+ I've been managing this server. However, a few days later, I found the same thing. Again, I restarted and the errors stopped. The really weird thing is that all the errors that were logged (~ 2.8 million lines) occurred over 34 seconds (from 2020-06-25 03:47:58 to 2020-06-25 03:48:36)! I'd guess that only *one* error occurred (memory error? buffer-overrun?) which somehow caused an infinite loop of logging. Although the last logged error was at 2020-06-25 03:48:36, I didn't discover the issue until 2020-06-25 04:10, which means there was at least a 20 minute delay between when the log line was time-stamped and when the line was finally added to the log, perhaps caused by IO constraints). So there's a couple issues I'm worried about: 1. why did spamdyke get stuck in an infinite loop? 2. what caused this error in the first place? Versions: OpenSSL 1.0.1e-fips 11 Feb 2013 spamdyke 4.3.1+TLS+CONFIGTEST+DEBUG+EXCESSIVE Spamdyke is executed via the /var/qmail/supervise/smtp/run file: QMAILDUID=`id -u vpopmail` NOFILESGID=`id -g vpopmail` MAXSMTPD=`cat /var/qmail/control/concurrencyincoming` SPAMDYKE="/usr/bin/spamdyke" SPAMDYKE_CONF="/etc/spamdyke.conf" SMTPD="/var/qmail/bin/qmail-smtpd" TCP_CDB="/etc/tcprules.d/tcp.smtp.cdb" HOSTNAME=`hostname` VCHKPW="/home/vpopmail/bin/vchkpw" REQUIRE_AUTH=0 exec /usr/bin/softlimit -m 9900 \ /usr/bin/tcpserver -R -l $HOSTNAME -x $TCP_CDB -c "$MAXSMTPD" \ -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \ $SPAMDYKE --config-file $SPAMDYKE_CONF \ $SMTPD $VCHKPW /bin/true 2>&1 Thanks! ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
