Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream

2020-06-25 Thread Quinn Comendant via spamdyke-users
Hi Sam,

Thanks for the thorough reply. 

On 25 Jun 2020 14:50:55, Sam Clippinger via spamdyke-users wrote:
> If tls_read() encountered an error in the 
> OpenSSL library and didn't actually read any data, spamdyke's main 
> loop would see data waiting and call tls_read() again.  That could 
> cause an infinite loop.

I'll upgrade spamdyke, and hope the fix was done.

> There are a few things you could try.  As Bucky Carr pointed out, the 
> softlimit program causes all kinds of problems and leads to very 
> strange errors and crashes.

What are your thought on the RCE in qmail that I mentioned my other email?  
I'll try raising the memory limit a bit.

> You could also try upgrading OpenSSL.

That's the latest version provide by the distro, which has an EOL Nov 30 of 
this year. Rebuilding the server on a new distro is on my todo list.

Thanks again,

Quinn
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
https://spamdyke.org/mailman/listinfo/spamdyke-users


Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream

2020-06-25 Thread Sam Clippinger via spamdyke-users
2.8M lines in 34 seconds?  Yikes!  Sounds like an infinite loop.

It's been a while since I've looked at that code (and I apologize I don't have 
time to go through it in detail), but that error message is only printed from 
one place in spamdyke's code.  It runs when a TLS/SSL session is active and 
data is waiting to be read from the network connection.  If tls_read() 
encountered an error in the OpenSSL library and didn't actually read any data, 
spamdyke's main loop would see data waiting and call tls_read() again.  That 
could cause an infinite loop.

If I'm reading the error message correctly, SSL_get_error() must have returned 
SSL_ERROR_SSL, which is the catch-all code for a protocol or library failure.  
I think reason 255 is SSL_R_UNKNOWN_STATE, another catch-all error code.

There are a few things you could try.  As Bucky Carr pointed out, the softlimit 
program causes all kinds of problems and leads to very strange errors and 
crashes.  If you can remove it, you should.  If not, you could try increasing 
the memory limit (try doubling it) and see if that changes anything.  Choosing 
a memory limit for softlimit is just a guessing game anyway; maybe you need to 
guess higher?

You could also try upgrading OpenSSL.  Your version is very old and this 
spamdyke error may be caused by a bug OpenSSL has already fixed.  (Plus, 
OpenSSL 1.0.1e contains a huge number of serious CVEs which upgrading would 
fix.)

And also you could try upgrading spamdyke.  Between versions 4 and 5, I made a 
lot of changes and the changelog mentions tls_read() specifically.  This could 
be a bug I've already fixed.

I hope that helps, good luck!

-- Sam Clippinger




> On Jun 24, 2020, at 11:36 PM, Quinn Comendant via spamdyke-users 
>  wrote:
> 
> Hello all,
> 
> Recently, I checked the smtp log files of my qmailtoaster server, and found 
> millions of the following error message written to the smtp log:
> 
>  spamdyke[4875]: ERROR: unable to read from SSL/TLS stream: A protocol or 
> library failure occurred, error:140800FF:lib(20):func(128):reason(255)
> 
> I restarted the mail-related services (qmailctl stop && qmailctl start), and 
> the errors stopped. I thought it was a fluke, since I had never seen this in 
> the decade+ I've been managing this server. However, a few days later, I 
> found the same thing. Again, I restarted and the errors stopped.
> 
> The really weird thing is that all the errors that were logged (~ 2.8 million 
> lines) occurred over 34 seconds (from 2020-06-25 03:47:58 to 2020-06-25 
> 03:48:36)! I'd guess that only *one* error occurred (memory error? 
> buffer-overrun?) which somehow caused an infinite loop of logging. 
> 
> Although the last logged error was at 2020-06-25 03:48:36, I didn't discover 
> the issue until 2020-06-25 04:10, which means there was at least a 20 minute 
> delay between when the log line was time-stamped and when the line was 
> finally added to the log, perhaps caused by IO constraints).
> 
> So there's a couple issues I'm worried about:
> 
> 1. why did spamdyke get stuck in an infinite loop?
> 2. what caused this error in the first place?
> 
> Versions:
> 
>  OpenSSL 1.0.1e-fips 11 Feb 2013
>  spamdyke 4.3.1+TLS+CONFIGTEST+DEBUG+EXCESSIVE
> 
> Spamdyke is executed via the /var/qmail/supervise/smtp/run file:
> 
>  QMAILDUID=`id -u vpopmail`
>  NOFILESGID=`id -g vpopmail`
>  MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
>  SPAMDYKE="/usr/bin/spamdyke"
>  SPAMDYKE_CONF="/etc/spamdyke.conf"
>  SMTPD="/var/qmail/bin/qmail-smtpd"
>  TCP_CDB="/etc/tcprules.d/tcp.smtp.cdb"
>  HOSTNAME=`hostname`
>  VCHKPW="/home/vpopmail/bin/vchkpw"
>  REQUIRE_AUTH=0
> 
>  exec /usr/bin/softlimit -m 9900 \
>   /usr/bin/tcpserver -R -l $HOSTNAME -x $TCP_CDB -c "$MAXSMTPD" \
>   -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
>   $SPAMDYKE --config-file $SPAMDYKE_CONF \
>   $SMTPD $VCHKPW /bin/true 2>&1
> 
> Thanks!
> 
> ___
> spamdyke-users mailing list
> spamdyke-users@spamdyke.org
> https://spamdyke.org/mailman/listinfo/spamdyke-users

___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
https://spamdyke.org/mailman/listinfo/spamdyke-users


Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream

2020-06-25 Thread Quinn Comendant via spamdyke-users
On 24 Jun 2020 23:36:44, Quinn Comendant via spamdyke-users wrote:
> I thought it was a fluke, since I had never seen this in the decade+ 
> I've been managing this server.

Actually, there were a few single instances of this error outside of the 34 
seconds when it blew up. I found four:

  2020-06-25 06:11:16.154545500 spamdyke[20782]: ERROR: unable to start SSL/TLS 
connection: A protocol or library failure occurred, 
error:1408A0C1:lib(20):func(138):reason(193)
  […]
  2020-06-25 11:56:18.666164500 spamdyke[14604]: ERROR: unable to start SSL/TLS 
connection: A protocol or library failure occurred, 
error:1408A0C1:lib(20):func(138):reason(193)
  […]
  2020-06-25 15:19:02.951764500 spamdyke[18632]: ERROR: unable to start SSL/TLS 
connection: A protocol or library failure occurred, 
error:1408A0C1:lib(20):func(138):reason(193)
  […]
  2020-06-25 15:19:13.346393500 spamdyke[18701]: ERROR: unable to start SSL/TLS 
connection: A protocol or library failure occurred, 
error:14094412:lib(20):func(148):reason(1042)

I notice that the reason codes (`reason(193)` and `reason(1042)`) are different 
than the reason in the other 2.8 million lines logged (`reason(255)`). I'm 
guessing those are openssl error codes?

Quinn
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
https://spamdyke.org/mailman/listinfo/spamdyke-users


Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream

2020-06-25 Thread Quinn Comendant via spamdyke-users
Hi Bucky,

On 25 Jun 2020 07:12:27, Bucky Carr via spamdyke-users wrote:
> Do you need to use 'softlimit'?

Yes, using softlimit to restrict process memory limit is useful, and in fact 
necessary to prevent this remotely-exploitable vulnerability in qmail:

https://www.qualys.com/2020/05/19/cve-2005-1513/remote-code-execution-qmail.txt

“TLDR: In 2005, three vulnerabilities were discovered in qmail but were
never fixed because they were believed to be unexploitable in a default
installation. We recently re-discovered these vulnerabilities and were
able to exploit one of them remotely in a default installation.”

The RCE can be mitigated by:

- using softlimit to restrict process memory limit, even on qmail-local
- configure databytes to limit email message size.

or by applying the patches included in the article linked above. 

Quinn
___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
https://spamdyke.org/mailman/listinfo/spamdyke-users


Re: [spamdyke-users] Infinite loop of logged errors: unable to read from SSL/TLS stream

2020-06-25 Thread Bucky Carr via spamdyke-users



I know next to nothing, so this may not help. Do you need to use 
'softlimit'? When I quit using it a bunch of troubles went away.



On 6/24/2020 10:36 PM, Quinn Comendant via spamdyke-users wrote:

   exec /usr/bin/softlimit -m 9900 \
/usr/bin/tcpserver -R -l $HOSTNAME -x $TCP_CDB -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
$SPAMDYKE --config-file $SPAMDYKE_CONF \
$SMTPD $VCHKPW /bin/true 2>&1



___
spamdyke-users mailing list
spamdyke-users@spamdyke.org
https://spamdyke.org/mailman/listinfo/spamdyke-users