Thankfully CentOS 7 using 1.0.2k so not affected - thanks for the tip though
David Bray 0418 745334 2 ∞ & < On Wed, 22 Apr 2020 at 09:40, Andrew Swartz <[email protected]> wrote: > David, > > I just received this OpenSSL security advisory which may be describing > your problem. It describes a vulnerability which allows a DOS attack by > submitting an invalid certificate. > > https://www.openssl.org/news/secadv/20200421.txt > > > -Andy > > > On 4/20/2020 8:15 PM, David Bray wrote: > > Hi Andy - you have grasped the problem accurately > > > > As I understand it, the transaction does not leave a great deal of scope > > - negotiate the encryption, send a password successfully or > > unsuccessfully - (at that point it's logged) > > > > So it has to be the negotiation phase ... > > but: > > > > * I've only just built this server > > * stuck to a standard install using a CentOS 7 VM - 4GB: 2 CPU, 80GB > > o I think this adequate I've seen no OOM events - and the size is > > what I've used before > > > > The only thing I'm really seeing here that could be an issue is - the > > newer machines are stricter on SSL - the TLS 1/1.2 deprecation thing > > I see lots of broken servers, and have to /make allowances/, I do this > by: > > > > touch /var/qmail/control/notlshosts/<the host name> > > > > > > Noting - that is an outbound thing ... (see > > https://www.qmailtoaster.org/notls.html) > > > > So ...... is it possible that a broken client is hitting the port, not > > being able to make the necessary handshake and causing this CPU load .... > > It's reported in the logs when the server makes an outbound transaction > > like that ... > > > > deferral: > > > > TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure;_connected_to_103.27.32.20./ > > > > > > What would it look like in my logs if they where to have the reverse > issue > > > > > > > > > > > > David Bray > > 0418 745334 > > 2 ∞ & < > > > > > > On Tue, 21 Apr 2020 at 02:54, Andrew Swartz <[email protected] > > <mailto:[email protected]>> wrote: > > > > Port 465 should be SMTP over SSL/TLS. Therefore the sequence of > > events is: > > > > 1. Establish TCP connection > > 2. Negotiate SSL/TLS session > > 3. Begin SMTP session > > > > Of these, the SSL/TLS negotiation is by far the most CPU-intensive. > > > > Consider trying to see what is happening with the SSL/TLS > negotiation. > > It may be failing in a way which is slamming the CPU but not showing > up > > in the SMTP logs because it never completes and thus an SMTP session > is > > never initiated. > > > > I'm unsure the best way to debug the incoming SSL/TLS negotiation. > You > > might set a firewall rule where incoming port 465 from that single > > IP is > > forwarded to stunnel (on another machine) which is set to output > > verbose > > debug info??? > > > > It would be interesting to know the cause. This could be some clever > > DOS attack where a single connection accomplishes the DOS by slamming > > the CPU by submitting something invalid to openSSL. But it might > just > > be that some spammer is using a home-brewed script which is buggy > > and is > > unintentionally causing this problem. > > > > There seems no efficient way to block this without figuring out the > > cause and doing something to make that cause be logged into some log > > file. Once that is accomplished, fail2ban (or something similar) can > > easily add firewall rules to block individual IP's which exhibit this > > behavior. > > > > -Andy > > > > > > > > > > > > On 4/19/2020 10:12 PM, David Bray wrote: > > > Hey thanks Remo > > > smtps is an inbound port, they are contacting me - this IP is in > > Russia > > > somewhere - so do I want to engage (perhaps, probably not but ..) > > > > > > I could of course block that IP - but that doesn't help, I'd have > to > > > block endless IPs > > > > > > I'd like to know what's taking the CPU load, in theory they > > should be > > > connecting, supplying a password (perhaps) and sending data > > > but, there are sometimes bad passwords (2 for the 20th recorded > > in maillog) > > > > > > So.. > > > What are they doing the other times and why is it taking so much > > CPU - > > > if it is just a port knock, why the CPU > > > > > > I need to be able to say, they are bad because ... *what is the > > because* ? > > > > > > David Bray > > > 0418 745334 > > > 2 ∞ & < > > > > > > > > > On Mon, 20 Apr 2020 at 15:32, Remo Mattei <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > Hi, > > > Can you reach the server? It maybe blocking you. So what > > does your > > > queue looks like? > > > > > > Here is mine for example: > > > > > > # qmHandle -L > > > Messages in local queue: 0 > > > Messages in remote queue: 0 > > > > > > My other server > > > > > > # qmHandle -L > > > 10355792 (19, L) > > > Return-path: [email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > From: Anacron <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>> > > > To: [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > Subject: Anacron job 'cron.daily' on qmailxxxxx.com > > <http://qmailxxxxx.com> > > > <http://qmailxxxxx.com> > > > Date: 19 Apr 2020 10:28:28 -0000 > > > Size: 509 bytes > > > > > > 10358746 (6, L) > > > Return-path: > > > From: [email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> > > > To: [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > Subject: failure notice > > > Date: 19 Apr 2020 11:30:30 -0000 > > > Size: 1089 bytes > > > > > > Messages in local queue: 2 > > > Messages in remote queue: 0 > > > > > > Just wonder it looks that you are using the SMTPs 465, did > > you try > > > the 587 Submission that address and see if it goes? > > > Just wonder if this is tight to that service. > > > > > > Maybe none of the above but just for troubleshooting steps, I > > would > > > try that. > > > > > > Remo > > > > > > > > >> On Apr 19, 2020, at 22:11, David Bray > > <[email protected] <mailto:[email protected]> > > >> <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > >> > > >> Ok - but after all the investigation etc, this is actually > the > > >> trigger which caught my eye in the first place > > >> > > >> How this comes about is: > > >> > > >> 1. User say hey I can't send > > >> 2. I look and see this high CPU load and intermittent > failures > > >> for client to send > > >> > > >> Any thoughts on where to start looking .. > > >> > > >> > > >> <image.png> > > >> > > >> my smtp/smtps are currently *10*/11 connections > > >> > > >> > > >> ==> /var/log/qmail/smtp/current <== > > >> 2020-04-20 05:07:50.207299500 tcpserver: end 29699 status 0 > > >> 2020-04-20 05:07:50.207300500 tcpserver: status: 0/60 > > >> > > >> ==> /var/log/qmail/smtps/current <== > > >> 2020-04-20 05:07:54.903665500 tcpserver: status: 9/60 > > >> 2020-04-20 05:07:54.936654500 tcpserver: pid 29725 from > > 185.50.149.5 > > >> 2020-04-20 05:07:54.936655500 tcpserver: ok 29725 > > >> dev.brayworth.com <http://dev.brayworth.com> > > <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >> <http://172.105.181.18:465> :185.50.149.5::5622 > > >> 2020-04-20 05:08:00.108657500 tcpserver: status: 10/60 > > >> 2020-04-20 05:08:00.152909500 tcpserver: pid 29734 from > > 185.50.149.5 > > >> 2020-04-20 05:08:00.152910500 tcpserver: ok 29734 > > >> dev.brayworth.com <http://dev.brayworth.com> > > <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >> <http://172.105.181.18:465> :185.50.149.5::62006 > > >> 2020-04-20 05:08:05.172650500 tcpserver: status: *11*/60 > > >> 2020-04-20 05:08:05.208983500 tcpserver: pid 29740 from > > 185.50.149.5 > > >> 2020-04-20 05:08:05.208984500 tcpserver: ok 29740 > > >> dev.brayworth.com <http://dev.brayworth.com> > > <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >> <http://172.105.181.18:465> :185.50.149.5::19686 > > >> 2020-04-20 05:08:13.601336500 tcpserver: end 29690 status 256 > > >> 2020-04-20 05:08:13.601337500 tcpserver: status: 10/60 > > >> > > >> David Bray > > >> 0418 745334 > > >> 2 ∞ & < > > >> > > >> > > >> On Sun, 19 Apr 2020 at 10:04, David Bray > > <[email protected] <mailto:[email protected]> > > >> <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > >> > > >> Thanks Eric > > >> > > >> It's hard to track things but I think I have had success > > >> monitoring the /var/log/maillog > > >> > > >> I'm not sure why I didn't pick this up earlier, I'm > > >> already using the fail2ban suggestion of the older > > >> qmailtoaster wiki > > >> (http://wiki.qmailtoaster.com/index.php/Fail2Ban), > actually > > >> had a rule to process it and have expanded on this now > > >> > > >> I've been running email servers most of my working life > and > > >> still get tripped up by simple stuff > > >> > > >> Thank for your efforts in this area, it helps to talk > > things out > > >> > > >> cheers > > >> > > >> David Bray > > >> 0418 745334 > > >> 2 ∞ & < > > >> > > >> > > >> On Sun, 19 Apr 2020 at 01:12, Eric Broch > > >> <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > >> > > >> It looks like a connect and disconnect. If there was > > >> authentication you'd see it. I don't think you have > > >> anything to worry about here. I'm not saying there's > not > > >> some jerk out there messing with your smtps...just > > saying > > >> it may be harmless. That said, do you have a good > > firewall > > >> in place that prevents DOS attacks. I use Sonicwall > > myself > > >> but you can do the same thing as others have shown > with > > >> iptables. > > >> > > >> Does anyone know how to do the same with the stock > > >> firewalld on COS7/8? > > >> > > >> On 4/17/2020 11:49 PM, David Bray wrote: > > >>> sure - thanks for replying, this comes in waves > taking > > >>> the server to it's maximum at times > > >>> > > >>> as far as I can see this only logs are this: > > >>> > > >>> ==> /var/log/qmail/smtps/current <== > > >>> 2020-04-18 05:04:48.450871500 tcpserver: status: > 6/60 > > >>> 2020-04-18 05:04:48.480785500 tcpserver: pid 13339 > from > > >>> 141.98.80.30 > > >>> 2020-04-18 05:04:48.480787500 tcpserver: ok 13339 > > >>> dev.brayworth.com <http://dev.brayworth.com> > > >>> <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >>> <http://172.105.181.18:465> :141.98.80.30::25638 > > >>> 2020-04-18 05:04:52.797644500 tcpserver: status: > 7/60 > > >>> 2020-04-18 05:04:52.830767500 tcpserver: pid 13340 > from > > >>> 141.98.80.30 > > >>> 2020-04-18 05:04:52.830768500 tcpserver: ok 13340 > > >>> dev.brayworth.com <http://dev.brayworth.com> > > >>> <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >>> <http://172.105.181.18:465> :141.98.80.30::14862 > > >>> 2020-04-18 05:04:57.248902500 tcpserver: status: > 8/60 > > >>> 2020-04-18 05:04:57.304003500 tcpserver: pid 13342 > from > > >>> 141.98.80.30 > > >>> 2020-04-18 05:04:57.304006500 tcpserver: ok 13342 > > >>> dev.brayworth.com <http://dev.brayworth.com> > > >>> <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >>> <http://172.105.181.18:465> :141.98.80.30::9646 > > >>> 2020-04-18 05:05:01.854790500 tcpserver: status: > 9/60 > > >>> 2020-04-18 05:05:01.902265500 tcpserver: pid 13345 > from > > >>> 141.98.80.30 > > >>> 2020-04-18 05:05:01.902266500 tcpserver: ok 13345 > > >>> dev.brayworth.com <http://dev.brayworth.com> > > >>> <http://dev.brayworth.com>:172.105.181.18:465 > > <http://172.105.181.18:465> > > >>> <http://172.105.181.18:465> :141.98.80.30::54058 > > >>> 2020-04-18 05:05:09.729711500 tcpserver: end 13338 > > status 256 > > >>> 2020-04-18 05:05:09.729713500 tcpserver: status: > 8/60 > > >>> 2020-04-18 05:06:05.965715500 tcpserver: end 13342 > > status 256 > > >>> 2020-04-18 05:06:05.965716500 tcpserver: status: > 7/60 > > >>> 2020-04-18 05:06:06.141272500 tcpserver: end 13340 > > status 256 > > >>> 2020-04-18 05:06:06.141273500 tcpserver: status: > 6/60 > > >>> > > >>> David Bray > > >>> 0418 745334 > > >>> 2 ∞ & < > > >>> > > >>> > > >>> On Sat, 18 Apr 2020 at 15:41, Eric Broch > > >>> <[email protected] > > <mailto:[email protected]> > > >>> <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > >>> > > >>> Can you send the log of one of the "bad" > > connections? > > >>> > > >>> On 4/17/2020 10:59 PM, David Bray wrote: > > >>> > > >>>> I can see I'm getting hammered on my smtps port > > >>>> > > >>>> How can I mitigate this? > > >>>> > > >>>> I can see the IP's in > /var/log/qmail/smtps/current > > >>>> > > >>>> *but where do I actually see that the smtp auth > > >>>> actually fails ?* > > >>>> > > >>>> or do I need to increase the logging somewhere > ? > > >>>> > > >>>> if I tail -f /var/log/dovecot.log > > >>>> > > >>>> I can see the imap and pop failures > > >>>> > > >>>> thanks in advance > > >>>> > > >>>> David Bray > > >>>> 0418 745334 > > >>>> 2 ∞ & < > > >>> > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: > > [email protected] > > <mailto:[email protected]> > > For additional commands, e-mail: > > [email protected] > > <mailto:[email protected]> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
