David,

I just received this OpenSSL security advisory which may be describing your problem. It describes a vulnerability which allows a DOS attack by submitting an invalid certificate.

https://www.openssl.org/news/secadv/20200421.txt


-Andy


On 4/20/2020 8:15 PM, David Bray wrote:
Hi Andy - you have grasped the problem accurately

As I understand it, the transaction does not leave a great deal of scope - negotiate the encryption, send a password successfully or unsuccessfully - (at that point it's logged)

So it has to be the negotiation phase ...
but:

  * I've only just built this server
  * stuck to a standard install using a CentOS 7 VM - 4GB: 2 CPU, 80GB
      o I think this adequate I've seen no OOM events - and the size is
        what I've used before

The only thing I'm really seeing here that could be an issue is - the newer machines are stricter on SSL - the TLS 1/1.2 deprecation thing
I see lots of broken servers, and have to /make allowances/, I do this by:

    touch /var/qmail/control/notlshosts/<the host name>


Noting - that is an outbound thing ... (see https://www.qmailtoaster.org/notls.html)

So ...... is it possible that a broken client is hitting the port, not being able to make the necessary handshake and causing this CPU load .... It's reported in the logs when the server makes an outbound transaction like that ...

    deferral:
    
TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure;_connected_to_103.27.32.20./


What would it look like in my logs if they where to have the reverse issue





David Bray
0418 745334
2 ∞ & <


On Tue, 21 Apr 2020 at 02:54, Andrew Swartz <awswa...@acsalaska.net <mailto:awswa...@acsalaska.net>> wrote:

    Port 465 should be SMTP over SSL/TLS.  Therefore the sequence of
    events is:

    1.  Establish TCP connection
    2.  Negotiate SSL/TLS session
    3.  Begin SMTP session

    Of these, the SSL/TLS negotiation is by far the most CPU-intensive.

    Consider trying to see what is happening with the SSL/TLS negotiation.
    It may be failing in a way which is slamming the CPU but not showing up
    in the SMTP logs because it never completes and thus an SMTP session is
    never initiated.

    I'm unsure the best way to debug the incoming SSL/TLS negotiation.  You
    might set a firewall rule where incoming port 465 from that single
    IP is
    forwarded to stunnel (on another machine) which is set to output
    verbose
    debug info???

    It would be interesting to know the cause.  This could be some clever
    DOS attack where a single connection accomplishes the DOS by slamming
    the CPU by submitting something invalid to openSSL.  But it might just
    be that some spammer is using a home-brewed script which is buggy
    and is
    unintentionally causing this problem.

    There seems no efficient way to block this without figuring out the
    cause and doing something to make that cause be logged into some log
    file.  Once that is accomplished, fail2ban (or something similar) can
    easily add firewall rules to block individual IP's which exhibit this
    behavior.

    -Andy





    On 4/19/2020 10:12 PM, David Bray wrote:
     > Hey thanks Remo
     > smtps is an inbound port, they are contacting me - this IP is in
    Russia
     > somewhere - so do I want to engage (perhaps, probably not but ..)
     >
     > I could of course block that IP - but that doesn't help, I'd have to
     > block endless IPs
     >
     > I'd like to know what's taking the CPU load, in theory they
    should be
     > connecting, supplying a password (perhaps) and sending data
     > but, there are sometimes bad passwords (2 for the 20th recorded
    in maillog)
     >
     > So..
     > What are they doing the other times and why is it taking so much
    CPU -
     > if it is just a port knock, why the CPU
     >
     > I need to be able to say, they are bad because ... *what is the
    because* ?
     >
     > David Bray
     > 0418 745334
     > 2 ∞ & <
     >
     >
     > On Mon, 20 Apr 2020 at 15:32, Remo Mattei <r...@mattei.org
    <mailto:r...@mattei.org>
     > <mailto:r...@mattei.org <mailto:r...@mattei.org>>> wrote:
     >
     >     Hi,
     >     Can you reach the server?  It maybe blocking you. So what
    does your
     >     queue looks like?
     >
     >     Here is mine for example:
     >
     >     # qmHandle -L
     >     Messages in local queue: 0
     >     Messages in remote queue: 0
     >
     >     My other server
     >
     >     # qmHandle -L
     >     10355792 (19, L)
     >        Return-path: r...@qmailxxxxx.com
    <mailto:r...@qmailxxxxx.com> <mailto:r...@qmailxxxxx.com
    <mailto:r...@qmailxxxxx.com>>
     >        From: Anacron <r...@qmailxxxx.com
    <mailto:r...@qmailxxxx.com> <mailto:r...@qmailxxxx.com
    <mailto:r...@qmailxxxx.com>>>
     >        To: r...@qmailxxxxx.com <mailto:r...@qmailxxxxx.com>
    <mailto:r...@qmailxxxxx.com <mailto:r...@qmailxxxxx.com>>
     >        Subject: Anacron job 'cron.daily' on qmailxxxxx.com
    <http://qmailxxxxx.com>
     >     <http://qmailxxxxx.com>
     >        Date: 19 Apr 2020 10:28:28 -0000
     >        Size: 509 bytes
     >
     >     10358746 (6, L)
     >        Return-path:
     >        From: mailer-dae...@qmailxxxxxx.com
    <mailto:mailer-dae...@qmailxxxxxx.com>
     >     <mailto:mailer-dae...@qmailxxxxxx.com
    <mailto:mailer-dae...@qmailxxxxxx.com>>
     >        To: r...@qmailxxxxxxxx.com <mailto:r...@qmailxxxxxxxx.com>
    <mailto:r...@qmailxxxxxxxx.com <mailto:r...@qmailxxxxxxxx.com>>
     >        Subject: failure notice
     >        Date: 19 Apr 2020 11:30:30 -0000
     >        Size: 1089 bytes
     >
     >     Messages in local queue: 2
     >     Messages in remote queue: 0
     >
     >     Just wonder it looks that you are using the SMTPs 465, did
    you try
     >     the 587 Submission that address and see if it goes?
     >     Just wonder if this is tight to that service.
     >
     >     Maybe none of the above but just for troubleshooting steps, I
    would
     >     try that.
     >
     >     Remo
     >
     >
     >>     On Apr 19, 2020, at 22:11, David Bray
    <da...@brayworth.com.au <mailto:da...@brayworth.com.au>
     >>     <mailto:da...@brayworth.com.au
    <mailto:da...@brayworth.com.au>>> wrote:
     >>
     >>     Ok - but after all the investigation etc, this is actually the
     >>     trigger which caught my eye in the first place
     >>
     >>     How this comes about is:
     >>
     >>      1. User say hey I can't send
     >>      2. I look and see this high CPU load and intermittent failures
     >>         for client to send
     >>
     >>     Any thoughts on where to start looking ..
     >>
     >>
     >>     <image.png>
     >>
     >>     my smtp/smtps are currently *10*/11 connections
     >>
     >>
     >>     ==> /var/log/qmail/smtp/current <==
     >>     2020-04-20 05:07:50.207299500 tcpserver: end 29699 status 0
     >>     2020-04-20 05:07:50.207300500 tcpserver: status: 0/60
     >>
     >>     ==> /var/log/qmail/smtps/current <==
     >>     2020-04-20 05:07:54.903665500 tcpserver: status: 9/60
     >>     2020-04-20 05:07:54.936654500 tcpserver: pid 29725 from
    185.50.149.5
     >>     2020-04-20 05:07:54.936655500 tcpserver: ok 29725
     >> dev.brayworth.com <http://dev.brayworth.com>
    <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>     <http://172.105.181.18:465> :185.50.149.5::5622
     >>     2020-04-20 05:08:00.108657500 tcpserver: status: 10/60
     >>     2020-04-20 05:08:00.152909500 tcpserver: pid 29734 from
    185.50.149.5
     >>     2020-04-20 05:08:00.152910500 tcpserver: ok 29734
     >> dev.brayworth.com <http://dev.brayworth.com>
    <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>     <http://172.105.181.18:465> :185.50.149.5::62006
     >>     2020-04-20 05:08:05.172650500 tcpserver: status: *11*/60
     >>     2020-04-20 05:08:05.208983500 tcpserver: pid 29740 from
    185.50.149.5
     >>     2020-04-20 05:08:05.208984500 tcpserver: ok 29740
     >> dev.brayworth.com <http://dev.brayworth.com>
    <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>     <http://172.105.181.18:465> :185.50.149.5::19686
     >>     2020-04-20 05:08:13.601336500 tcpserver: end 29690 status 256
     >>     2020-04-20 05:08:13.601337500 tcpserver: status: 10/60
     >>
     >>     David Bray
     >>     0418 745334
     >>     2 ∞ & <
     >>
     >>
     >>     On Sun, 19 Apr 2020 at 10:04, David Bray
    <da...@brayworth.com.au <mailto:da...@brayworth.com.au>
     >>     <mailto:da...@brayworth.com.au
    <mailto:da...@brayworth.com.au>>> wrote:
     >>
     >>         Thanks Eric
     >>
     >>         It's hard to track things but I think I have had success
     >>         monitoring the /var/log/maillog
     >>
     >>         I'm not sure why I didn't pick this up earlier, I'm
     >>         already using the fail2ban suggestion of the older
     >>         qmailtoaster wiki
     >>         (http://wiki.qmailtoaster.com/index.php/Fail2Ban), actually
     >>         had a rule to process it and have expanded on this now
     >>
     >>         I've been running email servers most of my working life and
     >>         still get tripped up by simple stuff
     >>
     >>         Thank for your efforts in this area, it helps to talk
    things out
     >>
     >>         cheers
     >>
     >>         David Bray
     >>         0418 745334
     >>         2 ∞ & <
     >>
     >>
     >>         On Sun, 19 Apr 2020 at 01:12, Eric Broch
     >>         <ebr...@whitehorsetc.com
    <mailto:ebr...@whitehorsetc.com> <mailto:ebr...@whitehorsetc.com
    <mailto:ebr...@whitehorsetc.com>>> wrote:
     >>
     >>             It looks like a connect and disconnect. If there was
     >>             authentication you'd see it. I don't think you have
     >>             anything to worry about here. I'm not saying there's not
     >>             some jerk out there messing with your smtps...just
    saying
     >>             it may be harmless. That said, do you have a good
    firewall
     >>             in place that prevents DOS attacks. I use Sonicwall
    myself
     >>             but you can do the same thing as others have shown with
     >>             iptables.
     >>
     >>             Does anyone know how to do the same with the stock
     >>             firewalld on COS7/8?
     >>
     >>             On 4/17/2020 11:49 PM, David Bray wrote:
     >>>             sure - thanks for replying, this comes in waves taking
     >>>             the server to it's maximum at times
     >>>
     >>>             as far as I can see this only logs are this:
     >>>
     >>>             ==> /var/log/qmail/smtps/current <==
     >>>             2020-04-18 05:04:48.450871500 tcpserver: status: 6/60
     >>>             2020-04-18 05:04:48.480785500 tcpserver: pid 13339 from
     >>>             141.98.80.30
     >>>             2020-04-18 05:04:48.480787500 tcpserver: ok 13339
     >>> dev.brayworth.com <http://dev.brayworth.com>
     >>>             <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>>             <http://172.105.181.18:465> :141.98.80.30::25638
     >>>             2020-04-18 05:04:52.797644500 tcpserver: status: 7/60
     >>>             2020-04-18 05:04:52.830767500 tcpserver: pid 13340 from
     >>>             141.98.80.30
     >>>             2020-04-18 05:04:52.830768500 tcpserver: ok 13340
     >>> dev.brayworth.com <http://dev.brayworth.com>
     >>>             <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>>             <http://172.105.181.18:465> :141.98.80.30::14862
     >>>             2020-04-18 05:04:57.248902500 tcpserver: status: 8/60
     >>>             2020-04-18 05:04:57.304003500 tcpserver: pid 13342 from
     >>>             141.98.80.30
     >>>             2020-04-18 05:04:57.304006500 tcpserver: ok 13342
     >>> dev.brayworth.com <http://dev.brayworth.com>
     >>>             <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>>             <http://172.105.181.18:465> :141.98.80.30::9646
     >>>             2020-04-18 05:05:01.854790500 tcpserver: status: 9/60
     >>>             2020-04-18 05:05:01.902265500 tcpserver: pid 13345 from
     >>>             141.98.80.30
     >>>             2020-04-18 05:05:01.902266500 tcpserver: ok 13345
     >>> dev.brayworth.com <http://dev.brayworth.com>
     >>>             <http://dev.brayworth.com>:172.105.181.18:465
    <http://172.105.181.18:465>
     >>>             <http://172.105.181.18:465> :141.98.80.30::54058
     >>>             2020-04-18 05:05:09.729711500 tcpserver: end 13338
    status 256
     >>>             2020-04-18 05:05:09.729713500 tcpserver: status: 8/60
     >>>             2020-04-18 05:06:05.965715500 tcpserver: end 13342
    status 256
     >>>             2020-04-18 05:06:05.965716500 tcpserver: status: 7/60
     >>>             2020-04-18 05:06:06.141272500 tcpserver: end 13340
    status 256
     >>>             2020-04-18 05:06:06.141273500 tcpserver: status: 6/60
     >>>
     >>>             David Bray
     >>>             0418 745334
     >>>             2 ∞ & <
     >>>
     >>>
     >>>             On Sat, 18 Apr 2020 at 15:41, Eric Broch
     >>>             <ebr...@whitehorsetc.com
    <mailto:ebr...@whitehorsetc.com>
     >>>             <mailto:ebr...@whitehorsetc.com
    <mailto:ebr...@whitehorsetc.com>>> wrote:
     >>>
     >>>                 Can you send the log of one of the "bad"
    connections?
     >>>
     >>>                 On 4/17/2020 10:59 PM, David Bray wrote:
     >>>
     >>>>                 I can see I'm getting hammered on my smtps port
     >>>>
     >>>>                 How can I mitigate this?
     >>>>
     >>>>                 I can see the IP's in /var/log/qmail/smtps/current
     >>>>
     >>>>                 *but where do I actually see that the smtp auth
     >>>>                 actually fails ?*
     >>>>
     >>>>                 or do I need to increase the logging somewhere ?
     >>>>
     >>>>                 if I tail -f /var/log/dovecot.log
     >>>>
     >>>>                 I can see the imap and pop failures
     >>>>
     >>>>                 thanks in advance
     >>>>
     >>>>                 David Bray
     >>>>                 0418 745334
     >>>>                 2 ∞ & <
     >>>
     >

    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    qmailtoaster-list-unsubscr...@qmailtoaster.com
    <mailto:qmailtoaster-list-unsubscr...@qmailtoaster.com>
    For additional commands, e-mail:
    qmailtoaster-list-h...@qmailtoaster.com
    <mailto:qmailtoaster-list-h...@qmailtoaster.com>


---------------------------------------------------------------------
To unsubscribe, e-mail: qmailtoaster-list-unsubscr...@qmailtoaster.com
For additional commands, e-mail: qmailtoaster-list-h...@qmailtoaster.com

Reply via email to