David,
I have no idea where (or even if) incoming TLS sessions are logged.
I've used "openssl s_client" numerous times to see the details of a
connection, but only from the client perspective.
Openssl does have the "s_server" complement of s_client, but I've never
used it:
https://www.openssl.org/docs/man1.1.0/man1/s_server.html
Maybe you could:
1. set a firewall rule to redirect the offending IP to a new port, then
2. run openssl s_server listening on the new port in a terminal window
and thus watch the output of the TLS negotiation (or redirect the output
to a file).
I've never done this. But it seems the easiest way to debug a TLS
negotiation from the server perspective (i.e. see what a remote client
is doing without access to that client). Others on the list may have
better ideas or even some experience doing this.
-Andy
On 4/20/2020 8:15 PM, David Bray wrote:
Hi Andy - you have grasped the problem accurately
As I understand it, the transaction does not leave a great deal of scope
- negotiate the encryption, send a password successfully or
unsuccessfully - (at that point it's logged)
So it has to be the negotiation phase ...
but:
* I've only just built this server
* stuck to a standard install using a CentOS 7 VM - 4GB: 2 CPU, 80GB
o I think this adequate I've seen no OOM events - and the size is
what I've used before
The only thing I'm really seeing here that could be an issue is - the
newer machines are stricter on SSL - the TLS 1/1.2 deprecation thing
I see lots of broken servers, and have to /make allowances/, I do this by:
touch /var/qmail/control/notlshosts/<the host name>
Noting - that is an outbound thing ... (see
https://www.qmailtoaster.org/notls.html)
So ...... is it possible that a broken client is hitting the port, not
being able to make the necessary handshake and causing this CPU load ....
It's reported in the logs when the server makes an outbound transaction
like that ...
deferral:
TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure;_connected_to_103.27.32.20./
What would it look like in my logs if they where to have the reverse issue
David Bray
0418 745334
2 ∞ & <
On Tue, 21 Apr 2020 at 02:54, Andrew Swartz <[email protected]
<mailto:[email protected]>> wrote:
Port 465 should be SMTP over SSL/TLS. Therefore the sequence of
events is:
1. Establish TCP connection
2. Negotiate SSL/TLS session
3. Begin SMTP session
Of these, the SSL/TLS negotiation is by far the most CPU-intensive.
Consider trying to see what is happening with the SSL/TLS negotiation.
It may be failing in a way which is slamming the CPU but not showing up
in the SMTP logs because it never completes and thus an SMTP session is
never initiated.
I'm unsure the best way to debug the incoming SSL/TLS negotiation. You
might set a firewall rule where incoming port 465 from that single
IP is
forwarded to stunnel (on another machine) which is set to output
verbose
debug info???
It would be interesting to know the cause. This could be some clever
DOS attack where a single connection accomplishes the DOS by slamming
the CPU by submitting something invalid to openSSL. But it might just
be that some spammer is using a home-brewed script which is buggy
and is
unintentionally causing this problem.
There seems no efficient way to block this without figuring out the
cause and doing something to make that cause be logged into some log
file. Once that is accomplished, fail2ban (or something similar) can
easily add firewall rules to block individual IP's which exhibit this
behavior.
-Andy
On 4/19/2020 10:12 PM, David Bray wrote:
> Hey thanks Remo
> smtps is an inbound port, they are contacting me - this IP is in
Russia
> somewhere - so do I want to engage (perhaps, probably not but ..)
>
> I could of course block that IP - but that doesn't help, I'd have to
> block endless IPs
>
> I'd like to know what's taking the CPU load, in theory they
should be
> connecting, supplying a password (perhaps) and sending data
> but, there are sometimes bad passwords (2 for the 20th recorded
in maillog)
>
> So..
> What are they doing the other times and why is it taking so much
CPU -
> if it is just a port knock, why the CPU
>
> I need to be able to say, they are bad because ... *what is the
because* ?
>
> David Bray
> 0418 745334
> 2 ∞ & <
>
>
> On Mon, 20 Apr 2020 at 15:32, Remo Mattei <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi,
> Can you reach the server? It maybe blocking you. So what
does your
> queue looks like?
>
> Here is mine for example:
>
> # qmHandle -L
> Messages in local queue: 0
> Messages in remote queue: 0
>
> My other server
>
> # qmHandle -L
> 10355792 (19, L)
> Return-path: [email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> From: Anacron <[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>>
> To: [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> Subject: Anacron job 'cron.daily' on qmailxxxxx.com
<http://qmailxxxxx.com>
> <http://qmailxxxxx.com>
> Date: 19 Apr 2020 10:28:28 -0000
> Size: 509 bytes
>
> 10358746 (6, L)
> Return-path:
> From: [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> To: [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> Subject: failure notice
> Date: 19 Apr 2020 11:30:30 -0000
> Size: 1089 bytes
>
> Messages in local queue: 2
> Messages in remote queue: 0
>
> Just wonder it looks that you are using the SMTPs 465, did
you try
> the 587 Submission that address and see if it goes?
> Just wonder if this is tight to that service.
>
> Maybe none of the above but just for troubleshooting steps, I
would
> try that.
>
> Remo
>
>
>> On Apr 19, 2020, at 22:11, David Bray
<[email protected] <mailto:[email protected]>
>> <mailto:[email protected]
<mailto:[email protected]>>> wrote:
>>
>> Ok - but after all the investigation etc, this is actually the
>> trigger which caught my eye in the first place
>>
>> How this comes about is:
>>
>> 1. User say hey I can't send
>> 2. I look and see this high CPU load and intermittent failures
>> for client to send
>>
>> Any thoughts on where to start looking ..
>>
>>
>> <image.png>
>>
>> my smtp/smtps are currently *10*/11 connections
>>
>>
>> ==> /var/log/qmail/smtp/current <==
>> 2020-04-20 05:07:50.207299500 tcpserver: end 29699 status 0
>> 2020-04-20 05:07:50.207300500 tcpserver: status: 0/60
>>
>> ==> /var/log/qmail/smtps/current <==
>> 2020-04-20 05:07:54.903665500 tcpserver: status: 9/60
>> 2020-04-20 05:07:54.936654500 tcpserver: pid 29725 from
185.50.149.5
>> 2020-04-20 05:07:54.936655500 tcpserver: ok 29725
>> dev.brayworth.com <http://dev.brayworth.com>
<http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>> <http://172.105.181.18:465> :185.50.149.5::5622
>> 2020-04-20 05:08:00.108657500 tcpserver: status: 10/60
>> 2020-04-20 05:08:00.152909500 tcpserver: pid 29734 from
185.50.149.5
>> 2020-04-20 05:08:00.152910500 tcpserver: ok 29734
>> dev.brayworth.com <http://dev.brayworth.com>
<http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>> <http://172.105.181.18:465> :185.50.149.5::62006
>> 2020-04-20 05:08:05.172650500 tcpserver: status: *11*/60
>> 2020-04-20 05:08:05.208983500 tcpserver: pid 29740 from
185.50.149.5
>> 2020-04-20 05:08:05.208984500 tcpserver: ok 29740
>> dev.brayworth.com <http://dev.brayworth.com>
<http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>> <http://172.105.181.18:465> :185.50.149.5::19686
>> 2020-04-20 05:08:13.601336500 tcpserver: end 29690 status 256
>> 2020-04-20 05:08:13.601337500 tcpserver: status: 10/60
>>
>> David Bray
>> 0418 745334
>> 2 ∞ & <
>>
>>
>> On Sun, 19 Apr 2020 at 10:04, David Bray
<[email protected] <mailto:[email protected]>
>> <mailto:[email protected]
<mailto:[email protected]>>> wrote:
>>
>> Thanks Eric
>>
>> It's hard to track things but I think I have had success
>> monitoring the /var/log/maillog
>>
>> I'm not sure why I didn't pick this up earlier, I'm
>> already using the fail2ban suggestion of the older
>> qmailtoaster wiki
>> (http://wiki.qmailtoaster.com/index.php/Fail2Ban), actually
>> had a rule to process it and have expanded on this now
>>
>> I've been running email servers most of my working life and
>> still get tripped up by simple stuff
>>
>> Thank for your efforts in this area, it helps to talk
things out
>>
>> cheers
>>
>> David Bray
>> 0418 745334
>> 2 ∞ & <
>>
>>
>> On Sun, 19 Apr 2020 at 01:12, Eric Broch
>> <[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>> wrote:
>>
>> It looks like a connect and disconnect. If there was
>> authentication you'd see it. I don't think you have
>> anything to worry about here. I'm not saying there's not
>> some jerk out there messing with your smtps...just
saying
>> it may be harmless. That said, do you have a good
firewall
>> in place that prevents DOS attacks. I use Sonicwall
myself
>> but you can do the same thing as others have shown with
>> iptables.
>>
>> Does anyone know how to do the same with the stock
>> firewalld on COS7/8?
>>
>> On 4/17/2020 11:49 PM, David Bray wrote:
>>> sure - thanks for replying, this comes in waves taking
>>> the server to it's maximum at times
>>>
>>> as far as I can see this only logs are this:
>>>
>>> ==> /var/log/qmail/smtps/current <==
>>> 2020-04-18 05:04:48.450871500 tcpserver: status: 6/60
>>> 2020-04-18 05:04:48.480785500 tcpserver: pid 13339 from
>>> 141.98.80.30
>>> 2020-04-18 05:04:48.480787500 tcpserver: ok 13339
>>> dev.brayworth.com <http://dev.brayworth.com>
>>> <http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>>> <http://172.105.181.18:465> :141.98.80.30::25638
>>> 2020-04-18 05:04:52.797644500 tcpserver: status: 7/60
>>> 2020-04-18 05:04:52.830767500 tcpserver: pid 13340 from
>>> 141.98.80.30
>>> 2020-04-18 05:04:52.830768500 tcpserver: ok 13340
>>> dev.brayworth.com <http://dev.brayworth.com>
>>> <http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>>> <http://172.105.181.18:465> :141.98.80.30::14862
>>> 2020-04-18 05:04:57.248902500 tcpserver: status: 8/60
>>> 2020-04-18 05:04:57.304003500 tcpserver: pid 13342 from
>>> 141.98.80.30
>>> 2020-04-18 05:04:57.304006500 tcpserver: ok 13342
>>> dev.brayworth.com <http://dev.brayworth.com>
>>> <http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>>> <http://172.105.181.18:465> :141.98.80.30::9646
>>> 2020-04-18 05:05:01.854790500 tcpserver: status: 9/60
>>> 2020-04-18 05:05:01.902265500 tcpserver: pid 13345 from
>>> 141.98.80.30
>>> 2020-04-18 05:05:01.902266500 tcpserver: ok 13345
>>> dev.brayworth.com <http://dev.brayworth.com>
>>> <http://dev.brayworth.com>:172.105.181.18:465
<http://172.105.181.18:465>
>>> <http://172.105.181.18:465> :141.98.80.30::54058
>>> 2020-04-18 05:05:09.729711500 tcpserver: end 13338
status 256
>>> 2020-04-18 05:05:09.729713500 tcpserver: status: 8/60
>>> 2020-04-18 05:06:05.965715500 tcpserver: end 13342
status 256
>>> 2020-04-18 05:06:05.965716500 tcpserver: status: 7/60
>>> 2020-04-18 05:06:06.141272500 tcpserver: end 13340
status 256
>>> 2020-04-18 05:06:06.141273500 tcpserver: status: 6/60
>>>
>>> David Bray
>>> 0418 745334
>>> 2 ∞ & <
>>>
>>>
>>> On Sat, 18 Apr 2020 at 15:41, Eric Broch
>>> <[email protected]
<mailto:[email protected]>
>>> <mailto:[email protected]
<mailto:[email protected]>>> wrote:
>>>
>>> Can you send the log of one of the "bad"
connections?
>>>
>>> On 4/17/2020 10:59 PM, David Bray wrote:
>>>
>>>> I can see I'm getting hammered on my smtps port
>>>>
>>>> How can I mitigate this?
>>>>
>>>> I can see the IP's in /var/log/qmail/smtps/current
>>>>
>>>> *but where do I actually see that the smtp auth
>>>> actually fails ?*
>>>>
>>>> or do I need to increase the logging somewhere ?
>>>>
>>>> if I tail -f /var/log/dovecot.log
>>>>
>>>> I can see the imap and pop failures
>>>>
>>>> thanks in advance
>>>>
>>>> David Bray
>>>> 0418 745334
>>>> 2 ∞ & <
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]
<mailto:[email protected]>
For additional commands, e-mail:
[email protected]
<mailto:[email protected]>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
