HI David, you can use TCPDump with source IP example tcpdump src 1.1.1.1 And or with ports if you want. Like this
tcpdump -nnvvS src thebadguyipaddress and dst port 25 > On Apr 21, 2020, at 17:15, David Bray <[email protected]> wrote: > > Hey Remo, just looking at Andy's suggestion though > > tcpdump - that only copies the data from the port ? > So if if I were to use Andy's idea - it would be an interference in the port > and the data would have to be proxied to the correct port (or lost) > > tcpdump - can I use that on an existing connection > > What I have here is a lot of connections during the day, but then I notice > the CPU going up (between swimming, running, hiking and other leisure > activities) > > So I just want to say - What are you doing and look - can tcpdump do that ? > I see tcpdump host <ip> is an option ... > > I'll see what I can discover there, thanks Remo/Andy > > David Bray > 0418 745334 > 2 ∞ & < > > > On Wed, 22 Apr 2020 at 09:46, <[email protected] <mailto:[email protected]>> > wrote: > The other is to leverage some of Andy’s suggestions and use tcpdump on that > port and see 😊 > > > Il giorno 21 apr 2020, alle ore 16:40, Andrew Swartz > > <[email protected] <mailto:[email protected]>> ha scritto: > > > > David, > > > > I have no idea where (or even if) incoming TLS sessions are logged. > > > > I've used "openssl s_client" numerous times to see the details of a > > connection, but only from the client perspective. > > > > Openssl does have the "s_server" complement of s_client, but I've never > > used it: > > > > https://www.openssl.org/docs/man1.1.0/man1/s_server.html > > <https://www.openssl.org/docs/man1.1.0/man1/s_server.html> > > > > Maybe you could: > > 1. set a firewall rule to redirect the offending IP to a new port, then > > 2. run openssl s_server listening on the new port in a terminal window and > > thus watch the output of the TLS negotiation (or redirect the output to a > > file). > > > > I've never done this. But it seems the easiest way to debug a TLS > > negotiation from the server perspective (i.e. see what a remote client is > > doing without access to that client). Others on the list may have better > > ideas or even some experience doing this. > > > > -Andy > > > > > > > >> On 4/20/2020 8:15 PM, David Bray wrote: > >> Hi Andy - you have grasped the problem accurately > >> As I understand it, the transaction does not leave a great deal of scope - > >> negotiate the encryption, send a password successfully or unsuccessfully - > >> (at that point it's logged) > >> So it has to be the negotiation phase ... > >> but: > >> * I've only just built this server > >> * stuck to a standard install using a CentOS 7 VM - 4GB: 2 CPU, 80GB > >> o I think this adequate I've seen no OOM events - and the size is > >> what I've used before > >> The only thing I'm really seeing here that could be an issue is - the > >> newer machines are stricter on SSL - the TLS 1/1.2 deprecation thing > >> I see lots of broken servers, and have to /make allowances/, I do this by: > >> touch /var/qmail/control/notlshosts/<the host name> > >> Noting - that is an outbound thing ... (see > >> https://www.qmailtoaster.org/notls.html > >> <https://www.qmailtoaster.org/notls.html>) > >> So ...... is it possible that a broken client is hitting the port, not > >> being able to make the necessary handshake and causing this CPU load .... > >> It's reported in the logs when the server makes an outbound transaction > >> like that ... > >> deferral: > >> > >> TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure;_connected_to_103.27.32.20./ > >> What would it look like in my logs if they where to have the reverse issue > >> David Bray > >> 0418 745334 > >> 2 ∞ & < > >> On Tue, 21 Apr 2020 at 02:54, Andrew Swartz <[email protected] > >> <mailto:[email protected]> <mailto:[email protected] > >> <mailto:[email protected]>>> wrote: > >> Port 465 should be SMTP over SSL/TLS. Therefore the sequence of > >> events is: > >> 1. Establish TCP connection > >> 2. Negotiate SSL/TLS session > >> 3. Begin SMTP session > >> Of these, the SSL/TLS negotiation is by far the most CPU-intensive. > >> Consider trying to see what is happening with the SSL/TLS negotiation. > >> It may be failing in a way which is slamming the CPU but not showing up > >> in the SMTP logs because it never completes and thus an SMTP session is > >> never initiated. > >> I'm unsure the best way to debug the incoming SSL/TLS negotiation. You > >> might set a firewall rule where incoming port 465 from that single > >> IP is > >> forwarded to stunnel (on another machine) which is set to output > >> verbose > >> debug info??? > >> It would be interesting to know the cause. This could be some clever > >> DOS attack where a single connection accomplishes the DOS by slamming > >> the CPU by submitting something invalid to openSSL. But it might just > >> be that some spammer is using a home-brewed script which is buggy > >> and is > >> unintentionally causing this problem. > >> There seems no efficient way to block this without figuring out the > >> cause and doing something to make that cause be logged into some log > >> file. Once that is accomplished, fail2ban (or something similar) can > >> easily add firewall rules to block individual IP's which exhibit this > >> behavior. > >> -Andy > >> On 4/19/2020 10:12 PM, David Bray wrote: > >> > Hey thanks Remo > >> > smtps is an inbound port, they are contacting me - this IP is in > >> Russia > >> > somewhere - so do I want to engage (perhaps, probably not but ..) > >> > > >> > I could of course block that IP - but that doesn't help, I'd have to > >> > block endless IPs > >> > > >> > I'd like to know what's taking the CPU load, in theory they > >> should be > >> > connecting, supplying a password (perhaps) and sending data > >> > but, there are sometimes bad passwords (2 for the 20th recorded > >> in maillog) > >> > > >> > So.. > >> > What are they doing the other times and why is it taking so much > >> CPU - > >> > if it is just a port knock, why the CPU > >> > > >> > I need to be able to say, they are bad because ... *what is the > >> because* ? > >> > > >> > David Bray > >> > 0418 745334 > >> > 2 ∞ & < > >> > > >> > > >> > On Mon, 20 Apr 2020 at 15:32, Remo Mattei <[email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> > <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: > >> > > >> > Hi, > >> > Can you reach the server? It maybe blocking you. So what > >> does your > >> > queue looks like? > >> > > >> > Here is mine for example: > >> > > >> > # qmHandle -L > >> > Messages in local queue: 0 > >> > Messages in remote queue: 0 > >> > > >> > My other server > >> > > >> > # qmHandle -L > >> > 10355792 (19, L) > >> > Return-path: [email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> > >> > From: Anacron <[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> > >> > To: [email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> > >> > Subject: Anacron job 'cron.daily' on qmailxxxxx.com > >> <http://qmailxxxxx.com/> > >> <http://qmailxxxxx.com <http://qmailxxxxx.com/>> > >> > <http://qmailxxxxx.com <http://qmailxxxxx.com/>> > >> > Date: 19 Apr 2020 10:28:28 -0000 > >> > Size: 509 bytes > >> > > >> > 10358746 (6, L) > >> > Return-path: > >> > From: [email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] > >> <mailto:[email protected]>> > >> > <mailto:[email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] > >> <mailto:[email protected]>>> > >> > To: [email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> > >> > Subject: failure notice > >> > Date: 19 Apr 2020 11:30:30 -0000 > >> > Size: 1089 bytes > >> > > >> > Messages in local queue: 2 > >> > Messages in remote queue: 0 > >> > > >> > Just wonder it looks that you are using the SMTPs 465, did > >> you try > >> > the 587 Submission that address and see if it goes? > >> > Just wonder if this is tight to that service. > >> > > >> > Maybe none of the above but just for troubleshooting steps, I > >> would > >> > try that. > >> > > >> > Remo > >> > > >> > > >> >> On Apr 19, 2020, at 22:11, David Bray > >> <[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: > >> >> > >> >> Ok - but after all the investigation etc, this is actually the > >> >> trigger which caught my eye in the first place > >> >> > >> >> How this comes about is: > >> >> > >> >> 1. User say hey I can't send > >> >> 2. I look and see this high CPU load and intermittent failures > >> >> for client to send > >> >> > >> >> Any thoughts on where to start looking .. > >> >> > >> >> > >> >> <image.png> > >> >> > >> >> my smtp/smtps are currently *10*/11 connections > >> >> > >> >> > >> >> ==> /var/log/qmail/smtp/current <== > >> >> 2020-04-20 05:07:50.207299500 tcpserver: end 29699 status 0 > >> >> 2020-04-20 05:07:50.207300500 tcpserver: status: 0/60 > >> >> > >> >> ==> /var/log/qmail/smtps/current <== > >> >> 2020-04-20 05:07:54.903665500 tcpserver: status: 9/60 > >> >> 2020-04-20 05:07:54.936654500 tcpserver: pid 29725 from > >> 185.50.149.5 > >> >> 2020-04-20 05:07:54.936655500 tcpserver: ok 29725 > >> >> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> :185.50.149.5::5622 > >> >> 2020-04-20 05:08:00.108657500 tcpserver: status: 10/60 > >> >> 2020-04-20 05:08:00.152909500 tcpserver: pid 29734 from > >> 185.50.149.5 > >> >> 2020-04-20 05:08:00.152910500 tcpserver: ok 29734 > >> >> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> :185.50.149.5::62006 > >> >> 2020-04-20 05:08:05.172650500 tcpserver: status: *11*/60 > >> >> 2020-04-20 05:08:05.208983500 tcpserver: pid 29740 from > >> 185.50.149.5 > >> >> 2020-04-20 05:08:05.208984500 tcpserver: ok 29740 > >> >> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> :185.50.149.5::19686 > >> >> 2020-04-20 05:08:13.601336500 tcpserver: end 29690 status 256 > >> >> 2020-04-20 05:08:13.601337500 tcpserver: status: 10/60 > >> >> > >> >> David Bray > >> >> 0418 745334 > >> >> 2 ∞ & < > >> >> > >> >> > >> >> On Sun, 19 Apr 2020 at 10:04, David Bray > >> <[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> wrote: > >> >> > >> >> Thanks Eric > >> >> > >> >> It's hard to track things but I think I have had success > >> >> monitoring the /var/log/maillog > >> >> > >> >> I'm not sure why I didn't pick this up earlier, I'm > >> >> already using the fail2ban suggestion of the older > >> >> qmailtoaster wiki > >> >> (http://wiki.qmailtoaster.com/index.php/Fail2Ban > >> <http://wiki.qmailtoaster.com/index.php/Fail2Ban>), actually > >> >> had a rule to process it and have expanded on this now > >> >> > >> >> I've been running email servers most of my working life and > >> >> still get tripped up by simple stuff > >> >> > >> >> Thank for your efforts in this area, it helps to talk > >> things out > >> >> > >> >> cheers > >> >> > >> >> David Bray > >> >> 0418 745334 > >> >> 2 ∞ & < > >> >> > >> >> > >> >> On Sun, 19 Apr 2020 at 01:12, Eric Broch > >> >> <[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> <mailto:[email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> > >> wrote: > >> >> > >> >> It looks like a connect and disconnect. If there was > >> >> authentication you'd see it. I don't think you have > >> >> anything to worry about here. I'm not saying there's not > >> >> some jerk out there messing with your smtps...just > >> saying > >> >> it may be harmless. That said, do you have a good > >> firewall > >> >> in place that prevents DOS attacks. I use Sonicwall > >> myself > >> >> but you can do the same thing as others have shown with > >> >> iptables. > >> >> > >> >> Does anyone know how to do the same with the stock > >> >> firewalld on COS7/8? > >> >> > >> >> On 4/17/2020 11:49 PM, David Bray wrote: > >> >>> sure - thanks for replying, this comes in waves taking > >> >>> the server to it's maximum at times > >> >>> > >> >>> as far as I can see this only logs are this: > >> >>> > >> >>> ==> /var/log/qmail/smtps/current <== > >> >>> 2020-04-18 05:04:48.450871500 tcpserver: status: 6/60 > >> >>> 2020-04-18 05:04:48.480785500 tcpserver: pid 13339 from > >> >>> 141.98.80.30 > >> >>> 2020-04-18 05:04:48.480787500 tcpserver: ok 13339 > >> >>> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> >>> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >>> <http://172.105.181.18:465 > >> <http://172.105.181.18:465/>> :141.98.80.30::25638 > >> >>> 2020-04-18 05:04:52.797644500 tcpserver: status: 7/60 > >> >>> 2020-04-18 05:04:52.830767500 tcpserver: pid 13340 from > >> >>> 141.98.80.30 > >> >>> 2020-04-18 05:04:52.830768500 tcpserver: ok 13340 > >> >>> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> >>> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >>> <http://172.105.181.18:465 > >> <http://172.105.181.18:465/>> :141.98.80.30::14862 > >> >>> 2020-04-18 05:04:57.248902500 tcpserver: status: 8/60 > >> >>> 2020-04-18 05:04:57.304003500 tcpserver: pid 13342 from > >> >>> 141.98.80.30 > >> >>> 2020-04-18 05:04:57.304006500 tcpserver: ok 13342 > >> >>> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> >>> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >>> <http://172.105.181.18:465 > >> <http://172.105.181.18:465/>> :141.98.80.30::9646 > >> >>> 2020-04-18 05:05:01.854790500 tcpserver: status: 9/60 > >> >>> 2020-04-18 05:05:01.902265500 tcpserver: pid 13345 from > >> >>> 141.98.80.30 > >> >>> 2020-04-18 05:05:01.902266500 tcpserver: ok 13345 > >> >>> dev.brayworth.com <http://dev.brayworth.com/> > >> <http://dev.brayworth.com <http://dev.brayworth.com/>> > >> >>> <http://dev.brayworth.com > >> <http://dev.brayworth.com/>>:172.105.181.18:465 > >> <http://172.105.181.18:465/> > >> <http://172.105.181.18:465 <http://172.105.181.18:465/>> > >> >>> <http://172.105.181.18:465 > >> <http://172.105.181.18:465/>> :141.98.80.30::54058 > >> >>> 2020-04-18 05:05:09.729711500 tcpserver: end 13338 > >> status 256 > >> >>> 2020-04-18 05:05:09.729713500 tcpserver: status: 8/60 > >> >>> 2020-04-18 05:06:05.965715500 tcpserver: end 13342 > >> status 256 > >> >>> 2020-04-18 05:06:05.965716500 tcpserver: status: 7/60 > >> >>> 2020-04-18 05:06:06.141272500 tcpserver: end 13340 > >> status 256 > >> >>> 2020-04-18 05:06:06.141273500 tcpserver: status: 6/60 > >> >>> > >> >>> David Bray > >> >>> 0418 745334 > >> >>> 2 ∞ & < > >> >>> > >> >>> > >> >>> On Sat, 18 Apr 2020 at 15:41, Eric Broch > >> >>> <[email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> >>> <mailto:[email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>>> > >> wrote: > >> >>> > >> >>> Can you send the log of one of the "bad" > >> connections? > >> >>> > >> >>> On 4/17/2020 10:59 PM, David Bray wrote: > >> >>> > >> >>>> I can see I'm getting hammered on my smtps port > >> >>>> > >> >>>> How can I mitigate this? > >> >>>> > >> >>>> I can see the IP's in /var/log/qmail/smtps/current > >> >>>> > >> >>>> *but where do I actually see that the smtp auth > >> >>>> actually fails ?* > >> >>>> > >> >>>> or do I need to increase the logging somewhere ? > >> >>>> > >> >>>> if I tail -f /var/log/dovecot.log > >> >>>> > >> >>>> I can see the imap and pop failures > >> >>>> > >> >>>> thanks in advance > >> >>>> > >> >>>> David Bray > >> >>>> 0418 745334 > >> >>>> 2 ∞ & < > >> >>> > >> > > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: > >> [email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] > >> <mailto:[email protected]>> > >> For additional commands, e-mail: > >> [email protected] > >> <mailto:[email protected]> > >> <mailto:[email protected] > >> <mailto:[email protected]>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > <mailto:[email protected]> > > For additional commands, e-mail: [email protected] > > <mailto:[email protected]> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > <mailto:[email protected]> > For additional commands, e-mail: [email protected] > <mailto:[email protected]>
