Re: ssh under attack - sessions in accepted state hogging CPU
On 10/08/10 05.13, Matt Emmerton wrote: I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? If the attack you're experiencing is trying to exhaust system resources by opening a large number of connections, then you may want to toggle these options in sshd_config: ClientAliveInterval LoginGraceTime MaxAuthTries MaxSessions MaxStartups Check the man-page. Secondly, check your logs if this attack is from a limited range of IPs, if so, you might want to try block those ranges. If your users will only connect from your country, then blocking other countries in your firewall is very effective. Thanks to everyone for their help. I did have MaxSessions set to a small number, but that essentially DoS'd my access to the server when enough sshd processes got hung. sshguard+ipfw was blocking a large number of attacks, but the other attacks that were coming in and hanging sshd weren't getting caught (because they weren't repetitive.) I have moved some of my servers to alternate ports, and on the others I tweaked some of the settings Erik suggested which has helped a lot. Thanks for all the advice. -- Matt ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
One thing I don't see mentioned a lot is port knocking. It's not perfect but it does have it's uses. Since it sounds like you have a lot of users that need to connect you might be able to adapt it to your situation. I haven't tried this specific port knocking sequence but you could setup a knock where if a user attempts to connect to port 22 say 3 times (most clients should auto retry) it then opens up port 22 to that ip and allows them to connect to sshd. This would depend on the type of brute force being done. A distributed botnet might only try an ip/port once or twice then move on. This would be pretty seemless to the end user except for an initial delay when connecting as their client retries the connection until the specific knock threshold has been hit. It's a middle ground to changing the port sshd is operating on. You can do this with firewall rules or http://www.freshports.org/security/knock/. A lot of SSH attacks are coming from large numbers of compromised hosts that make them very hard to stop with sshguard which is pretty annoying. On 8/9/2010 8:13 PM, Matt Emmerton wrote: Hi all, I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? root 39127 35.2 0.1 6724 3036 ?? Rs 11:10PM 0:37.91 sshd: [accepted] (sshd) root 39368 33.6 0.1 6724 3036 ?? Rs 11:10PM 0:22.99 sshd: [accepted] (sshd) root 39138 33.1 0.1 6724 3036 ?? Rs 11:10PM 0:41.94 sshd: [accepted] (sshd) root 39137 32.5 0.1 6724 3036 ?? Rs 11:10PM 0:36.56 sshd: [accepted] (sshd) root 39135 31.0 0.1 6724 3036 ?? Rs 11:10PM 0:35.09 sshd: [accepted] (sshd) root 39366 30.9 0.1 6724 3036 ?? Rs 11:10PM 0:23.01 sshd: [accepted] (sshd) root 39132 30.8 0.1 6724 3036 ?? Rs 11:10PM 0:35.21 sshd: [accepted] (sshd) root 39131 30.7 0.1 6724 3036 ?? Rs 11:10PM 0:38.07 sshd: [accepted] (sshd) root 39134 30.2 0.1 6724 3036 ?? Rs 11:10PM 0:40.96 sshd: [accepted] (sshd) root 39367 29.3 0.1 6724 3036 ?? Rs 11:10PM 0:22.08 sshd: [accepted] (sshd) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 39597 root 1 1030 6724K 3036K RUN 3 0:28 35.06% sshd 39599 root 1 1030 6724K 3036K RUN 0 0:26 34.96% sshd 39596 root 1 1030 6724K 3036K RUN 0 0:27 34.77% sshd 39579 root 1 1030 6724K 3036K CPU33 0:28 33.69% sshd 39592 root 1 1020 6724K 3036K RUN 2 0:27 32.18% sshd 39591 root 1 1020 6724K 3036K CPU22 0:27 31.88% sshd -- Matt Emmerton ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
Hi, Matt-- On Aug 9, 2010, at 8:13 PM, Matt Emmerton wrote: I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? If I wasn't allowed to require that in order to SSH to arbitrary internal machines one would need to do a VPN session, the second choice would be to install the openssh port with tcpwrappers support + denyhosts. Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
On 8/9/2010 8:13 PM, Matt Emmerton wrote: Hi all, I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? root 39127 35.2 0.1 6724 3036 ?? Rs 11:10PM 0:37.91 sshd: [accepted] (sshd) root 39368 33.6 0.1 6724 3036 ?? Rs 11:10PM 0:22.99 sshd: [accepted] (sshd) root 39138 33.1 0.1 6724 3036 ?? Rs 11:10PM 0:41.94 sshd: [accepted] (sshd) root 39137 32.5 0.1 6724 3036 ?? Rs 11:10PM 0:36.56 sshd: [accepted] (sshd) root 39135 31.0 0.1 6724 3036 ?? Rs 11:10PM 0:35.09 sshd: [accepted] (sshd) root 39366 30.9 0.1 6724 3036 ?? Rs 11:10PM 0:23.01 sshd: [accepted] (sshd) root 39132 30.8 0.1 6724 3036 ?? Rs 11:10PM 0:35.21 sshd: [accepted] (sshd) root 39131 30.7 0.1 6724 3036 ?? Rs 11:10PM 0:38.07 sshd: [accepted] (sshd) root 39134 30.2 0.1 6724 3036 ?? Rs 11:10PM 0:40.96 sshd: [accepted] (sshd) root 39367 29.3 0.1 6724 3036 ?? Rs 11:10PM 0:22.08 sshd: [accepted] (sshd) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 39597 root 1 1030 6724K 3036K RUN 3 0:28 35.06% sshd 39599 root 1 1030 6724K 3036K RUN 0 0:26 34.96% sshd 39596 root 1 1030 6724K 3036K RUN 0 0:27 34.77% sshd 39579 root 1 1030 6724K 3036K CPU33 0:28 33.69% sshd 39592 root 1 1020 6724K 3036K RUN 2 0:27 32.18% sshd 39591 root 1 1020 6724K 3036K CPU22 0:27 31.88% sshd -- Matt Emmerton Hi. There is a cracking/DoS technique, that tries to exhaust a servers resources, by continualy issuing connect requests, in the hope that when the stack croaks in some way, it'll somehow drop it's guard, or go off air permanently. Have you upset anyone recently? Can you not move your services to non standard IP ports, moving away from the standard ports, where all the script kiddies bots hang out, or are your clients cast in concrete? I've got FTP, Web and SSH systems running on two sites, on very non standard ports, with next to no one trying to get in as a result, but maintaining full visibility to the clients that need them, and know where they are! All my standard ports (80, 21, 22 etc) show as non existant to the outside world, except on one site, where the mail server is continualy getting hammered, but the site's ISP say they cant forward mail to any other port. The users have no problems, so long as I correctly specify the port with the address to them, as in 'address:port' if I send them a link etc, or an example how to fill in a connection dialog. DJB. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
In freebsd-questions Digest, Vol 323, Issue 3, Message: 35 On Mon, 9 Aug 2010 23:36:57 -0400 Matt Emmerton m...@gsicomp.on.ca wrote: I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? 1. switch over to using solely RSA keys In the works; I have too many users to convert :( 2. switch to a non-standard port This is not attractive, even though it would be effective. I tried this once already and my support volume skyrocketed so I had to switch back. Matt, I've seen later responses; portknocking, tcpwrappers + denyhosts etc. The latter works, well but keeping lists of $badguys updated is becoming more intensive all the time against botnets. If you're in a position to permit only connections from a table of IP addresses, maybe subnets, there's lots you can do to block connections from elsewhere before they get to sshd (or tcpwrappers), eg with ipfw: ipfw add $rule allow tcp from table(22) to me 22 in recv $ext_if setup ipfw add deny $logifdesired tcp from any to me 22 in recv $ext_if setup Add keep-state, or earlier allow established connections, to taste. For users with varying IPs you can have them do a (say) POP mail ckeck or anything requiring auth, tail its log either live or from a maybe 5 minute cronjob to add $goodguys table entries, simple scripting and it's not too onerous training roaming users to (eg) check mail before login. Adding `date +%s` as the value for added table entries, it's easy enough deleting dynamic entries after some period of time, by cron. If you can't limit connections to just $goodguys for logistic reasons you can at least use ipfw 'limit' rules to allow only say one or two ssh connections from one IP, which should help the open connections issue. You could also impose connection limits running sshd from inetd(8): [/max-child[/max-connections-per-ip-per-minute[/max-child-per-ip]]] HTH, Ian ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
On 10/08/2010 15:25, Dave wrote: On 8/9/2010 8:13 PM, Matt Emmerton wrote: Hi all, I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? root 39127 35.2 0.1 6724 3036 ?? Rs 11:10PM 0:37.91 sshd: [accepted] (sshd) root 39368 33.6 0.1 6724 3036 ?? Rs 11:10PM 0:22.99 sshd: [accepted] (sshd) root 39138 33.1 0.1 6724 3036 ?? Rs 11:10PM 0:41.94 sshd: [accepted] (sshd) root 39137 32.5 0.1 6724 3036 ?? Rs 11:10PM 0:36.56 sshd: [accepted] (sshd) root 39135 31.0 0.1 6724 3036 ?? Rs 11:10PM 0:35.09 sshd: [accepted] (sshd) root 39366 30.9 0.1 6724 3036 ?? Rs 11:10PM 0:23.01 sshd: [accepted] (sshd) root 39132 30.8 0.1 6724 3036 ?? Rs 11:10PM 0:35.21 sshd: [accepted] (sshd) root 39131 30.7 0.1 6724 3036 ?? Rs 11:10PM 0:38.07 sshd: [accepted] (sshd) root 39134 30.2 0.1 6724 3036 ?? Rs 11:10PM 0:40.96 sshd: [accepted] (sshd) root 39367 29.3 0.1 6724 3036 ?? Rs 11:10PM 0:22.08 sshd: [accepted] (sshd) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 39597 root 1 1030 6724K 3036K RUN 3 0:28 35.06% sshd 39599 root 1 1030 6724K 3036K RUN 0 0:26 34.96% sshd 39596 root 1 1030 6724K 3036K RUN 0 0:27 34.77% sshd 39579 root 1 1030 6724K 3036K CPU33 0:28 33.69% sshd 39592 root 1 1020 6724K 3036K RUN 2 0:27 32.18% sshd 39591 root 1 1020 6724K 3036K CPU22 0:27 31.88% sshd -- Matt Emmerton Hi. There is a cracking/DoS technique, that tries to exhaust a servers resources, by continualy issuing connect requests, in the hope that when the stack croaks in some way, it'll somehow drop it's guard, or go off air permanently. Have you upset anyone recently? Can you not move your services to non standard IP ports, moving away from the standard ports, where all the script kiddies bots hang out, or are your clients cast in concrete? I've got FTP, Web and SSH systems running on two sites, on very non standard ports, with next to no one trying to get in as a result, but maintaining full visibility to the clients that need them, and know where they are! All my standard ports (80, 21, 22 etc) show as non existant to the outside world, except on one site, where the mail server is continualy getting hammered, but the site's ISP say they cant forward mail to any other port. I'm in agreement with dave here, about ssh anyway moving ssh to a non std port makes a massive difference, do it now! Paul. -- - Paul Macdonald IFDNRG Ltd Web and video hosting - t: 0131 5548070 m: 07534206249 e: p...@ifdnrg.com w: http://www.ifdnrg.com - IFDNRG 40 Maritime Street Edinburgh EH6 6SA - ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
On 8/9/2010 8:13 PM, Matt Emmerton wrote: Hi all, I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? There is a cracking/DoS technique, that tries to exhaust a servers resources, by continualy issuing connect requests, in the hope that when the stack croaks in some way, it'll somehow drop it's guard, or go off air permanently. Have you upset anyone recently? Not that I know of - unless my wife counts :) Can you not move your services to non standard IP ports, moving away from the standard ports, where all the script kiddies bots hang out, or are your clients cast in concrete? Right now, they are cast in concrete. I want to move many of them to public keys, so maybe I will change the port at the same time too. I've got FTP, Web and SSH systems running on two sites, on very non standard ports, with next to no one trying to get in as a result, but maintaining full visibility to the clients that need them, and know where they are! All my standard ports (80, 21, 22 etc) show as non existant to the outside world, except on one site, where the mail server is continualy getting hammered, but the site's ISP say they cant forward mail to any other port. I have two servers on the same IP block, and one is getting brute-forced and the other is not. I guess it's just a matter of time before the botnets seek it out. The users have no problems, so long as I correctly specify the port with the address to them, as in 'address:port' if I send them a link etc, or an example how to fill in a connection dialog. I'm seriously going to consider this. -- Matt ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
On 10/08/10 05.13, Matt Emmerton wrote: I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? If the attack you're experiencing is trying to exhaust system resources by opening a large number of connections, then you may want to toggle these options in sshd_config: ClientAliveInterval LoginGraceTime MaxAuthTries MaxSessions MaxStartups Check the man-page. Secondly, check your logs if this attack is from a limited range of IPs, if so, you might want to try block those ranges. If your users will only connect from your country, then blocking other countries in your firewall is very effective. BR, Erik ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
ssh under attack - sessions in accepted state hogging CPU
Hi all, I'm in the middle of dealing with a SSH brute force attack that is relentless. I'm working on getting sshguard+ipfw in place to deal with it, but in the meantime, my box is getting pegged because sshd is accepting some connections which are getting stuck in [accepted] state and eating CPU. I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? root 39127 35.2 0.1 6724 3036 ?? Rs 11:10PM 0:37.91 sshd: [accepted] (sshd) root 39368 33.6 0.1 6724 3036 ?? Rs 11:10PM 0:22.99 sshd: [accepted] (sshd) root 39138 33.1 0.1 6724 3036 ?? Rs 11:10PM 0:41.94 sshd: [accepted] (sshd) root 39137 32.5 0.1 6724 3036 ?? Rs 11:10PM 0:36.56 sshd: [accepted] (sshd) root 39135 31.0 0.1 6724 3036 ?? Rs 11:10PM 0:35.09 sshd: [accepted] (sshd) root 39366 30.9 0.1 6724 3036 ?? Rs 11:10PM 0:23.01 sshd: [accepted] (sshd) root 39132 30.8 0.1 6724 3036 ?? Rs 11:10PM 0:35.21 sshd: [accepted] (sshd) root 39131 30.7 0.1 6724 3036 ?? Rs 11:10PM 0:38.07 sshd: [accepted] (sshd) root 39134 30.2 0.1 6724 3036 ?? Rs 11:10PM 0:40.96 sshd: [accepted] (sshd) root 39367 29.3 0.1 6724 3036 ?? Rs 11:10PM 0:22.08 sshd: [accepted] (sshd) PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 39597 root 1 1030 6724K 3036K RUN 3 0:28 35.06% sshd 39599 root 1 1030 6724K 3036K RUN 0 0:26 34.96% sshd 39596 root 1 1030 6724K 3036K RUN 0 0:27 34.77% sshd 39579 root 1 1030 6724K 3036K CPU33 0:28 33.69% sshd 39592 root 1 1020 6724K 3036K RUN 2 0:27 32.18% sshd 39591 root 1 1020 6724K 3036K CPU22 0:27 31.88% sshd -- Matt Emmerton ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? 1. switch over to using solely RSA keys In the works; I have too many users to convert :( 2. switch to a non-standard port This is not attractive, even though it would be effective. I tried this once already and my support volume skyrocketed. 3. what version of openssh are you currently using? Whatever ships with 8.0-REL, which appears to be: Best James= ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? 1. switch over to using solely RSA keys In the works; I have too many users to convert :( 2. switch to a non-standard port This is not attractive, even though it would be effective. I tried this once already and my support volume skyrocketed so I had to switch back. 3. what version of openssh are you currently using? Whatever ships with 8.0-REL, which appears to be: OpenSSL version OpenSSL 0.9.8k 25 Mar 2009 Regards, -- Matt ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: ssh under attack - sessions in accepted state hogging CPU
Hi Matt, I know there's not much I can do about the brute force attacks, but will upgrading openssh avoid these stuck connections? 1. switch over to using solely RSA keys 2. switch to a non-standard port 3. what version of openssh are you currently using? Best James___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
