Chris, check out fail2ban for something simple , or modsecurity for a (much) more complicated but thorough solution. Also, papertrailapp.com is a very nice logging-as-a-service.
- John On Sunday, October 27, 2013, Chris McCann wrote: > SD Ruby, > > This morning I checked on a Rails app I've had in production for many > years and found that the app was dumping a full stacktrace to the screen > instead of either rendering the expected page or showing the "Oops. > Something went wrong" error page. > > I immediately tried to do a cap web:deploy:disable to at least put up the > maintenance page but that failed, too. It was slightly unnerving since > nothing like this had happened to me before. > > Looking into it I saw a MySQL error with an error code of 28 in the > message. A quick Google of that showed that's what MySQL does when it's > out of disk space. > > A quick "df -h" and sure enough, the 6 GB disk showed 0% available. WTF, > I wondered. > > I headed over to the Rails log directory to see what the production log > showed. Well, the first thing it showed was the production.log file was > almost 1 GB. That's not normal, as logrotate is set to rotate the log > every week and I've barely ever seen one even 1/10 that size. > > I gzipped the log and pulled it down locally for a little spelunking. > Before long I found a suspicious IP address hitting the Site#index page > over, and over, and over -- 2,131,853 times, to be exact. That caused the > production log to blow way past the available disk space on the slice. > Truncating the log immediately restored the apps functionality. > > As a quick reaction measure I added the offending IP to be dropped via > iptables. The customer support techs at Rimuhosting.com (who are, frankly, > the most responsive I've ever worked with) were very helpful, and they even > let me know I was eligible for a free upgrade of over 1/2 GB of RAM and 6 > GB of disk space, which they promptly added. > > I modified my logrotate conf file for the app to rotate every 100MB > instead of weekly. If you're rotating on time like I was you're setting > yourself up for what I'm calling a "log-bloat denial of service" attack. > > A slightly scary but instructive lesson, which leads me to my questions: > > - has anyone dealt with a similar style attack in a Rails app? > > - are there some security best practices I'm not implementing? > > - has anyone used fail2ban or similar software with a Rails app to > automatically block nefarious traffic? > > - what are your favorite monitoring solutions to get early warnings of > disk space, RAM, or other impending problems? > > Thanks, > > Chris McCann > > -- > -- > SD Ruby mailing list > [email protected] <javascript:_e({}, 'cvml', > '[email protected]');> > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to the Google Groups > "SD Ruby" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:_e({}, 'cvml', > 'sdruby%[email protected]');>. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
