Matt, Yeah, I hesitated to call it an "attack" per se but the sheer volume of requests for the apps default page, several per second, is clearly not anything resembling normal, and the end result was my server going down. Sure, it could be something misconfigured on the sending end but very abnormal nonetheless.
I did modify the logrotate configuration to use file size. Thanks for the suggestion on tweaking the verbosity and format of the Rails log output -- I'll definitely do that. I am looking into using logentries.com, too. It looks like a nice service. Cheers, Chris On Sun, Oct 27, 2013 at 10:24 PM, Matt Aimonetti <[email protected]>wrote: > Hey Chris, > > Thanks for sharing, interesting issue I've also seen with people simply > forgetting to set logrotate. > I wouldn't call that an attack, the IP could be a proxy, a scrapper, a > response time monitor or indeed someone doing a very slow, single IP based > DDOS. > > I think that really the issue isn't with the traffic you were getting but > with your log management. > You can use logrotate with min/maxsize and timeperiod to avoid these kind > of issues. > You might also want to not use the default very verbose Rails output and > instead use a more compact, more easily parsable log format. > Another option is to use something like logentries.com and not keep any > logs locally (or logrotate more aggressively). > > Let me know if have any questions and best of luck. > > - Matt > > > On Sun, Oct 27, 2013 at 9:17 PM, Chris McCann <[email protected]>wrote: > >> SD Ruby, >> >> This morning I checked on a Rails app I've had in production for many >> years and found that the app was dumping a full stacktrace to the screen >> instead of either rendering the expected page or showing the "Oops. >> Something went wrong" error page. >> >> I immediately tried to do a cap web:deploy:disable to at least put up the >> maintenance page but that failed, too. It was slightly unnerving since >> nothing like this had happened to me before. >> >> Looking into it I saw a MySQL error with an error code of 28 in the >> message. A quick Google of that showed that's what MySQL does when it's >> out of disk space. >> >> A quick "df -h" and sure enough, the 6 GB disk showed 0% available. WTF, >> I wondered. >> >> I headed over to the Rails log directory to see what the production log >> showed. Well, the first thing it showed was the production.log file was >> almost 1 GB. That's not normal, as logrotate is set to rotate the log >> every week and I've barely ever seen one even 1/10 that size. >> >> I gzipped the log and pulled it down locally for a little spelunking. >> Before long I found a suspicious IP address hitting the Site#index page >> over, and over, and over -- 2,131,853 times, to be exact. That caused the >> production log to blow way past the available disk space on the slice. >> Truncating the log immediately restored the apps functionality. >> >> As a quick reaction measure I added the offending IP to be dropped via >> iptables. The customer support techs at Rimuhosting.com (who are, frankly, >> the most responsive I've ever worked with) were very helpful, and they even >> let me know I was eligible for a free upgrade of over 1/2 GB of RAM and 6 >> GB of disk space, which they promptly added. >> >> I modified my logrotate conf file for the app to rotate every 100MB >> instead of weekly. If you're rotating on time like I was you're setting >> yourself up for what I'm calling a "log-bloat denial of service" attack. >> >> A slightly scary but instructive lesson, which leads me to my questions: >> >> - has anyone dealt with a similar style attack in a Rails app? >> >> - are there some security best practices I'm not implementing? >> >> - has anyone used fail2ban or similar software with a Rails app to >> automatically block nefarious traffic? >> >> - what are your favorite monitoring solutions to get early warnings of >> disk space, RAM, or other impending problems? >> >> Thanks, >> >> Chris McCann >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to the Google Groups >> "SD Ruby" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/P7-OnkqYffA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
