I keep a list of which IPs have hit my app, and how many times over the last 10 minutes. If something is at 200 requests in 10 minutes I send me an email. And either white list it in my app, or black list it.
I have an "Attack mode" that has rules I enable when I feel I'm being attacked that looks for malformed Browser header, byteGet requests, query parameters that my app doesn't support. Those then start returning access denied with 0 byte bodies. From: [email protected] [mailto:[email protected]] On Behalf Of Alexander Konovalenko Sent: Friday, September 30, 2011 9:34 AM To: Google App Engine group Subject: [google-appengine] Defending against serious DDoS attacks and out of preview Now that App Engine is going out of preview, I'm concerned about the way DDoS attacks on individual apps are handled. If my paid app is hit by a serious DDoS attack, what can I do? The goal is to keep the site up for users while not paying too much due to excessive resource usage. Say, it is Friday night, 11 p.m. PDT. My site suddenly starts to see a lot of requests from a zombie botnet. 1000 zombies issue 10 requests per second each. Those are all dynamic requests that return 30 KB of compressed HTML on average. Another 9000 zombie hosts send the average of 0.3 requests per second (1/3 of which are dynamic requests). The bots try to mimic human behavior by downloading images, stylesheets and other embedded files. Average static response size is 100 KB. Totals: 10,900 dynamic and 1800 static requests per second. The attack consumes 52 Mbps of incoming bandwidth and 4 Gbps of outgoing bandwidth (at peak while not yet blocked), not counting the TCP/IP overhead. My off-site monitoring system alerts me that something is wrong two minutes after the attack started. 30 minutes later, I have completed the initial investigation, have learned how to automatically identify some malicious requests and used up all 100 entries in my dos.yaml blacklist to block some of the 1000 most active zombies. Now what? * What's the right channel to contact the right person from the App Engine team for help? Should I file a Production issue in the bug tracker? A billing <http://code.google.com/support/bin/request.py?contact_type=AppEngineBilling Support> support <http://code.google.com/support/bin/request.py?contact_type=AppEngineBilling Support> request <http://code.google.com/support/bin/request.py?contact_type=AppEngineBilling Support> ? Should I write to this group? * If the attack is big, should I notify someone else, perhaps the Google security team ([email protected])? What constitutes big enough? * How fast can someone react if I file a support request on Friday night? * Are you able to raise the maximum number of blacklist entries to 10,000 or more on a case-by-case basis? * Should I increase my app's budget indefinitely to keep the site running for customers in the hope that I will be issued a credit <http://code.google.com/appengine/kb/billing.html#dos> later as per the Billing FAQ? How can I tell in advance whether I will get full refund for the DDoS-related charges? If left untreated, my sample DDoS attack would cost $43.6 per hour for front-end instances (assuming each request uses 50 ms of CPU time under a multi-threaded Python runtime) and $276 per hour for bandwidth. That amounts to $320 per hour or $7680 per day. Plus up to $94 per hour for datastore operations if every dynamic request is specially crafted to bypass memcache and cause the app to issue 2 datastore reads and 1 write. However, there is not much I can do on my own. The blacklist is limited to 100 entries which is ridiculous when you face a real DDoS. And the dos.yaml even does not prevent the attackers from exhausting my bandwidth via static files. Two issues to star: DoS protection should protect static files http://code.google.com/p/googleappengine/issues/detail?id=4607 Raise the maximum number of entries in dos.yaml http://code.google.com/p/googleappengine/issues/detail?id=6000 See also: Debugging the "unusual traffic" error (another DoS protection mechanism) http://code.google.com/p/googleappengine/issues/detail?id=5462 Running a reverse proxy in front of App Engine (that'd be also relevant if you decide to route all traffic through a DDoS scrubbing center) http://code.google.com/p/googleappengine/issues/detail?id=5463 http://code.google.com/p/googleappengine/issues/detail?id=5464 Disable access via .appspot.com URL http://code.google.com/p/googleappengine/issues/detail?id=5576 Advice from Thomas Ptacek http://news.ycombinator.com/item?id=1987345 -- Alexander -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
