The Internet is a wild place. :) You will get hits from all kinds of bots, from search engines to trackers of various sorts to spies to malicious attacks. The Internet also has a long memory. Years from now, you will still get hits for pages that have been removed from your application today. You'll need to be constantly tending your application to make sure you're dealing with these efficiently. At the very least, make sure to keep your robots.txt and sitemap.xml up to date, and also use log analysis to make sure you understand the traffic.
Is your JVM facing your Internet connection? I would recommend having some proxy in front of it, whether it's Apache, or a load balancer like Perlbal, or a balancer/cache like Varnish. They can filter a lot of problems before they hit your vulnerable Restlet application. Restlet/Grizzly does very well with high saturation from my experience, but you really want a mature front-facing product to make sure you're not vulnerable to DoS and other storms. I swear by Perlbal, myself. Especially if you're on EC2, you want the potential, at least, to spin up new Restlet instances behind the balancer, for redundancy if not for scalability. Perlbal's documentation is ... non existent. Here's a sample configuration I've been using for a while on production systems: http://threecrickets.com/prudence/manual/http-proxy/ -Tal On Wed, Sep 22, 2010 at 2:46 PM, David Fogel <[email protected]> wrote: > Hi All- > > We've been developing our new web site/service using Restlet, and > we're getting very close to a full launch. In the process, we've been > deploying parts of our site on publicly-accessible servers (mostly > Amazon EC2 instances). In addition to the small amount of traffic > we're generating ourselves, we're seeing a fairly large amount of > "noise" requests, which mostly seem to be robots looking for > vulnerabilities in common systems like PHP and apache. Below are some > examples of resources being requested. > > These requests usually result in a 404, although some have resulted in > 30x redirects, and some have caused errors / stack traces due to > malformed HTTP requests. In particular, restlet threw a > NullPointerException in response to a request that must have had a > malformed tag or conditional request headers: > > WARN Exception or error caught in resource > java.lang.NullPointerException > org.restlet.data.Conditions.getStatus(Conditions.java:310) > org.restlet.data.Conditions.getStatus(Conditions.java:366) > ... > > Anyhow, obviously this is just part of the reality of the internet, > but we're wondering how other Restlet folks deal with this. In > general we would like to avoid devoting CPU or other resources > (including log file space) on this crap. Ideally we would like to > find a solution that not only reduces our resource utilization, but > perhaps increases the attacker's cost in some manner, so they are more > likely to move on and leave our servers alone. Also, we're trying to > get a better grip on any potential Restlet issues that could be > triggered by this stuff that would be disproportionately costly. On > the one hand, it seems unlikely to us any of these attacks will be > specifically written to look for Restlet (or Jetty, or Simple) > vulnerabilities. On the other hand, based on looking at the > discussion archive, it doesn't seem like there's been very much > attention given to dealing with these attacks on a Restlet-based web > stack. > > Any ideas/suggestions/thoughts? > > Thanks, > Dave Fogel > > (Here are the examples I mentioned:) > > INFO GET /HTTP/1.1 => 404 > INFO GET /install.txt => 404 > INFO GET /cart/install.txt => 404 > INFO GET /zencart/install.txt => 404 > INFO GET /zen-cart/install.txt => 404 > INFO GET /zen/install.txt => 404 > INFO GET /shop/install.txt => 404 > INFO GET /butik/install.txt => 404 > INFO GET /zcart/install.txt => 404 > INFO GET /shop2/install.txt => 404 > INFO GET /catalog/install.txt => 404 > INFO GET /boutique/install.txt => 404 > INFO GET /cart/install.txt => 404 > INFO GET /store/install.txt => 404 > INFO GET / => 404 > INFO Couldn't find the mandatory "Host" HTTP header. > INFO GET /robots.txt => 404 > INFO GET / => 301 > INFO GET /w00tw00t.at.blackhats.romanian.anti-sec:) => 404 > INFO GET /scripts/setup.php => 404 > INFO GET /admin/scripts/setup.php => 404 > INFO GET /admin/pma/scripts/setup.php => 404 > INFO GET /admin/phpmyadmin/scripts/setup.php => 404 > INFO GET /db/scripts/setup.php => 404 > INFO GET /dbadmin/scripts/setup.php => 404 > INFO GET /myadmin/scripts/setup.php => 404 > INFO GET /mysql/scripts/setup.php => 404 > INFO GET /mysqladmin/scripts/setup.php => 404 > INFO GET /typo3/phpmyadmin/scripts/setup.php => 404 > INFO GET /phpadmin/scripts/setup.php => 404 > INFO GET /phpMyAdmin/scripts/setup.php => 404 > INFO GET /phpmyadmin/scripts/setup.php => 404 > INFO GET /phpmyadmin1/scripts/setup.php => 404 > INFO GET /phpmyadmin2/scripts/setup.php => 404 > INFO GET /pma/scripts/setup.php => 404 > INFO GET /web/phpMyAdmin/scripts/setup.php => 404 > INFO GET /xampp/phpmyadmin/scripts/setup.php => 404 > INFO GET /web/scripts/setup.php => 404 > INFO GET /php-my-admin/scripts/setup.php => 404 > INFO GET /websql/scripts/setup.php => 404 > INFO GET /phpmyadmin/scripts/setup.php => 404 > INFO GET /phpMyAdmin/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2/scripts/setup.php => 404 > INFO GET /php-my-admin/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.2.3/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.2.6/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.5.1/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.8.0.4/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.8.1-rc1/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.8.1/scripts/setup.php => 404 > INFO GET /phpMyAdmin-2.8.2/scripts/setup.php => 404 > INFO GET /sqlmanager/scripts/setup.php => 404 > INFO GET /mysqlmanager/scripts/setup.php => 404 > INFO GET /p/m/a/scripts/setup.php => 404 > INFO GET /PMA2005/scripts/setup.php => 404 > INFO GET /pma2005/scripts/setup.php => 404 > INFO GET /phpmanager/scripts/setup.php => 404 > INFO GET /php-myadmin/scripts/setup.php => 404 > INFO GET /phpmy-admin/scripts/setup.php => 404 > INFO GET /webadmin/scripts/setup.php => 404 > INFO GET /sqlweb/scripts/setup.php => 404 > INFO GET /websql/scripts/setup.php => 404 > INFO GET /webdb/scripts/setup.php => 404 > INFO GET /mysqladmin/scripts/setup.php => 404 > INFO GET /mysql-admin/scripts/setup.php => 404 > > ------------------------------------------------------ > > http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447&dsMessageId=2663440 > ------------------------------------------------------ http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447&dsMessageId=2663470
