Hey Mark, I read about this on a design blog recently while developing a clients mobile site and it had an article relating to things like this.
http://perishablepress.com/press/2010/04/26/stop-404-requests-for-mobile-versions-of-your-site/ According to the article, these types of bots just spider your site for pages listings of varied sorts and are attempting to harvest date from them. The article also shows a couple .htaccess techniques to stop 404 requests like these Hope this helps. - Brandtley McMinn http://giggleboxstudios.net On Jun 3, 5:29 pm, Mark Phillip <[email protected]> wrote: > Evening folks, > > I have pretty high expectations for the Refresh Austin list whenever I have > a tough question, but I might have found one stump-worthy. > > A couple months ago I started seeing requests in my web server access log > for "/ombudsman". I don't have an Ombudsman page, so it returned a 404. > Digging a little deeper, the same IP was repeatedly searching for the same > set of non-existent pages on my site: > > /about/privacypolicy.html > /about/termsofuse.html > /audiohelp/progstream.html > /blogs > /corrections > /email > /help > /help/communityfaq.html > /music > /ombudsman > /podcast > > After a bit more digging, I realized that it wasn't coming from just one IP > address. Turns out there are dozens of IP addresses all requesting the same > non-existent URLs. Each IP is scattered across the globe without any common > thread. The only user-agent listed in each request is a member of the > "Java/1.6.0" family. > > I am 100% stumped on this one. All Googling for community-sourced > Java-based search spiders comes up completely empty. > > Any thoughts? Solve this and I'll buy you a beer on Tuesday. > > Thanks, > Markhttp://markphillip.com -- Our Web site: http://www.RefreshAustin.org/ You received this message because you are subscribed to the Google Groups "Refresh Austin" group. [ Posting ] To post to this group, send email to [email protected] Job-related postings should follow http://tr.im/refreshaustinjobspolicy We do not accept job posts from recruiters. [ Unsubscribe ] To unsubscribe from this group, send email to [email protected] [ More Info ] For more options, visit this group at http://groups.google.com/group/Refresh-Austin
