I added sitemaps that show all the pages on my appengine site. The sitemaps are crawled by the googlebot with monotonous regularity. However, the actual pages are pulled much less often. Also, something happened at one point and many of my pages fell out of the index. I was up to almost 10,000 indexed pages, and then it dropped to 1,500. Since then, I now record the number of indexed pages (according to a search for site:mysite.appspot.com). Over the last month (which is when I started recording daily numbers):
2009-09-27 11:45:01 1550 2009-10-08 03:27:01 1310 2009-10-11 03:27:01 1170 2009-10-17 03:27:01 1370 2009-10-22 03:27:00 1640 2009-10-27 03:27:01 2000 These numbers don't correspond with the numbers in the webmaster tools. The webmaster tools tells me that over the last three months, the googlebot has downloaded 1000 pages per day. I'm wondering if these pages are actually just my sitemaps.... Also, the sitemaps page says: Sitemap stats Total URLs: 161,845 Indexed URLs: 10,155 It could be that the site:zzzzz.appengine.com query on the main google index is broken and doesn't return the correct number of indexed URLs. Anyway, I'm not sure how much of this is relevant to your site..... Philip On Oct 25, 8:33 pm, klong86 <[email protected]> wrote: > I have noticed that googlebot crawls my app-engine site extremely > slow. I've been monitoring this with the Google Webmaster Tools. > > I can't change my crawl rate. It says: > > "Your site has been assigned special crawl rate settings. You > will not be able to change the crawl rate." > > From reading articles on the web, it appears that google controls the > rate for large sites like blogger.com. I'm guessing that googlebot has > determined that this site is hosted on app engine and slowed the rate > (even with a custom domain name). > > The site was previously written in ASP and had no problems in getting > googlebot to crawl the site quickly. > > The site has 11,000 pages and only 5000 have been crawled by > googlebot. The site has been on app engine for 2 months. > > Has anyone else ran into this same problem? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
