Thanks a bunch, unfortunately I have already done all of your suggestions, still no go, this is starting to drive me nuts.
I also added a blog now just to drive up the content and "change rate" of the site, we add articles on a frequent basis, still nothing. grrrrr. On Sep 17, 1:12 pm, nickmilon <[email protected]> wrote: > I am quite confident your problem has nothing to do with app engine. > Your robots.txt looks fine to me. > As you also mention there can be some legitimate reasons thatGoogle > can visit a page excluded by robots.txt > /datasource/ can be an html page oranyother resource for that > matter as far a browser is concerned.Mysmall piece of advice : sign in for G > webmasters tools and submit > a sitemap.xml, then use webmasters tools to see what is going on with > your site'sindexingand all seo related issues hm.. I have seen you > have already submitted the site. > By the way you have left DEBUG=True in your production server. > hope all this helped ;-) > > On Sep 16, 11:25 pm, "[email protected]" <[email protected]> wrote: > > > I asked this question already in theGoogleWebmasters group but I was > > recommended to ask it here as they seem to think it might be AppEngine > > related ... > > > "I have read the FAQs and checked for similar issues: YES > > >Mysite'sURL(web address) is:http://bptrack.appspot.com > > > Description (including timeline ofanychanges made): > > I have the following inmyrobot.txtfile: > > > User-agent: * > >Allow: / > >Disallow: /datasource/ > > > It has been that for over a month now yet when I look formysite on > >google(site:http://bptrack.appspot.com), > >itonlyfindshttp://bptrack.appspot.com/datasource/, exactly theURLI donotwant > > it to index. It doesn't findanyother URLs, including the main one > > (http://bptrack.appspot.com). Can anybody please explain to me how > > that is possible? Do I have a mistake inmyrobot.txtfile > > (eventhough I had it generated byGoogleWebmasters Tool)? > > > Cheers, > > Mark." > > > Through thegoogleWebsmasters Group I figured out that therobot.txt > >filemightnotactually prevent thatURLfrom being crawled and/or > > indexed, however,http://bptrack.appspot.com/datasource/isnoteven a > > webpage, so it is "empty" AND inaccessible so why would that one be > > indexed at all and none of the other pages which are accessible > > ("home" page, "about" and "contacts")? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
