Sorry, I've been out a bit and gotten behind on replying to some of these emails. A few thoughts below.
On Mon, Apr 20, 2009 at 3:33 PM, Mark Marijnissen <[email protected]> wrote: > > Hello Everybody, > > I was checking out what pages Google did index, and I found out Google > did pretty well! This is because a Wiki is full of links to eachother, > making it easy for Google to crawl. However, Google also indexed some > pages that do not exist anymore! However, since BoltWire generates a > "page does not exist" page instead of a 404 error, the page stays > indexed in google! > > Now this does not matter much for the everyday user, since this > useless page will never get high when somebody searches with google. > However, if somebody searches the site (http://www.google.nl/search? > q=site:http://markmarijnissen.ruhosting.nl/wiki/robot/) there are some > invalid results ("onderdelen" for example). These results could easily > be blocked by google, if the "missing page"-page contains the > following tag: > > <meta name="robots" content="noindex"> > > This tag tells the Google Webcrawler it should not index this page, so > the next time Google indexes this page, it will be removed. (I wonder > if google will ever reindex the page since no page links to it, but I > sure hope so). > > I would have added this tag myself to the "page missing" page, but I > do not know where to find this in the source code. So basically you want to add the noindex meta tag to a page if it doesn't exist? Is that correct? It should be easy enough to do. Please verify, then I'll add a line to the code. Unless anyone else has objections. I don't know what Google would do if you later added the page, but my guess is, it would reindex it next time through if the noindex tag is no longer there. > Best regards & enjoying your software, > > Mark Marijnissing > > > (As a side note: The indexing is a lot harder with for example a > wordpress blog, because the blog entries tend to get lost in archive > pages and dont have many links to eachother. Fortunaly, there is a > plugin that generates a Sitemaps XML file that lists all the URLs > google has to index. Google recognises this XML file. Sitemaps for > BoltWire still would be nice, but it should be created automatically > as the wiki changes - would be a nice plugin!.) If you could show me a sample of what the output is supposed to look like, I'll be happy to look into this. My guess is we could handle it very much like the rss plugin, which I've been working on lately. It would possibly involve a change to the core, but having that option available would be a nice option. Thanks for trying BoltWire. Glad you are enjoying it. Cheers, Dan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "BoltWire" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/boltwire?hl=en -~----------~----~----~----~------~----~------~--~---
