Hi You are correct: if a page does not exists, the noindex meta-tag should be added to the page. The output is pretty simple. It is a XML file according to the Sitemaps protocol: (See: http://sitemaps.org/protocol.php ) >From sitemaps.org:
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";> <url> <loc>http://www.example.com/</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> <url> <loc>http://www.example.com/catalog? item=12&desc=vacation_hawaii</loc> <changefreq>weekly</changefreq> </url> <url> <loc>http://www.example.com/catalog? item=73&desc=vacation_new_zealand</loc> <lastmod>2004-12-23</lastmod> <changefreq>weekly</changefreq> </url> <url> <loc>http://www.example.com/catalog? item=74&desc=vacation_newfoundland</loc> <lastmod>2004-12-23T18:00:15+00:00</lastmod> <priority>0.3</priority> </url> <url> <loc>http://www.example.com/catalog? item=83&desc=vacation_usa</loc> <lastmod>2004-11-23</lastmod> </url> </urlset> The main advantage of a Sitemap plugin would be that you could filter out all the action-pages. (For example": /index.php?p=action.register" and "index.php?p=microcontroller&action=search"). You can see this if you google my wiki: "http://www.google.nl/search?q=site:http:// markmarijnissen.ruhosting.nl/wiki/robot/" However, since these action pages are not really useful, the will never end up high in google search results, so it does not really matter. As a side advantage, you can set how important pages are in your site (priority), and how often your pages change (this helps for google to index) If the plugin were available, i would install it, but I would not program it myself (and I often modify code or write plugins for things like wordpress or boltwire)- so it has a low priority. I do not know how much work it is, but I would not bother unless there are more people interested. cheers, Mark On 27 apr, 16:56, The Editor <[email protected]> wrote: > Sorry, I've been out a bit and gotten behind on replying to some of > these emails. A few thoughts below. > > On Mon, Apr 20, 2009 at 3:33 PM, Mark Marijnissen > > > > <[email protected]> wrote: > > > Hello Everybody, > > > I was checking out what pages Google did index, and I found out Google > > did pretty well! This is because a Wiki is full of links to eachother, > > making it easy for Google to crawl. However, Google also indexed some > > pages that do not exist anymore! However, since BoltWire generates a > > "page does not exist" page instead of a 404 error, the page stays > > indexed in google! > > > Now this does not matter much for the everyday user, since this > > useless page will never get high when somebody searches with google. > > However, if somebody searches the site (http://www.google.nl/search? > > q=site:http://markmarijnissen.ruhosting.nl/wiki/robot/) there are some > > invalid results ("onderdelen" for example). These results could easily > > be blocked by google, if the "missing page"-page contains the > > following tag: > > > <meta name="robots" content="noindex"> > > > This tag tells the Google Webcrawler it should not index this page, so > > the next time Google indexes this page, it will be removed. (I wonder > > if google will ever reindex the page since no page links to it, but I > > sure hope so). > > > I would have added this tag myself to the "page missing" page, but I > > do not know where to find this in the source code. > > So basically you want to add the noindex meta tag to a page if it > doesn't exist? Is that correct? It should be easy enough to do. > Please verify, then I'll add a line to the code. Unless anyone else > has objections. > > I don't know what Google would do if you later added the page, but my > guess is, it would reindex it next time through if the noindex tag is > no longer there. > > > Best regards & enjoying your software, > > > Mark Marijnissing > > > (As a side note: The indexing is a lot harder with for example a > > wordpress blog, because the blog entries tend to get lost in archive > > pages and dont have many links to eachother. Fortunaly, there is a > > plugin that generates a Sitemaps XML file that lists all the URLs > > google has to index. Google recognises this XML file. Sitemaps for > > BoltWire still would be nice, but it should be created automatically > > as the wiki changes - would be a nice plugin!.) > > If you could show me a sample of what the output is supposed to look > like, I'll be happy to look into this. My guess is we could handle it > very much like the rss plugin, which I've been working on lately. It > would possibly involve a change to the core, but having that option > available would be a nice option. > > Thanks for trying BoltWire. Glad you are enjoying it. > > Cheers, > Dan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "BoltWire" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/boltwire?hl=en -~----------~----~----~----~------~----~------~--~---
