[for those robots.txt fans] White House site prevents Iraq material being archived http://www.theage.com.au/articles/2003/10/28/1067233141495.html By Sam Varghese October 28, 2003
The White House website <http://www.whitehouse.gov/> effectively prevents search engines indexing and archiving material on the site related to Iraq. The directories on a site which can be searched by the bots sent out by search engines can be limited by means of a file called robots.txt, which resides in the root directory of a site. Adding a directory to robots.txt ensures that nothing in that folder will ever show up in a search and will never be archived by search sites. The White House's robots.txt <http://www.whitehouse.gov/robots.txt> file lists a huge number of directories all related to Iraq. The Democrat National Committee blog<http://www.democrats.org/blog/display/00010130.html> claims a change in the robots.txt file took place sometime between April and October this year. Earlier this year, the White House changed pages on its website which claimed that "combat" was over in Iraq; these pages were changed to say "major combat." These changes were noticed and proved by readers<http://www.differentstrings.info/archives/002813.html>because Google had archived them before the changes were made. With the new robots.txt file, any future changes will be extremely difficult to spot - and even more difficult to prove.
