[WSG] question - follow, index meta tag
Reading the archived discussions on meta tags, I couldn't find the answer to this question, but I did see people saying the tags should appear on every page that differs in any way from other pages at a site. If I have the following on my index page, do I need to repeat it on every page at my site? Doesn't this tag appearing once send the robots forward to all the other pages? meta name=robots content=index, follow / Thanks in advance for any advice. Barry Cranmer from the east coast of the US ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
Re: [WSG] question - follow, index meta tag
Greetings from Northeast ;) At 03:14 PM 12/13/2004 -0500, you wrote: If I have the following on my index page, do I need to repeat it on every page at my site? Doesn't this tag appearing once send the robots forward to all the other pages? meta name=robots content=index, follow / Per se you do not need this meta tag at all, since the default value is just this, however, if search engines are concerned, a couple of other lists I'm on suggests that they may be necessary to put back in for some of the new spiders that value well built pages. Personally after reading that I include it on the front page of every site, since the front is carrying most of the SES (search engine spiders) weight just for good measure, I mean the extra few bytes on the page will not do any difference for download speed purposes so.. what could be hurt? HTH Regards ~Veine Veine K Vikberg http://www.vikberg.net Professional Web Guru
Re: [WSG] question - follow, index meta tag
Good afternoon; At 01:11 PM 12/13/2004 -0800, you wrote: I know I should read about Robots from the Robot FAQ web site. However, I am a little pressed for time right now. What do I need to web sites to stop Robots reading my web sites I maintain? Thank you. If I get your question right you want none to visit and index, if that is the case, then in robots.txt you put the following: User-agent: * Disallow: / HTH Regards ~Veine Veine K Vikberg http://www.vikberg.net Professional Web Guru
Re: [WSG] question - follow, index meta tag
The Man With His Guide Dog At The Tent Store wrote: I know I should read about Robots from the Robot FAQ web site. However, I am a little pressed for time right now. What do I need to web sites to stop Robots reading my web sites I maintain? Thank you. in robots.txt User-agent: * Disallow: /www.mysite.com/ but, it has its limitations. using an .htaccess file is the only real way to stop a robot. http://www.website-promotion-ranking-services.com/tutorials/27.htm ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
Re: [WSG] question - follow, index meta tag
AFIAK it's safe to assume that robots will index and follow by default, so this tag may be redundant. Read more about robots here: http://www.robotstxt.org/wc/faq.html Terrence Wood. On 2004-12-14 9:14 AM, Barry Cranmer wrote: If I have the following on my index page, do I need to repeat it on every page at my site? Doesn't this tag appearing once send the robots forward to all the other pages? meta name=robots content=index, follow / -- You know you've achieved perfection in design, not when you have nothing more to add, but when you have nothing more to take away. -Antoine de Saint-Exupery ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
Re: [WSG] question - follow, index meta tag
If I have the following on my index page, do I need to repeat it on every page at my site? Doesn't this tag appearing once send the robots forward to all the other pages? meta name=robots content=index, follow / A robots.txt file is a better option for controlling site-wide search engine behavior. http://www.motive.co.nz/glossary/robots.php For a search engine to crawl a site the site must be well-linked, i.e. if a webpage is posted but has no incoming links it won't be 'found' by the spider (unless it is registered separately). (See reference links and additional related glossary terms for more info on search engines, meta tags etc.) -- Andy Kirkwood | Creative Director MOTIVE | web.design.integrity http://www.motive.co.nz/ ph: +64 4 3 800 800 fx: +64 4 970 9693 mob: 021 369 693 93 Rintoul St, Newtown PO Box 7150, Wellington South, New Zealand ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
Re: [WSG] question - follow, index meta tag
If I have the following on my index page, do I need to repeat it on every page at my site? Doesn't this tag appearing once send the robots forward to all the other pages? meta name=robots content=index, follow / You don't need that at all. Index, follow is default behaviour and won't override robots.txt anyway. -- regards, Kornel Lesiski ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
Re: [WSG] question - follow, index meta tag
I know I should read about Robots from the Robot FAQ web site. However, I am a little pressed for time right now. What do I need to web sites to stop Robots reading my web sites I maintain? Thank you. Create a text file and name it 'robots.txt' Paste the following code, save and upload to the root directory of each site you want to ban search engines from indexing: User-agent: * Disallow: / The first line identifies the search engine spiders (user-agents) the directive applies to (all) The second line specifies the directories and files to be excluded, (all-from the root directory) -- Andy Kirkwood | Creative Director MOTIVE | web.design.integrity http://www.motive.co.nz/ ph: +64 4 3 800 800 fx: +64 4 970 9693 mob: 021 369 693 93 Rintoul St, Newtown PO Box 7150, Wellington South, New Zealand ** The discussion list for http://webstandardsgroup.org/ See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list getting help **
