I'm after assistance on formatting for the robots.txt file.
Basically, my default is
useragent: *
disallow: /lists
I don't want my link choked with robots indexing the thousands of
messages in my archive of slug list, etc
To run htdig, I had to change it to read
useragent: htdig
disallow
useragent: *
disallow: /lists
However, I really only want [EMAIL PROTECTED] to be able to run on
/lists.
Is there a way of specifying this?
--
Terry Collins {:-)}}} Ph(02) 4627 2186 Fax(02) 4628 7861
email: [EMAIL PROTECTED] www: http://www.woa.com.au
or [EMAIL PROTECTED]
WOA Computer Services <lan/wan, linux/unix, novell>
snail: PO Box 1047, Campbelltown, NSW 2560.
"People without trees are like fish without clean water"
--
SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/
More Info: http://slug.org.au/lists/listinfo/slug
