://www.robotstxt.org/wc/norobots-rfc.txt.
[...]
Bye,
Cedric.
- Original Message -
From: Hack Kampbjørn [EMAIL PROTECTED]
To: Cédric Rosa [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Saturday, July 06, 2002 8:21 PM
Subject: Re: wget and meta name=robots content=noindex,nofollow
Cédric Rosa wrote
Hello Jakub,
try:
--user-agent=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
(with quotes)
Regards,
Cedric Rosa.
At 10:35 01/07/2002 +0100, you wrote:
Thanks for reply and sorry for my long response.
I was trying to use:
--user-agent=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
Hello Jason,
I'm working on a search engine project and I'm thinking about to use wget
as webcrawler.
I've just tested fwget and I'm not sure it is better than wget because it
was not updated since one year ...
I'm interested about any idea on this subject.
Cedric.
At 10:55 24/06/2002
Hello,
Is-it normal that wget saves web pages which contain meta name=robots
content=noindex ?
Or does wget considerate that it is not a search engine and respects only
the follow/nofollow rules ?
Or is-it a bug ? :)
Thanks.
Cedric.
this problem ?
Date: Fri, 21 Jun 2002 16:37:02 +0200
To: [EMAIL PROTECTED]
From: Cédric Rosa [EMAIL PROTECTED]
Subject: Bug with wget ? I need help.
Hello,
First, scuse my english but I'm french.
When I try with wget (v 1.8.1) to download an url which is behind a router,
the software wait for ever even
thanks for your help :)
I'm installing version 1.9 to check. I think this update may solve my
problem.
Cedric Rosa.
- Original Message -
From: Hack Kampbjørn [EMAIL PROTECTED]
To: Cédric Rosa [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Friday, June 21, 2002 7:27 PM
Subject: Re: Bug
