Sandeep,
have a look at the implementation of the libwww robot:
http://www.w3.org/Robot/
It is a highly portable C library and is IMHO a good starting point.
Regards,
Christian
Sandeep Mehta wrote:
[snip]
> work and how they crawl the web etc. so just help me
> by providing information about web robot.
[snip]
--
CSK Computer Services Kaisha Deutschland GmbH
Christian Brunner | Tel: (+49) 69 / 50952-234
Holzhausenstrasse 44 | Fax: (+49) 69 / 50952-299
60322 Frankfurt a. M. | E-Mail: [EMAIL PROTECTED]
