Does anyone know if there are any open source/free .NET spiders under
development? I am developing a custom search engine system, and don't want
to have to reinvent the wheel, but I would like have everything in .NET.
Thanks,
Erick
--
This message was sent by the Internet robots and spiders
Is there any value in saving the request and/or response strings generated
during an index? I can't think of any use for them, aside from statistical
analysis of the index, but I wanted to make sure.
Thanks,
Erick
I'm working on a multi language spider, and I've come to a point where I'm
not sure what assumption to make. If I download a page from a Chinese (for
example) server, there is a chance that
1. http header says a Chinese charset but page is in English
2. http header says nothing, meta tag says a
There are built in methods/objects for dealing with both cookies and http
authencation. It can even handle x509 cert.
see
HttpWebRequest.Credentials
HttpWebRequest.ClientCertificates
HttpWebRequest.CookieContainer
So if you wanted to simulate a session, you'd have to hold onto an instance of a
