Hi,

I am crawling a large website, which is our university's. From the logs and some grep'ing, I see that some pdf files were not crawled. Why could this happen? I'm crawling with -depth 100 -topN 5.

Regards,

Reply via email to