Hi,I am crawling a large website, which is our university's. From the logs and some grep'ing, I see that some pdf files were not crawled. Why could this happen? I'm crawling with -depth 100 -topN 5.
Regards,
Hi,I am crawling a large website, which is our university's. From the logs and some grep'ing, I see that some pdf files were not crawled. Why could this happen? I'm crawling with -depth 100 -topN 5.
Regards,