Hi all, has anyone suggestions how I could solve following task:
I want to crawl a sub-domain of our network completely. I always did it by multiple fetch / parse / update cycles manually. After a few cycles I checked if there are unfetched pages in the crawldb. If so, I started the cycle over again. I repeated that until no new pages were discovered. But that is annoying me and that is why I am looking for a way to do this steps automatic until no unfetched pages are left.
Any ideas?
