Re: how do fetch wait times work?

2018-04-11 Thread Sebastian Nagel
Hi Fred, Nutch does nothing "proactively", the crawl jobs must be explicitly called. But you need no special command: - let's say the you didn't change the defaults and db.fetch.interval.default == 30 days - if you launch bin/crawl one month later, all pages are refetched, and optionally

how do fetch wait times work?

2018-04-09 Thread Fred Zimmerman
When I run bin/crawl once and it generates a segment list with a bunch of fetch dates in the future, does nutch proactively run those fetches on those future dates, or do I have to do something to make that happen?