STOP Nutch without breaking the crawled data
--------------------------------------------
Key: NUTCH-866
URL: https://issues.apache.org/jira/browse/NUTCH-866
Project: Nutch
Issue Type: New Feature
Reporter: Pham Tuan Minh
Fix For: 2.0
How we can stop running nutch instance in local mode and in reducer mode
without breaking the crawled data?
For example, you push a list of site that take a long time to complete crawl;
then you want to stop nutch instance suddenly ...
- For local mode, I suggest as below
We create a stop.txt file in specific directory, then for a piece of time,
nutch instance will check whether this file existed or not; if existed, nutch
instance will stop itself normally
- For reducer mode, may we use zookeper to keep state of each instance?
Any other suggestion?
Thanks,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
