Sebastian Nagel created NUTCH-2628:
--------------------------------------
Summary: Fetcher: optionally generate signature of unparsed content
Key: NUTCH-2628
URL: https://issues.apache.org/jira/browse/NUTCH-2628
Project: Nutch
Issue Type: Improvement
Components: fetcher
Reporter: Sebastian Nagel
Fix For: 1.16
To generate a document signature (MD5 digest) of the binary content requires
that documents are parsed during the parse or fetch step. The signature is
required for deduplication and detection of unmodified content and should be
always available, also in a workflow which does not require that documents are
parsed, e.g., because HTML content is exported to WARC files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
