[
https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446676#comment-17446676
]
Sebastian Nagel commented on NUTCH-2865:
----------------------------------------
Hi [~markus17], looks good and tests went successfully. I've made two changes
in
[6bb30c7|https://github.com/sebastian-nagel/nutch/commit/6bb30c719550f6d7b8efc127c052ed1a21816321]:
- add the Content-Type to metadata and text records. So, it's now:
-- metadata: {{WARC-Type: metadata\r\nContent-Type: application/json}}, cf.
[WARC "metadata"
record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#metadata]
-- text: {{WARC-Type: conversion\r\nContent-Type: text/plain}}, cf. [WARC
"conversion"
record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#conversion]
- apply code formats
If there are no objections I'll commit this.
> WARC exporter support for metadata and dropping empty responses
> ---------------------------------------------------------------
>
> Key: NUTCH-2865
> URL: https://issues.apache.org/jira/browse/NUTCH-2865
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.19
>
> Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch
>
>
> WARCExporter is a handy tool to dump the segments. Unfortunately it also
> emits WARC records for status' other than success of notmodified, which
> accounts for a decent number in each crawl cycle. It also doesn't emit parsed
> metadata or extracted text. It does now.
>
> This patch adds three switches:
> * -includeOnlySuccessfulResponses to only emit records of success or
> notmodified
> * -includeParseData to also emit parse metadata as WARC metadata record
> * -includeParseText to also emit extracted text as WARC metadata
> Both metadata objects are stored in the same WARC metadata record to save
> space.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
