[
https://issues.apache.org/jira/browse/NUTCH-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347600#comment-14347600
]
Hudson commented on NUTCH-1949:
-------------------------------
SUCCESS: Integrated in Nutch-trunk #3001 (See
[https://builds.apache.org/job/Nutch-trunk/3001/])
NUTCH-1949 Dump out the Nutch data into the Common Crawl format (lewismc:
http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1664109)
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/ivy/ivy.xml
* /nutch/trunk/src/bin/nutch
* /nutch/trunk/src/java/org/apache/nutch/tools/AbstractCommonCrawlFormat.java
* /nutch/trunk/src/java/org/apache/nutch/tools/Benchmark.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlFormat.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlFormatFactory.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlFormatJackson.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlFormatJettinson.java
* /nutch/trunk/src/java/org/apache/nutch/tools/CommonCrawlFormatSimple.java
* /nutch/trunk/src/java/org/apache/nutch/tools/FileDumper.java
> Dump out the Nutch data into the Common Crawl format
> ----------------------------------------------------
>
> Key: NUTCH-1949
> URL: https://issues.apache.org/jira/browse/NUTCH-1949
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb, linkdb, storage, tool
> Reporter: Giuseppe Totaro
> Assignee: Lewis John McGibbney
> Fix For: 1.10
>
> Attachments: CommonCrawlDataDumper.pdf, CommonCrawlDataDumper.xlsx,
> CommonCrawlDataDumper_v02.pdf
>
>
> We are going to develop a {{CommonCrawlDataDumper.java}} class. The
> {{CommonCrawlDataDumper}} is a tool able to perfom the following steps:
> # deserialize the crawled data from Nutch
> # map serialized data on the proper JSON structure
> # serialize the data into [CBOR|http://cbor.io] format
> # optionally, compress the serialized data using {{gzip}}
> This tool has to be able to work with either single Nutch segments or
> directory including segments as input data.
> Thanks [~lewismc] and [~chrismattmann] for your great suggestions, support
> and code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
