[
https://issues.apache.org/jira/browse/NUTCH-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513044
]
Doğacan Güney commented on NUTCH-506:
-------------------------------------
If there are no objections, I am going to commit this one.
Just to get more comments, here is a break-down of what this patch does:
* Remove all compression code from nutch. This means no more
writeCompressedString-s or writeCompressedStringArrays, also no more
CompressedWritable-s. All changes are done in a backward compatible manner.
Also after this change, Content's version is -1 and new changes should
*decrease* that number. See NUTCH-392 for more details.
* Respect io.seqfile.compression.type setting for all structures except
ParseText. ParseText is always compressed as RECORD. Also for some reason
crawl_generate is not compressed.
Why are we doing this? Because hadoop can efficiently (both in space and in
time) compress these structures for us. I have done some tests with different
compression settings in NUTCH-392 and BLOCK compression really does a
difference. I think for a large enough crawl, overall space savings will be
around %20 - %40. Note that this is basically for free (there may even be a
small performance gain) if you are using hadoop's native libraries.
> Nutch should delegate compression to Hadoop
> -------------------------------------------
>
> Key: NUTCH-506
> URL: https://issues.apache.org/jira/browse/NUTCH-506
> Project: Nutch
> Issue Type: Improvement
> Reporter: Doğacan Güney
> Fix For: 1.0.0
>
> Attachments: compress.patch, NUTCH-506.patch
>
>
> Some data structures within nutch (such as Content, ParseText) handle their
> own compression. We should delegate all compressions to Hadoop.
> Also, nutch should respect io.seqfile.compression.type setting. Currently
> even if io.seqfile.compression.type is BLOCK or RECORD, nutch overrides it
> for some structures and sets it to NONE (However, IMO, ParseText should
> always be compressed as RECORD because of performance reasons).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
