[ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524323#comment-15524323 ]
churro morales commented on HADOOP-13578: ----------------------------------------- [~jlowe] Thank you for taking a look. bq. The BUILDING.txt addition must have been copied from snappy, since a lot of it still refers to snappy. Yes did a copy and paste with the Building.txt, will take a look. bq. There's a TODO in the code that should get sorted out: Yes I think I have it figured out, had to read up a bit :). Since we empty compress the entire buffer at once, that means it's transformed into a single ZStandard Frame. So the overhead is a Magic # (4 bytes) + Header (14 bytes max) + Checksum (optional 4 bytes). So I believe that can be a constant of 22 Bytes because of the way we compress in hadoop. I'll update that as well. bq. Shouldn't this come from IO_COMPRESSION_CODEC_ZSTD_LEVEL_DEFAULT that's already in the config keys? Correct!, will change that back to 3. Also will use the constant. I'll incorporate your fixes. Comments much appreciated. > Add Codec for ZStandard Compression > ----------------------------------- > > Key: HADOOP-13578 > URL: https://issues.apache.org/jira/browse/HADOOP-13578 > Project: Hadoop Common > Issue Type: New Feature > Reporter: churro morales > Assignee: churro morales > Attachments: HADOOP-13578.patch > > > ZStandard: https://github.com/facebook/zstd has been used in production for 6 > months by facebook now. v1.0 was recently released. Create a codec for this > library. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org