[jira] [Commented] (HADOOP-13126) Add Brotli compression codec
[ https://issues.apache.org/jira/browse/HADOOP-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558037#comment-17558037 ] Martin Tzvetanov Grigorov commented on HADOOP-13126: [~ste...@apache.org] Rebased my branch to latest trunk. Hopefully Hadoop QA will like it this time. > Add Brotli compression codec > > > Key: HADOOP-13126 > URL: https://issues.apache.org/jira/browse/HADOOP-13126 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch, > HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch > > Time Spent: 20m > Remaining Estimate: 0h > > I've been testing [Brotli|https://github.com/google/brotli/], a new > compression library based on LZ77 from Google. Google's [brotli > benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf] > look really good and we're also seeing a significant improvement in > compression size, compression speed, or both. > {code:title=Brotli preliminary test results} > [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet > --compression-codec snappy --overwrite > real1m17.106s > user1m30.804s > sys 0m4.404s > [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet > --compression-codec brotli --overwrite > real1m16.640s > user1m24.244s > sys 0m6.412s > [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet > --compression-codec gzip --overwrite > real3m39.496s > user3m48.736s > sys 0m3.880s > [blue@work Downloads]$ ls -l > -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet > -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet > -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet > {code} > Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. > Another test resulted in a slightly larger Brotli file than gzip produced, > but Brotli was 4x faster. I'd like to get this compression codec into Hadoop. > [Brotli is licensed with the MIT > license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI > library jbrotli is > ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE]. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13126) Add Brotli compression codec
[ https://issues.apache.org/jira/browse/HADOOP-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291446#comment-17291446 ] Martin Tzvetanov Grigorov edited comment on HADOOP-13126 at 2/26/21, 7:36 AM: -- I am not sure why Hadoop QA says that the Pull Request does not apply to trunk. GitHub UI says: {code:java} This branch has no conflicts with the base branch Only those with write access to this repository can merge pull requests. {code} *Update*: I think it talks about the latest .patch, not the Pull Request. was (Author: mgrigorov): I am not sure why Hadoop QA says that the Pull Request does not apply to trunk. GitHub UI says: {code:java} This branch has no conflicts with the base branch Only those with write access to this repository can merge pull requests. {code} > Add Brotli compression codec > > > Key: HADOOP-13126 > URL: https://issues.apache.org/jira/browse/HADOOP-13126 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch, > HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > I've been testing [Brotli|https://github.com/google/brotli/], a new > compression library based on LZ77 from Google. Google's [brotli > benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf] > look really good and we're also seeing a significant improvement in > compression size, compression speed, or both. > {code:title=Brotli preliminary test results} > [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet > --compression-codec snappy --overwrite > real1m17.106s > user1m30.804s > sys 0m4.404s > [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet > --compression-codec brotli --overwrite > real1m16.640s > user1m24.244s > sys 0m6.412s > [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet > --compression-codec gzip --overwrite > real3m39.496s > user3m48.736s > sys 0m3.880s > [blue@work Downloads]$ ls -l > -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet > -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet > -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet > {code} > Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. > Another test resulted in a slightly larger Brotli file than gzip produced, > but Brotli was 4x faster. I'd like to get this compression codec into Hadoop. > [Brotli is licensed with the MIT > license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI > library jbrotli is > ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13126) Add Brotli compression codec
[ https://issues.apache.org/jira/browse/HADOOP-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291446#comment-17291446 ] Martin Tzvetanov Grigorov commented on HADOOP-13126: I am not sure why Hadoop QA says that the Pull Request does not apply to trunk. GitHub UI says: {code:java} This branch has no conflicts with the base branch Only those with write access to this repository can merge pull requests. {code} > Add Brotli compression codec > > > Key: HADOOP-13126 > URL: https://issues.apache.org/jira/browse/HADOOP-13126 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.7.2 >Reporter: Ryan Blue >Assignee: Ryan Blue >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch, > HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > I've been testing [Brotli|https://github.com/google/brotli/], a new > compression library based on LZ77 from Google. Google's [brotli > benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf] > look really good and we're also seeing a significant improvement in > compression size, compression speed, or both. > {code:title=Brotli preliminary test results} > [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet > --compression-codec snappy --overwrite > real1m17.106s > user1m30.804s > sys 0m4.404s > [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet > --compression-codec brotli --overwrite > real1m16.640s > user1m24.244s > sys 0m6.412s > [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet > --compression-codec gzip --overwrite > real3m39.496s > user3m48.736s > sys 0m3.880s > [blue@work Downloads]$ ls -l > -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet > -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet > -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet > {code} > Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. > Another test resulted in a slightly larger Brotli file than gzip produced, > but Brotli was 4x faster. I'd like to get this compression codec into Hadoop. > [Brotli is licensed with the MIT > license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI > library jbrotli is > ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org