[
https://issues.apache.org/jira/browse/HADOOP-13126?focusedWorklogId=784538&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784538
]
ASF GitHub Bot logged work on HADOOP-13126:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jun/22 11:07
Start Date: 24/Jun/22 11:07
Worklog Time Spent: 10m
Work Description: martin-g commented on PR #2723:
URL: https://github.com/apache/hadoop/pull/2723#issuecomment-1165466276
What is the policy about using `Object#finalize()` in the `compress` package
?
The last check fails because of 4 new usages of `finalize` in the
Brotli[De]Compressor` classes.
```
grep -rnHi finalize
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/brotli/BrotliCompressor.java:251:
protected void finalize() throws Throwable {
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/brotli/BrotliCompressor.java:252:
super.finalize();
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/brotli/BrotliDecompressor.java:173:
protected void finalize() throws Throwable {
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/brotli/BrotliDecompressor.java:174:
super.finalize();
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java:711:
protected void finalize() throws Throwable {
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java:713:
super.finalize();
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java:294:
protected void finalize() {
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardDecompressor.java:249:
protected void finalize() {
```
Issue Time Tracking
-------------------
Worklog Id: (was: 784538)
Time Spent: 50m (was: 40m)
> Add Brotli compression codec
> ----------------------------
>
> Key: HADOOP-13126
> URL: https://issues.apache.org/jira/browse/HADOOP-13126
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.7.2
> Reporter: Ryan Blue
> Assignee: Ryan Blue
> Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch,
> HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I've been testing [Brotli|https://github.com/google/brotli/], a new
> compression library based on LZ77 from Google. Google's [brotli
> benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf]
> look really good and we're also seeing a significant improvement in
> compression size, compression speed, or both.
> {code:title=Brotli preliminary test results}
> [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet
> --compression-codec snappy --overwrite
> real 1m17.106s
> user 1m30.804s
> sys 0m4.404s
> [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet
> --compression-codec brotli --overwrite
> real 1m16.640s
> user 1m24.244s
> sys 0m6.412s
> [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet
> --compression-codec gzip --overwrite
> real 3m39.496s
> user 3m48.736s
> sys 0m3.880s
> [blue@work Downloads]$ ls -l
> -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet
> -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet
> -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet
> {code}
> Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9.
> Another test resulted in a slightly larger Brotli file than gzip produced,
> but Brotli was 4x faster. I'd like to get this compression codec into Hadoop.
> [Brotli is licensed with the MIT
> license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI
> library jbrotli is
> ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
