GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21837
[SPARK-24881][SQL] New Avro options - compression and compressionLevel
## What changes were proposed in this pull request?
In the PR, I added new options for Avro datasource - `compression` and
`compressionLevel`. The first option allows to specify compression codec for
saved Avro files. This option is similar to `compression` option in another
datasources like `JSON` and `CSV`. The second option specifies a level of
compression - `1` fast compression, `9` for the best compression. In such way,
valid range for compression level is `[1..9]`. For now, only `deflate` codec
take the option into account.
For both options there are SQL configs `spark.sql.avro.compression.codec`
and `spark.sql.avro.deflate.level`. In the PR, I put the configs into
`SQLConf`. If new options are not specified by an user, those SQL configs are
taken into account.
## How was this patch tested?
I added new tests for the options
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 avro-compression
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21837.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21837
----
commit 311482ab379446c631d642e594193d2e7f1ed442
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-21T21:51:08Z
Adding compression and compression level options
commit 0caa0f09e5b4a9f2f1a383f29820904783a65f02
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-21T21:56:29Z
Adding a test for new options
commit c5802dffd6b99832ce5abc826334f9b788608054
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-22T08:47:30Z
Ticket number is added to test title.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
