dongjoon-hyun opened a new pull request #32051:
URL: https://github.com/apache/spark/pull/32051
### What changes were proposed in this pull request?
This PR aims to add `zstd` codec names in the Spark generated ORC file names
for consistency.
### Why are the changes needed?
Like the other ORC supported codecs, we had better have `zstd` in the Spark
generated ORC file names.
**SNAPPY**
```
scala> spark.range(10).repartition(1).write.option("compression",
"snappy").orc("/tmp/snappy")
$ ls -al /tmp/snappy
total 24
drwxr-xr-x 6 dongjoon wheel 192 Apr 4 12:17 .
drwxrwxrwt 14 root wheel 448 Apr 4 12:17 ..
-rw-r--r-- 1 dongjoon wheel 8 Apr 4 12:17 ._SUCCESS.crc
-rw-r--r-- 1 dongjoon wheel 12 Apr 4 12:17
.part-00000-833bb7ad-d1e1-48cc-9719-07b2d594aa4c-c000.snappy.orc.crc
-rw-r--r-- 1 dongjoon wheel 0 Apr 4 12:17 _SUCCESS
-rw-r--r-- 1 dongjoon wheel 231 Apr 4 12:17
part-00000-833bb7ad-d1e1-48cc-9719-07b2d594aa4c-c000.snappy.orc
```
**ZSTD (AS-IS)**
```
scala> spark.range(10).repartition(1).write.option("compression",
"zstd").orc("/tmp/zstd")
$ ls -al /tmp/zstd
total 24
drwxr-xr-x 6 dongjoon wheel 192 Apr 4 12:17 .
drwxrwxrwt 14 root wheel 448 Apr 4 12:17 ..
-rw-r--r-- 1 dongjoon wheel 8 Apr 4 12:17 ._SUCCESS.crc
-rw-r--r-- 1 dongjoon wheel 12 Apr 4 12:17
.part-00000-2f403ce9-7314-4db5-bca3-b1c1dd83335f-c000.orc.crc
-rw-r--r-- 1 dongjoon wheel 0 Apr 4 12:17 _SUCCESS
-rw-r--r-- 1 dongjoon wheel 231 Apr 4 12:17
part-00000-2f403ce9-7314-4db5-bca3-b1c1dd83335f-c000.orc
```
**ZSTD (After this PR)**
```
scala> spark.range(10).repartition(1).write.option("compression",
"zstd").orc("/tmp/zstd_new")
$ ls -al /tmp/zstd_new
total 24
drwxr-xr-x 6 dongjoon wheel 192 Apr 4 12:28 .
drwxrwxrwt 15 root wheel 480 Apr 4 12:28 ..
-rw-r--r-- 1 dongjoon wheel 8 Apr 4 12:28 ._SUCCESS.crc
-rw-r--r-- 1 dongjoon wheel 12 Apr 4 12:28
.part-00000-49d57329-7196-4caf-839c-4251c876e26b-c000.zstd.orc.crc
-rw-r--r-- 1 dongjoon wheel 0 Apr 4 12:28 _SUCCESS
-rw-r--r-- 1 dongjoon wheel 231 Apr 4 12:28
part-00000-49d57329-7196-4caf-839c-4251c876e26b-c000.zstd.orc
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with the updated UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
