[
https://issues.apache.org/jira/browse/IMPALA-9075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044664#comment-17044664
]
ASF subversion and git services commented on IMPALA-9075:
---------------------------------------------------------
Commit 571131fdc11acecf4c2003668dbccde0667efe07 in impala's branch
refs/heads/master from xiaomeng
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=571131f ]
IMPALA-9075: Add support for reading zstd text files
In this patch, we add support for reading zstd encoded text files.
This includes:
1. support reading zstd file written by Hive which uses streaming.
2. support reading zstd file compressed by standard zstd library which
uses block.
To support decompressing both formats, a function ProcessBlockStreaming
is added in zstd decompressor.
Testing done:
Added two backend tests:
1. streaming decompress test.
2. large data test for both block and streaming decompress.
Added two end to end tests:
1. hive and impala integration. For four compression codecs, write in
hive and read from impala.
2. zstd library and impala integration. Copy a zstd lib compressed file
to HDFS, and read from impala.
Change-Id: I2adce9fe00190558525fa5cd3d50cf5e0f0b0aa4
Reviewed-on: http://gerrit.cloudera.org:8080/15023
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add support for reading zstd text files
> ---------------------------------------
>
> Key: IMPALA-9075
> URL: https://issues.apache.org/jira/browse/IMPALA-9075
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 3.3.0
> Reporter: Andrew Sherman
> Assignee: Xiaomeng Zhang
> Priority: Critical
>
> IMPALA-8450 added support for zstd in parquet.
> We should also support support for reading zstd encoded text files.
> Another useful jira to look at is IMPALA-8549 (Add support for scanning
> DEFLATE text files)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
