Zoltán Borók-Nagy created IMPALA-15715:
------------------------------------------
Summary: Support zstandard compression codec in the Avro scanner
Key: IMPALA-15715
URL: https://issues.apache.org/jira/browse/IMPALA-15715
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
Impala's Avro scanner only accepts the {{null}}, {{snappy}} and {{deflate}}
codecs (see {{HdfsAvroScanner::ReadFileHeader()}} in
{{be/src/exec/avro/hdfs-avro-scanner.cc}}). Files with {{avro.codec =
zstandard}} are rejected with "Unknown Avro compression codec: zstandard".
As a result, Impala can't read Iceberg tables with Avro data files that Trino
writes with default settings, because Trino's Iceberg connector uses ZSTD
compression by default ({{iceberg.compression-codec=ZSTD}}).
Steps to reproduce, with Trino and Impala sharing the same Hive Metastore:
In Trino:
{code:sql}
CREATE TABLE iceberg.default.avro_zstd (i INT, s VARCHAR) WITH (format =
'AVRO');
INSERT INTO iceberg.default.avro_zstd VALUES (1, 'a'), (2, 'b'), (3, 'c');
SELECT * FROM iceberg.default.avro_zstd; -- 3 rows
{code}
In Impala (master f990ce4811):
{code:sql}
SELECT * FROM default.avro_zstd; -- 0 rows
{code}
{noformat}
WARNINGS: Unknown Avro compression codec: zstandard
{noformat}
With the default {{abort_on_error=false}} the error is only a warning and the
file is skipped, so the query silently returns incomplete results.
Implementation notes:
* {{util/decompress.h}} already has a {{ZstandardDecompressor}}, but it
requires a preallocated output buffer ({{MaxOutputLen()}} returns -1), while
the Avro scanner calls {{ProcessBlock()}} with {{output_preallocated=false}}.
Avro blocks don't record the decompressed size, and the zstd frame header may
not contain it either, so the scanner needs to handle an unknown output size
(e.g. use {{ZSTD_getFrameContentSize()}} when available, otherwise decompress
in streaming mode).
* The other optional codecs of the Avro spec ({{bzip2}}, {{xz}}) aren't
supported either.
Workaround: write the Avro files with snappy or deflate (e.g.
{{iceberg.compression-codec=SNAPPY}} in the Trino catalog config, or the
{{compression_codec}} catalog session property), or use Parquet.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
