hubgeter opened a new pull request, #68854:
URL: https://github.com/apache/doris/pull/68854
### What problem does this PR solve?
Issue Number: N/A
Related PR: #64723
Problem Summary:
Doris failed to write Parquet files into Iceberg/Hive tables whose Parquet
codec is GZIP:
```
INSERT INTO iceberg_tbl VALUES (1, 'a');
-- errCode = 2, detailMessage = [INTERNAL_ERROR]Unsupported compress type GZ
with parquet
```
`write.parquet.compression-codec = gzip` (the Iceberg table default before
1.4, and still a
valid value) is read by the FE (`IcebergWritePlanProvider.getFileCompress` ->
`toTFileCompressType`) and mapped to `TFileCompressType.GZ`, but the BE
Parquet writers only
handled `PLAIN/SNAPPY/ZSTD/LZ4` and rejected everything else. The
position/equality delete-file
writer had the same gap, so merge-on-read `DELETE`/`UPDATE` failed as well.
Spark and Trino
wrote the very same table successfully, so this was a Doris-only failure.
Hive tables with `parquet.compression = GZIP` reach the same BE writer
through
`HiveSinkHelper.getTFileCompressType`, and `CREATE TABLE ... ENGINE=hive`
rejected `gzip` in
the Parquet compression allow-list before the write could even be planned.
What this PR changes:
- BE: map `TFileCompressType.GZ` -> `TParquetCompressionType.GZIP` in the
Iceberg Parquet
data-file writer, the Iceberg delete-file writer and the Hive Parquet
writer. GZIP is the
standard Parquet codec for gzip and is exactly what Spark/Trino write for
that table
property; the BE Arrow build already enables zlib (`ARROW_WITH_ZLIB`), so
the codec is
honoured instead of rejected, and the resulting files stay readable by
Doris, Spark and Trino.
- FE: accept `gzip` in `HmsWriteConverter.SUPPORTED_PARQUET_COMPRESSIONS` so
`CREATE TABLE ... ENGINE=hive PROPERTIES ('compression' = 'gzip')` is
allowed for Parquet tables.
Tests:
-
`regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_parquet_compression.groovy`:
the previous case asserted the GZIP rejection. It now checks INSERT + read
back +
`parquet_meta()` codec == `GZIP`, for both the data file and a
merge-on-read position-delete file.
- `regression-test/suites/external_table_p0/hive/ddl/test_hive_ddl.groovy`:
adds `gzip` to the
Parquet compression matrix and verifies the insert and the written file's
codec. ORC has no
gzip codec (it spells it `zlib`), so gzip is skipped for ORC.
- FE unit tests: `HmsWriteConverterTest.testExplicitParquetGzipCompression`
and the Parquet gzip
codec mapping in `HiveWritePlanProviderTest.planWriteCompressionPerFormat`.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change --> Iceberg/Hive Parquet
writes with GZIP no longer fail with `Unsupported compress type GZ with
parquet`; the file is written with the standard Parquet GZIP codec, the same
codec Spark/Trino write for that table property.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
