andygrove opened a new issue, #7008:
URL: https://github.com/apache/datafusion-comet/issues/7008
### Describe the bug
The native Iceberg write gate admits `write.parquet.compression-codec=none`,
and the native writer writes the table uncompressed. iceberg-java rejects that
value. `Parquet.WriteBuilder` resolves the codec with
`CompressionCodecName.valueOf(codec.toUpperCase(Locale.ENGLISH))`, and
parquet-mr's `CompressionCodecName` has `UNCOMPRESSED` but no `NONE`, so the
write fails with `Unsupported compression codec: none`. With native Iceberg
writes enabled, a write that fails without Comet succeeds with it.
The mapping is deliberate. `IcebergWriteProtoTranslation.resolveCompression`
maps `none` to uncompressed, its scaladoc says Iceberg accepts `none`, and
`IcebergWriteProtoTranslationSuite` pins the mapping. `none` is ORC's spelling.
Iceberg documents `uncompressed` for Parquet.
Three more settings pass the gate because it trims or lowercases them, while
iceberg-java reads them as written:
- The codec itself. `resolveCompression` trims and lowercases it, so `'
gzip'` writes natively.
- `write.parquet.row-group-check-min-record-count` and
`write.parquet.row-group-check-max-record-count`. `requireIntPropertyAtDefault`
trims them, so `' 100'` passes as the default. iceberg-java reads them with
`Integer.parseInt`.
- `write.parquet.page-version` on Iceberg 1.11.
`requireParquetPageVersionDefault` trims and lowercases it, so `V1` passes.
parquet-mr's `WriterVersion.fromString` accepts only `v1`, `v2` or the enum
names. Earlier Iceberg versions have no such property.
The native files are ordinary Parquet, so readers are not affected. The cost
is that the same table accepts some writes and fails others: a write that falls
back to iceberg-java, for example one whose input does not run in Comet, fails
on the setting a native write accepted.
### Steps to reproduce
iceberg-java's side was run with a single-file Java program that calls
`Parquet.write(Files.localOutput(f)).set(key,
value).createWriterFunc(...).build()` and writes one row, against
`iceberg-spark-runtime-4.1_2.13:1.11.0` and
`iceberg-spark-runtime-3.4_2.12:1.5.2`:
```
write.parquet.compression-codec='none'
IllegalArgumentException: Unsupported compression codec: none
write.parquet.compression-codec=' gzip'
IllegalArgumentException: Unsupported compression codec: gzip
write.parquet.row-group-check-min-record-count=' 100'
NumberFormatException: For input string: " 100"
write.parquet.page-version='V1' (1.11.0 only)
IllegalArgumentException: Unsupported Parquet page version: V1 (must be v1 or
v2)
```
`uncompressed`, `GZIP` and `v1` write successfully in the same program.
Comet's side is read from the code and was not run end to end. No rule in
`CometIcebergNativeWrite` looks at the codec name, so with
`spark.comet.write.iceberg.enabled=true` and
`spark.comet.exec.localTableScan.enabled=true`:
```sql
CREATE TABLE local.db.t (id bigint) USING iceberg
TBLPROPERTIES ('write.parquet.compression-codec'='none');
INSERT INTO local.db.t VALUES (1), (2); -- expected to plan
CometIcebergWrite and write uncompressed files
```
Without Comet, the `INSERT` fails with `Unsupported compression codec: none`.
### Expected behavior
The gate should decline a write whose codec or other setting iceberg-java
would reject, so that it falls back and fails as it does without Comet. For the
codec, that means admitting only the names iceberg-java accepts, compared after
upper-casing and without trimming. For the record counts and the page version,
it means parsing them the way iceberg-java does. The `none` case in
`IcebergWriteProtoTranslationSuite` would change with it.
### Additional context
Found while auditing the native Iceberg write docs against the code. #6960
is a related case: there, a numeric gzip level that iceberg-java fails on was
listed as an accepted divergence rather than reproduced, because the level is a
documented Iceberg setting that Hadoop parses as an enum name. None of the
values here is a valid iceberg-java setting, so falling back keeps
iceberg-java's behavior and its error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
