andygrove opened a new pull request, #7027: URL: https://github.com/apache/datafusion-comet/pull/7027
## Which issue does this PR close? Closes #7008. ## Rationale for this change The native write gate trims and lowercases several Parquet settings that iceberg-java reads strictly, so some writes that fail without Comet succeed with it. iceberg-java resolves the codec with `CompressionCodecName.valueOf(codec.toUpperCase(Locale.ENGLISH))` (parquet-mr has `UNCOMPRESSED` but no `NONE`), parses the row-group record counts with `Integer.parseInt`, and on Iceberg 1.11 accepts only `v1` or `v2` for the page version. The native files are ordinary Parquet, so readers are not affected. The cost is that one table accepts some writes and fails others, depending on whether the write runs natively. ## What changes are included in this PR? - The compression codec is resolved the way iceberg-java does: matched case-insensitively and never trimmed. `none` is no longer mapped to uncompressed, because it is ORC's spelling. A new gate rule, `compressionCodecRejection`, declines any other name. That includes `lzo` and `lz4_raw`, which iceberg-java accepts but the native writer has no equivalent for. - `write.parquet.row-group-check-min-record-count` and `-max-record-count` are parsed with `Integer.parseInt` on the raw value, so `' 100'` falls back. - `write.parquet.page-version` is compared exactly to `v1`, so `V1` falls back. - `iceberg-writes.md` describes the stricter rules. ## How are these changes tested? - `IcebergWriteProtoTranslationSuite`: new tests for the codec rejection rules and for the codec not being trimmed. The `none` mapping tests now use `uncompressed`. - `CometIcebergWriteDetectionSuite`: new tests for a rejected codec (`none`, `' gzip'`), whitespace in a record count and `V1`, plus a `GZIP` table that stays `Compatible`. - Both suites pass locally on the default profile (Spark 4.1): 105 tests, 0 failed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
