andygrove opened a new issue, #6954: URL: https://github.com/apache/datafusion-comet/issues/6954
### What is the problem the feature request solves? Each of these settings makes a write fall back, but each has a direct parquet-rs equivalent: - **`parquet.enable.dictionary=false`** as a table or write property. iceberg-java reads it in `Parquet.WriteBuilder` and turns dictionary encoding off for every column. Native: `WriterProperties::set_dictionary_enabled(false)`, skipping the `DictionaryChooser`. - **`parquet.columnindex.truncate.length`**, as a table property or in the writer's Hadoop configuration. iceberg-java's `ParquetWriter` reads it (default 64) for the column index. Native: `set_column_index_truncate_length`. - **A zstd `write.parquet.compression-level` of `0`**, which zstd defines as its default level, 3. parquet-rs accepts 1 to 22, so translate 0 to 3 on the JVM side. Negative levels wait on apache/arrow-rs#11545. - **Non-default `write.parquet.row-group-check-min-record-count` and `row-group-check-max-record-count`.** iceberg-java uses them for its own row-group size checks and passes them to parquet-mr as the minimum and maximum row counts between page size checks. The one native dependency is `FIRST_PAGE_SIZE_CHECK_ROWS` in `iceberg_dictionary.rs`, which hard-codes the default minimum of 100 for the first page, the page that decides dictionary encoding; pass the table's value instead. Row-group and page boundaries are already accepted divergences. - **`write.parquet.stats-enabled.column.<col>`** (Iceberg 1.10+). iceberg-java turns off parquet-mr statistics for the column. Native: `set_column_statistics_enabled(path, EnabledStatistics::None)`. Check that the committed manifest metrics still match, including the float and double NaN counts and bounds the native writer carries over. - **`write.parquet.page-version=v2`** (Iceberg 1.11). Native: `WriterVersion::PARQUET_2_0`. parquet-rs and parquet-mr choose the same v2 fallback encodings (`RLE` for booleans, `DELTA_BINARY_PACKED` for int32 and int64, `DELTA_BYTE_ARRAY` for binary and fixed-length binary, `PLAIN` for float and double), and under v2 both dictionary-encode fixed-length binary, so `DictionaryChooser` has to consider fixed-length binary columns too. ### Describe the potential solution One setting at a time: vet the key in `CometIcebergNativeWrite`, translate it in `IcebergWriteProtoTranslation` and `build_writer_properties`, and add a test that writes through both writers and compares rows and `readable_metrics`, plus, for the encoding settings, each column chunk's encodings. Add a `CometIcebergWriteDetectionSuite` case, and update the user guide's eligibility table and its "Plans for each fallback" lists in the same change. ### Additional context Part of #5643 and the native write epic #5649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
