kamcheungting-db opened a new issue, #943: URL: https://github.com/apache/iceberg-cpp/issues/943
The C++ Parquet writer does not apply `write.parquet.page-size-bytes` or `write.parquet.dict-size-bytes`. Both properties are already declared in `TableProperties`, but passing them through `WriterProperties::FromMap` has no effect on the underlying Parquet writer configuration. On the current main, the [`ParquetWriter::Impl::Open`](https://github.com/apache/iceberg-cpp/blob/525d60f372c11226998f27c4d9f4e2ee063bbe2c/src/iceberg/parquet/parquet_writer.cc#L240) has already configured the `compression`, `compression level`, and the `row-count limit`, however it builds Arrow's writer properties without setting the `page` or `dictionary` size. This prevents callers from using the standard Iceberg properties to tune these sizes. For instance, providing the following properties should configure a 64 KiB data-page target and a 256 KiB dictionary-page limit on output parquet writer: ```text write.parquet.page-size-bytes=65536 write.parquet.dict-size-bytes=262144 ``` Proposed change: - Expose these two settings through `WriterProperties` and apply them when constructing Arrow's Parquet writer properties. - Use the Iceberg defaults already declared in [`table_properties.h`](https://github.com/apache/iceberg-cpp/blob/525d60f372c11226998f27c4d9f4e2ee063bbe2c/src/iceberg/table_properties.h#L117): 1 MiB for data pages and 2 MiB for dictionary pages. - Reject invalid or non-positive values before creating the output file. - Add tests for defaults, overrides, invalid values, and propagation into the underlying writer, plus write/read coverage with custom settings. Background: - Java resolves and validates both properties in [`Parquet.WriteBuilder.Context.dataContext`](https://github.com/apache/iceberg/blob/ef99ad3f84127b41b68bb2f66026aa21bed7bbda/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L609). - Rust applies them in [`ParquetWriterBuilder::from_table_properties`](https://github.com/apache/iceberg-rust/blob/8030cf44445b3f957e367515e9d3c5eeb00f104d/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L90). Note: Byte-based row-group sizing (`write.parquet.row-group-size-bytes`) can be handled in a separate follow-up because it requires examining the buffering and flush policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
