kszucs opened a new pull request, #3608: URL: https://github.com/apache/iceberg-python/pull/3608
# Rationale for this change PyArrow's `ParquetWriter` has natively supported content-defined chunking (CDC) since 21.0.0, producing stable page boundaries across appends (useful for content-addressable storage / dedup). PyIceberg's write path already funnels through a single kwargs builder (`_get_parquet_writer_kwargs`), so this wires CDC through as `write.parquet.content-defined-chunking.*` table properties, mirroring the property names and defaults iceberg-rust already uses for cross-engine consistency. A `pyarrow>=21.0.0` version guard raises a clear `ImportError` if CDC is requested on an older PyArrow (extracted into a shared `_require_pyarrow_version` helper, reused by the existing Azure-filesystem version guard). ## Are these changes tested? Yes: unit tests for `_get_parquet_writer_kwargs` (disabled by default, enabled with defaults, enabled with custom values, unsupported PyArrow version) and an integration-style test that writes a table with CDC enabled end-to-end and reads it back. ## Are there any user-facing changes? Yes: four new table properties (`write.parquet.content-defined-chunking.enabled`, `.min-chunk-size`, `.max-chunk-size`, `.norm-level`), documented in `configuration.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
