kszucs opened a new pull request, #3818:
URL: https://github.com/apache/parquet-java/pull/3818
### Rationale for this change
Parquet C++ and parquet-rs can end data pages at content-defined boundaries.
Files that share a run of values then share byte-identical pages, which
content-addressable storage (e.g. Hugging Face Xet) can deduplicate. This PR
adds the same opt-in feature to parquet-java. Iceberg Java needs it to honor
the `write.parquet.content-defined-chunking.*` table properties.
### What changes are included in this PR?
- A port of the Arrow C++ / arrow-rs chunker. It uses the same gear tables,
mask derivation, defaults and cut rules, so page boundaries match Arrow C++.
- `ContentDefinedChunkingOptions`: min/max chunk size and normalization
level, defaulting to 256 KiB / 1 MiB / 0.
- `ParquetProperties` and `ParquetWriter.Builder`:
`withContentDefinedChunkingEnabled`, `withContentDefinedChunkingOptions`.
- `ParquetOutputFormat`:
`parquet.page.content-defined-chunking.{enabled,min.size,max.size,norm.level}`.
- `ChunkingColumnWriter` wraps the column writer only when chunking is
enabled, so the default write path is unchanged.
- `dev/cdc`: a parity check against pyarrow and a dedup probe.
`ContentDefinedChunkingWriteBenchmarks` measures the write cost.
### Are these changes tested?
Yes. Unit tests cover the options, gear hash table, mask derivation, chunker
and the end-to-end writer. `TestContentDefinedChunkerGoldenBoundaries` asserts
page boundaries produced by Arrow C++. `dev/cdc/run.sh parity` regenerates
these boundaries from pyarrow; all six cases match.
### Are there any user-facing changes?
Yes: new writer options and configuration keys, marked EXPERIMENTAL and off
by default. Written files are ordinary Parquet files and need no reader support.
Closes #3817
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
