goutamadwant opened a new pull request, #24155: URL: https://github.com/apache/datafusion/pull/24155
## Which issue does this PR close? - Closes #21404. ## Rationale for this change Parquet content-defined chunking is available as an experimental writer feature, but the user documentation does not explain when it is useful or how to configure it. Users need to understand that CDC benefits storage or transfer systems that reuse duplicate byte ranges, does not perform deduplication itself, and requires the sequential writer path for each output file. ## What changes are included in this PR? - Add a user guide explaining Parquet content-defined chunking from a user perspective. - Describe appropriate use cases, storage requirements, limitations, and the sequential-writer tradeoff. - Provide SQL examples for configuring CDC per write and at the session level. - Provide a Rust API example using `ParquetCdcOptions`. - Document the default chunking parameters and tuning considerations. - Add the guide to the documentation index. ## Are these changes tested? No test code is changed because this is a documentation-only update. The documented behavior and examples were verified with: - `cargo test --profile=ci --test sqllogictests -- parquet_cdc.slt` - Compile-checking and executing the Rust example from the guide - `ci/scripts/doc_prettier_check.sh` - `git diff --check` ## Are there any user-facing changes? Yes. This adds user-facing documentation for the existing experimental Parquet CDC feature. It does not change runtime behavior or public APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
