kszucs opened a new pull request, #51685: URL: https://github.com/apache/arrow/pull/51685
### Rationale for this change The content-defined chunker was owned by the column writer, which is recreated for every row group, so the chunking state was reset at each row group boundary and page boundaries depended on where a row group starts. See #51684. ### What changes are included in this PR? - The file writer keeps one chunker per leaf column for the whole file. It is created with the column's first column writer and passed on to the column writers of the following row groups. The public `ColumnWriter::Make()` still creates its own chunker. - The column writer chunks the data only if it has a chunker. - `NextColumn()` takes the column ordinal together with the column metadata, so a retry after a failed close can't pick another column's chunker. ### Are these changes tested? Yes. `IndependentOfRowGroupBoundaries` writes the same table into one and into multiple row groups and expects the same page boundaries apart from the row group ends; the first commit adds it alone, failing without the fix. `ColumnWriterWithoutFileWriter` covers the standalone column writer and `NextColumnAfterFailedClose` the retried column. `UpdateOnce` and `Append` relied on the reset and are adjusted. ### Are there any user-facing changes? Files with multiple row groups get different page boundaries than before, since the chunking now continues across row groups. ### Was AI used for this PR? In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR. Claude Code wrote the code, tests and description under human direction. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
