AynSiddiqui opened a new pull request, #1571: URL: https://github.com/apache/arrow-go/pull/1571
### Rationale for this change Fixes #1538. `serializedPageWriter.pageOrdinal` is an `int16`. After 65,536 data pages in one column chunk it wraps back to 0, and the writer records the next page's position as the chunk's `data_page_offset`. Readers then start at the wrong page; in the issue's repro `ReadTable` fails with EOF. ### What changes are included in this PR? - `pageOrdinal` is now an `int32`, as in Arrow C++. - The encryption AAD stores the page ordinal as a signed 16-bit value, so encrypted chunks keep a limit. When a data page's ordinal is above 32,767 (that is, after 32,768 data pages), the writer returns an error before writing that page instead of letting the ordinal wrap. Arrow C++ (`CheckPageOrdinal`) and parquet-java reject the same ordinal, and the error text follows the C++ wording. - The reader's `pageOrd` is unchanged. I asked on the issue whether to widen it here or in a follow-up. ### Are these changes tested? Yes. `TestSerializedPageWriterDataPageOffsetPageOrdinalWrap` writes 65,537 pages to a discarding sink and checks that `data_page_offset` still points at the first page; it fails on main. `TestSerializedPageWriterEncryptionRejectsMoreThanMaxInt16Pages` starts an encrypted writer at ordinal 32,767, checks that this page is written, and that the next one returns the error. `go test ./parquet/...` passes with no tags, with `-tags assert` and with `-tags assert,noasm` (with `PARQUET_TEST_DATA` set). ### Are there any user-facing changes? Unencrypted writers can now write more than 65,536 data pages per column chunk without corrupting the file. Encrypted writers now get an error after 32,768 data pages in one chunk, where the ordinal used to wrap silently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
