AynSiddiqui opened a new pull request, #1597: URL: https://github.com/apache/arrow-go/pull/1597
### Rationale for this change After an encrypted column reaches the page limit added in #1571, later closes can succeed and write a footer for incomplete data. In a buffered row group, the failed close can also leave 4 MiB allocated. Thanks to @hcrosse for finding and describing these four cases [on #1571](https://github.com/apache/arrow-go/pull/1571#issuecomment-6925530401). This is a follow-up to that PR, related to #1538. ### What changes are included in this PR? The shared column writer retains page-write and close errors so later column, row-group and file closes report the failure instead of finishing the file. The pqarrow file writer also retains its close error, including when `Write` has already closed the writer after a failure. Column cleanup now releases remaining dictionary-encoded data pages and the buffered page writer's in-memory sink through `release()`. The separate pager release in `discard()` is removed because `discard()` already calls `release()`. This also releases the buffer when closing fails before the pager can finish. Regression tests cover plain columns, pqarrow's automatic close after a failed write, serial dictionary columns and buffered dictionary columns. They check repeated closes, the missing footer and the checked allocator returning to zero. ### Are these changes tested? The four regressions fail against main at `4e345140` and pass with this change. With the Arrow and Parquet test-data environment variables set: ```sh go test -count=1 ./parquet/file/... ./parquet/pqarrow/... go test -race -count=1 -run '^(TestEncryptedPageLimitClose|TestFileWriterEncryptedPageLimitClose)$' ./parquet/file ./parquet/pqarrow go vet ./parquet/file/... ./parquet/pqarrow/... ``` `gofmt` and `git diff --check` are clean. `go generate ./parquet/file` leaves the diff unchanged; the template and generated source match main. ### Are there any user-facing changes? After the encrypted page-limit failure, repeated closes report the page error and do not write a footer. The failed buffered row group also releases its allocated memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
