AynSiddiqui opened a new pull request, #1597:
URL: https://github.com/apache/arrow-go/pull/1597

   ### Rationale for this change
   
   After an encrypted column reaches the page limit added in #1571, later 
closes can succeed and write a footer for incomplete data. In a buffered row 
group, the failed close can also leave 4 MiB allocated.
   
   Thanks to @hcrosse for finding and describing these four cases [on 
#1571](https://github.com/apache/arrow-go/pull/1571#issuecomment-6925530401). 
This is a follow-up to that PR, related to #1538.
   
   ### What changes are included in this PR?
   
   The shared column writer retains page-write and close errors so later 
column, row-group and file closes report the failure instead of finishing the 
file. The pqarrow file writer also retains its close error, including when 
`Write` has already closed the writer after a failure.
   
   Column cleanup now releases remaining dictionary-encoded data pages and the 
buffered page writer's in-memory sink through `release()`. The separate pager 
release in `discard()` is removed because `discard()` already calls 
`release()`. This also releases the buffer when closing fails before the pager 
can finish.
   
   Regression tests cover plain columns, pqarrow's automatic close after a 
failed write, serial dictionary columns and buffered dictionary columns. They 
check repeated closes, the missing footer and the checked allocator returning 
to zero.
   
   ### Are these changes tested?
   
   The four regressions fail against main at `4e345140` and pass with this 
change. With the Arrow and Parquet test-data environment variables set:
   
   ```sh
   go test -count=1 ./parquet/file/... ./parquet/pqarrow/...
   go test -race -count=1 -run 
'^(TestEncryptedPageLimitClose|TestFileWriterEncryptedPageLimitClose)$' 
./parquet/file ./parquet/pqarrow
   go vet ./parquet/file/... ./parquet/pqarrow/...
   ```
   
   `gofmt` and `git diff --check` are clean. `go generate ./parquet/file` 
leaves the diff unchanged; the template and generated source match main.
   
   ### Are there any user-facing changes?
   
   After the encrypted page-limit failure, repeated closes report the page 
error and do not write a footer. The failed buffered row group also releases 
its allocated memory.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to