[
https://issues.apache.org/jira/browse/ARROW-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney resolved ARROW-8127.
---------------------------------
Fix Version/s: 0.17.0
Resolution: Fixed
Issue resolved by pull request 6637
[https://github.com/apache/arrow/pull/6637]
> [C++] [Parquet] Incorrect column chunk metadata for multipage batch writes
> --------------------------------------------------------------------------
>
> Key: ARROW-8127
> URL: https://issues.apache.org/jira/browse/ARROW-8127
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: TP Boudreau
> Assignee: TP Boudreau
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.17.0
>
> Attachments: multipage-batch-write.cc
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When writing to a buffered column writer using PLAIN encoding, if the size of
> the batch supplied for writing exceeds the page size for the writer, the
> resulting file has an incorrect data_page_offset set in its column chunk
> metadata. This causes an exception to be thrown when reading the file (file
> appears to be too short to the reader).
> For example, the attached code, which attempts to write a batch of 262145
> Int32's (= 1048576 + 4 bytes) using the default page size of 1048576 bytes
> (with buffered writer, PLAIN encoding), fails on reading, throwing the error:
> "Tried reading 1048678 bytes starting at position 1048633 from file but only
> got 333".
> The error is caused by the second page write tripping the conditional here
> https://github.com/apache/arrow/blob/master/cpp/src/parquet/column_writer.cc#L302,
> in the serialized in-memory writer wrapped by the buffered writer.
> The fix builds the metadata with offsets from the terminal sink rather than
> the in memory buffered sink. A PR is coming.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
