kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2016500977
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1382,11 +1383,15 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
chunk.levels_to_write, *chunk_array,
ctx,
maybe_parent_nulls));
}
- if (num_buffered_values_ > 0) {
+ bool is_last_chunk = i == chunks.size() - 1;
+ if (num_buffered_values_ > 0 && !is_last_chunk) {
// Explicitly add a new data page according to the content-defined
chunk
// boundaries. This way the same chunks will have the same
byte-sequence
// in the resulting file, which can be identified by content
addressible
// storage.
+ // Note that the last chunk doesn't trigger a new data page in order
to
+ // allow subsequent WriteArrow() calls to continue writing to the
same
+ // data page, the chunker's state is not being reset after the last
chunk.
Review Comment:
Case 2.
When we feed the last chunk through the chunker, it doesn't detect a
boundary and it doesn't exceed the maximum size, so its state is not being
reset. We can continue to feed values through it and it will continue the chunk
calculation from that point.
The writer has a partial chunk at the end so we don't need to explicitly
call `AddDataPage()`. The user will either call `WriteArrow()` again to
continue writing the chunk, or the writer will be closed which flushes the
currently active page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
