Copilot commented on code in PR #55639:
URL: https://github.com/apache/arrow/pull/55639#discussion_r4591803830
##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -326,13 +326,11 @@ class ContentDefinedChunker::Impl {
}
if (rep_level == 0 && gearhash.NeedNewChunk()) {
- // if we are at a record boundary and need a new chunk, we create a
new chunk
- auto levels_to_write = offset - prev_offset;
- if (levels_to_write > 0) {
- chunks.push_back({prev_offset, prev_value_offset,
levels_to_write});
- prev_offset = offset;
- prev_value_offset = value_offset;
- }
+ // if we are at a record boundary and need a new chunk, we create a
new chunk,
+ // at offset 0 it is empty and closes the page of the previous call
+ chunks.push_back({prev_offset, prev_value_offset, offset -
prev_offset});
+ prev_offset = offset;
+ prev_value_offset = value_offset;
Review Comment:
This now unconditionally emits a chunk even when `offset - prev_offset ==
0`. While the intention is to allow an empty chunk at offset 0 to close the
previous call’s page, emitting zero-length chunks at non-zero offsets (e.g., if
`NeedNewChunk()` can be true in consecutive iterations without advancing
`offset`/`prev_offset` as expected) would reintroduce empty-page/empty-chunk
behavior and could regress performance. Recommend guarding so that zero-length
chunks are only emitted for the specific intended case (`prev_offset == 0 &&
offset == 0`), and otherwise keep the previous `levels_to_write > 0` behavior.
##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -396,6 +396,7 @@ ParquetInfo GetColumnParquetInfo(const
std::shared_ptr<Buffer>& data, int column
while (auto page = page_reader->NextPage()) {
if (page->type() == PageType::DATA_PAGE || page->type() ==
PageType::DATA_PAGE_V2) {
auto data_page = static_cast<DataPage*>(page.get());
+ EXPECT_GT(data_page->num_values(), 0) << "empty data page";
Review Comment:
The failure message is quite generic and may be hard to debug when this
helper is used across many parametrized tests. Consider including additional
context in the message (e.g., `column_index`, row group index, and/or page
ordinal and page type/version) so a failure immediately indicates where the
empty page originated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]