leaves12138 commented on code in PR #8893:
URL: https://github.com/apache/paimon/pull/8893#discussion_r3667219876
##########
paimon-python/pypaimon/read/reader/format_pyarrow_reader.py:
##########
@@ -115,12 +167,15 @@ def __init__(self, file_io: FileIO, file_format: str,
file_path: str,
# Read projected VARIANT columns in bounded batches.
self._parquet_file = None
- if self._bounded_variant_read:
+ if (self._bounded_variant_read
+ or self._selected_parquet_row_groups is not None):
Review Comment:
Blocking: this new row-group path also handles projections with no physical
Parquet columns. In that case, `_select_existing_fields(batch)` rebuilds the
zero-column batch with `pa.RecordBatch.from_arrays([], ...)`, resetting its row
count to 0. This occurs for `with_projection(["_ROW_ID"])` and for
schema-evolution files where every projected column is missing. With `_ROW_ID
BETWEEN 2 AND 7`, the base commit returns `[2, 3, 4, 5, 6, 7]`, but this branch
returns no rows on PyArrow 16.0.0. Please preserve the row count of zero-column
batches and add a system-only/missing-only projection regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
