JingsongLi commented on code in PR #8404:
URL: https://github.com/apache/paimon/pull/8404#discussion_r3506302962
##########
paimon-python/pypaimon/read/reader/concat_batch_reader.py:
##########
@@ -335,6 +344,18 @@ def _expand_ranges(ranges: List[Range]) -> List[int]:
for row_id in range(row_range.from_, row_range.to + 1)
]
+ def _is_deleted(self, row_id: int) -> bool:
Review Comment:
all-deleted data-evolution groups can crash. This PR applies DV before
MergeAllBatchReader; if every row in a field bunch is deleted,
ApplyDeletionVectorReader.read_arrow_batch() returns None, all_batches stays
empty, then ds.InMemoryDataset(self.merged_batch) receives None and raises
TypeError. I reproduced it locally with a one-batch reader whose two rows are
both deleted. Fix should return None before constructing InMemoryDataset when
all_batches is empty, and add a test for “all rows in one data-evolution row-id
group are deleted”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
