XiaoHongbo-Hope commented on code in PR #9365:
URL: https://github.com/apache/paimon/pull/9365#discussion_r3868657127
##########
paimon-python/pypaimon/read/datasource/torch_dataset.py:
##########
@@ -100,10 +104,35 @@ def _row_to_dict(self, offset_row) -> dict:
row_dict[field_name] = value
return row_dict
+ def _limit_covers_all_splits(self) -> bool:
+ limit = self.table_read.limit
+ if limit is None:
+ return True
+ total_rows = 0
+ for split in self.splits:
+ row_count = getattr(split, "row_count", None)
Review Comment:
Fixed in cc695c0e0. The worker-limit proof now prefers merged_row_count()
only when it is a valid nonnegative count no larger than the physical row
count; otherwise it falls back to the physical upper bound. The regression
covers 20 physical rows, 8 merged rows, and limit=8 retaining both workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
