JingsongLi commented on code in PR #9365:
URL: https://github.com/apache/paimon/pull/9365#discussion_r3868455406
##########
paimon-python/pypaimon/read/datasource/torch_dataset.py:
##########
@@ -100,10 +104,35 @@ def _row_to_dict(self, offset_row) -> dict:
row_dict[field_name] = value
return row_dict
+ def _limit_covers_all_splits(self) -> bool:
+ limit = self.table_read.limit
+ if limit is None:
+ return True
+ total_rows = 0
+ for split in self.splits:
+ row_count = getattr(split, "row_count", None)
Review Comment:
[P2] Use merged row counts before collapsing DataLoader workers
Split.row_count is the sum of physical file rows, so it can substantially
exceed the logical output for deletion-vector and Data Evolution splits. For
example, if two splits each have 10 physical rows but 4 merged rows and the
limit is 8, the limit is non-binding; this code nevertheless totals 20 and
routes every split to worker 0, unnecessarily regressing row, shuffle, and
batch streaming from multi-worker to single-worker execution.
TableRead._limit_covers_all_splits already uses split.merged_row_count() for
the same proof. Please prefer a validated merged_row_count() when it is
available, fall back conservatively when it is unknown, and add a non-binding
test where physical and merged counts differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
