leaves12138 commented on code in PR #8863:
URL: https://github.com/apache/paimon/pull/8863#discussion_r3663666231
##########
paimon-python/pypaimon/write/writer/data_writer.py:
##########
@@ -188,16 +192,24 @@ def _append_file_sequence_range(self, row_count: int) ->
Tuple[int, int]:
def _check_and_roll_if_needed(self):
while self.pending_data is not None:
- current_size = self.pending_data.nbytes
- if current_size <= self.target_file_size:
+ num_rows = self.pending_data.num_rows
+ # Row-count trigger: keep at most target_file_row_num rows per
file.
+ split_row = num_rows
+ if num_rows > self.target_file_row_num:
+ split_row = self.target_file_row_num
Review Comment:
This also enables row-count rolling for `ShardTableUpdator`, which
constructs `AppendOnlyDataWriter` directly. That path only overrides
`target_file_size` and `SingleWriter.end()` requires exactly one output file.
Repro: create a data-evolution table with one 5-row file, alter
`target-file-row-num` to `2`, then run `new_shard_updator`; the writer produces
three files and fails with `Should have one file.` The base revision produces
one 5-row file. Please disable row rolling in `ShardTableUpdator` as already
done by `TableUpdateByRowId` (or otherwise scope this logic to ordinary append
writes), and add a regression test for this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
