XiaoHongbo-Hope commented on code in PR #9375:
URL: https://github.com/apache/paimon/pull/9375#discussion_r3843629633
##########
paimon-python/pypaimon/write/writer/key_value_data_writer.py:
##########
@@ -127,20 +131,27 @@ def _roll_write(self, data: pa.Table) -> None:
size does not violate the LSM file-internal invariant.
Reuses ``_find_optimal_split_point`` / ``_write_data_to_file``
from the base class.
+
+ The buffer is narrowed to the rows still unwritten after each file, so
+ a failure part way through leaves the remainder -- and only the
+ remainder -- for whoever flushes next.
"""
while data.num_rows > 0:
if data.nbytes <= self.target_file_size:
self._write_data_to_file(data)
+ self._buffer.reset()
return
split_row = self._find_optimal_split_point(
data, self.target_file_size)
if split_row <= 0:
# Single row already exceeds target_file_size; nothing
# to gain from further slicing, write it as-is.
self._write_data_to_file(data)
+ self._buffer.reset()
return
self._write_data_to_file(data.slice(0, split_row))
Review Comment:
Changelog failure can commit the same data chunk twice on retry, making data
and changelog inconsistent. Please make the data/changelog flush retry-safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
