yugan95 commented on code in PR #9375:
URL: https://github.com/apache/paimon/pull/9375#discussion_r3849418810
##########
paimon-python/pypaimon/write/writer/key_value_data_writer.py:
##########
@@ -127,20 +131,27 @@ def _roll_write(self, data: pa.Table) -> None:
size does not violate the LSM file-internal invariant.
Reuses ``_find_optimal_split_point`` / ``_write_data_to_file``
from the base class.
+
+ The buffer is narrowed to the rows still unwritten after each file, so
+ a failure part way through leaves the remainder -- and only the
+ remainder -- for whoever flushes next.
"""
while data.num_rows > 0:
if data.nbytes <= self.target_file_size:
self._write_data_to_file(data)
+ self._buffer.reset()
return
split_row = self._find_optimal_split_point(
data, self.target_file_size)
if split_row <= 0:
# Single row already exceeds target_file_size; nothing
# to gain from further slicing, write it as-is.
self._write_data_to_file(data)
+ self._buffer.reset()
return
self._write_data_to_file(data.slice(0, split_row))
Review Comment:
Fixed in `DataWriter._write_data_to_file`, which both `_roll_write` and
`prepare_commit` go through.
The data file's meta was appended before the changelog was written, and the
`try` did not cover that write. So a changelog failure kept the data file and
its meta while the rows stayed buffered, and the retry wrote a second file for
the same rows. Now the meta is appended only after the data file, its sidecar
and its changelog have all landed, and a failure deletes whatever it wrote.
New test in `changelog_producer_test.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
