yugan95 commented on code in PR #9375:
URL: https://github.com/apache/paimon/pull/9375#discussion_r3842318913
##########
paimon-python/pypaimon/write/writer/data_writer.py:
##########
@@ -122,9 +123,8 @@ def write(self, data: pa.RecordBatch):
raise e
def prepare_commit(self) -> List[DataFileMeta]:
- if self.pending_data is not None and self.pending_data.num_rows > 0:
- self._write_data_to_file(self.pending_data)
- self.pending_data = None
+ if self._buffer.num_rows > 0:
+ self._write_data_to_file(self._buffer.take())
Review Comment:
Good catch, fixed. Neither `prepare_commit` nor `close` drains up front any
more: `prepare_commit` resets after the write returns, and `close` keeps its
`finally` reset — both matching master. Regression test added.
Same pattern in `KeyValueDataWriter._flush_all` (present on master too),
fixed as well but differently: `_roll_write` spans several files, so keeping
every row would make a retry rewrite the ones that already landed. The buffer
now keeps only the rows no file has taken. Safe because the sort and the fold
are idempotent on their own output — PKs are unique once folded.
`BlobWriter.take()` is fine as-is: it only runs inside `write()`, which
aborts on failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
