XiaoHongbo-Hope commented on PR #8124:
URL: https://github.com/apache/paimon/pull/8124#issuecomment-4628655866
Thanks for the PR! I agree abort is useful before prepare_commit()
succeeds, but I’m worried about aborting after commit messages have already
been produced.
The main risk here is after prepare_commit() succeeds. RayDatasink.write()
will return the prepared commit messages after close(). But
DedicatedFormatWriter.close() catches exceptions, calls abort(), and does not
re-raise.
So if close() fails, abort() may delete files referenced by the prepared
commit messages, while the worker still returns those messages to the driver.
The driver may then commit a manifest with missing files.
We hit an abort-related data-safety issue in production before; see #7232
for context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
