JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5119240596
> Incremental commits are not atomic across the whole operation. Ordinary
exceptions
raised while applying a group are reported as results so other groups can
finish;
completed groups are flushed before the error is raised.
> Ray task retry options in `ray_remote_args` do not retry group exceptions
reported
as results, so a transient group failure can leave a partial update.
Worker loss and
other task-level failures which bypass Python exception handling may still
lose
results buffered by that task.
If this is indeed the case, as stated in your documentation, then the
implications are significant—far beyond what a simple description like
“max_groups_per_commit” can capture. This is a major semantic change that has
been overlooked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
