SEZ9 commented on issue #12624: URL: https://github.com/apache/seatunnel/issues/12624#issuecomment-5987255190
Thanks for the detailed analysis and for the deterministic SQLite tests, that makes both paths easy to follow. The root cause in both cases looks the same: in manual-commit mode, everything flushed since the last commit lives in one open transaction, so losing or replacing the connection before commit silently discards earlier batches while the checkpoint still succeeds. A few thoughts on the fix direction: 1. Flush retry: when a flush fails with a connection or transaction-level error (08xxx/40xxx) in manual-commit mode, retrying only the latest batch is not safe. It seems better to fail the task so the engine restores from the last checkpoint, rather than reconnecting and re-sending only the current batch. 2. Pooled connection replaced before commit: `commitIfNeeded()` should not transparently commit on a freshly swapped-in connection. If the connection that held the open transaction has been replaced, that should surface as an error so the checkpoint fails and the data is replayed. Since you mentioned you are willing to submit a PR, that would be very welcome. It would be great if the PR includes the two `JdbcMultiTableReconnectTest` cases you described so the fix is covered by tests. Please also note in the PR whether the Oracle path (which always forces manual commit) is handled, since Oracle users would hit this with default settings. <!-- streview-comment:1526 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
