chengwei977 commented on issue #11433: URL: https://github.com/apache/seatunnel/issues/11433#issuecomment-5193044421
> Thanks for the detailed report. This looks like a valid gap: after the 2PC commit call the connector should optionally poll transaction status (e.g. via `SHOW TRANSACTION` or the stream load status endpoint) until it reaches VISIBLE before treating the commit as complete. > > A reasonable fix would be a new sink option such as `sink.wait-for-visible` (with a configurable timeout) so users with strict downstream dependencies can enable it without penalizing latency-sensitive pipelines. Would you be willing to submit a PR along these lines? > > For context, could you share your Doris version and whether you observe the same behavior with `sink.enable-2pc = false`? This will help confirm the scope of the fix. 1、The Doris version is 2.1.11 2、With 2pc set to false, the transaction state will be in the VISIBLE state before seatunnel returns job FINISHED. It's worth noting that we considered enabling 2PC precisely because we encountered situations where the seatunnel worker node crashed while running this type of job, leading to data inconsistency issues between the source and sink sides after retries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
