chengwei977 commented on issue #11433: URL: https://github.com/apache/seatunnel/issues/11433#issuecomment-5225165197
> Thanks for the clear report. This matches Doris semantics: after a 2PC commit the transaction enters COMMITTED and only becomes VISIBLE after the publish phase. A reasonable fix is for DorisCommitter to poll the transaction status (e.g., via SHOW TRANSACTION or the stream load transaction status API) until VISIBLE, with a configurable timeout, before completing the commit. > > Could you confirm the Doris version you are using? This helps determine which API is available for checking transaction visibility (e.g., 'sync' publish options or transaction status endpoints differ across versions). > > As a workaround until this is fixed, you can add a short wait/retry in the downstream DolphinScheduler task, or set 'sync_publish' related session/table properties on the Doris side so commit returns only after data is visible. For Doris version 2.1.11, I think it's worth considering setting a timeout parameter `doris.config.wait-for-visible`. Within this timeout period, the transaction status API (`get_load_state`) should be called repeatedly until the transaction reaches `VISIBLE` and `seatunnel` returns `FINISHED`, or until the transaction is still `COMMITTED` after `doris.config.wait-for-visible` is reached. In this case, the `seatunnel` job status should be `FAILED`, prompting the user to check if the Doris job has encountered an error. This parameter should have a default value of 30 seconds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
