DanielLeens commented on issue #11433: URL: https://github.com/apache/seatunnel/issues/11433#issuecomment-4976041463
Thanks for spelling out the COMMITTED versus VISIBLE gap clearly. I traced the current Doris 2PC completion path, and this does look like a real issue rather than a usage mistake. Right now SeaTunnel stops the load, gets the transaction id, and then the Doris committer treats the 2PC commit as successful once the commit request succeeds or Doris reports the transaction as already COMMITTED / VISIBLE. There is no extra wait or polling step that keeps the SeaTunnel job open until the data is actually visible in Doris. So the completion boundary exposed by SeaTunnel is currently earlier than the visibility boundary your downstream SQL jobs depend on. That matches the behavior you are seeing with scheduler-chained jobs. This is worth keeping open. The likely fix direction is to distinguish transaction committed from data visible, and only report the sink as finished after the visible stage is reached, or make that boundary explicit if Doris-side behavior forces a different contract. If you happen to have a concrete Doris FE response sample for the COMMITTED -> VISIBLE transition on your side, that would help nail down the final polling condition, but the issue itself is already valid as a bug report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
