chengwei977 commented on issue #11433: URL: https://github.com/apache/seatunnel/issues/11433#issuecomment-4978805509
> Thanks for spelling out the COMMITTED versus VISIBLE gap clearly. I traced the current Doris 2PC completion path, and this does look like a real issue rather than a usage mistake. > > Right now SeaTunnel stops the load, gets the transaction id, and then the Doris committer treats the 2PC commit as successful once the commit request succeeds or Doris reports the transaction as already COMMITTED / VISIBLE. There is no extra wait or polling step that keeps the SeaTunnel job open until the data is actually visible in Doris. > > So the completion boundary exposed by SeaTunnel is currently earlier than the visibility boundary your downstream SQL jobs depend on. That matches the behavior you are seeing with scheduler-chained jobs. > > This is worth keeping open. The likely fix direction is to distinguish transaction committed from data visible, and only report the sink as finished after the visible stage is reached, or make that boundary explicit if Doris-side behavior forces a different contract. > > If you happen to have a concrete Doris FE response sample for the COMMITTED -> VISIBLE transition on your side, that would help nail down the final polling condition, but the issue itself is already valid as a bug report. > Thanks for spelling out the COMMITTED versus VISIBLE gap clearly. I traced the current Doris 2PC completion path, and this does look like a real issue rather than a usage mistake. > > Right now SeaTunnel stops the load, gets the transaction id, and then the Doris committer treats the 2PC commit as successful once the commit request succeeds or Doris reports the transaction as already COMMITTED / VISIBLE. There is no extra wait or polling step that keeps the SeaTunnel job open until the data is actually visible in Doris. > > So the completion boundary exposed by SeaTunnel is currently earlier than the visibility boundary your downstream SQL jobs depend on. That matches the behavior you are seeing with scheduler-chained jobs. > > This is worth keeping open. The likely fix direction is to distinguish transaction committed from data visible, and only report the sink as finished after the visible stage is reached, or make that boundary explicit if Doris-side behavior forces a different contract. > > If you happen to have a concrete Doris FE response sample for the COMMITTED -> VISIBLE transition on your side, that would help nail down the final polling condition, but the issue itself is already valid as a bug report. There happens to be a case study, this is the configuration of a seatunnel job, where 2pc is enabled in the sink doris stage. <img width="736" height="734" alt="Image" src="https://github.com/user-attachments/assets/acd42a2c-e7d2-406d-8e3a-0da0f7c38666"; /> Additional screenshot of the seatunnel job returning "finish" <img width="630" height="634" alt="Image" src="https://github.com/user-attachments/assets/2b69de88-0c5c-4973-a06e-7def6c11b274"; /> The response process for the COMMITTED -> VISIBLE transformation can be found in Doris FE using sink.label-prefix. <img width="1902" height="343" alt="Image" src="https://github.com/user-attachments/assets/48b9a8d3-728a-4a75-a0a2-c5e078c68c5a"; /> Regarding your point about "helping to determine the final polling conditions," I think we can refer to the Doris FE API: GET /api/example_db/get_load_state?label=my_label { "msg": "success", "code": 0, "data": "VISIBLE", "count": 0 } -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
