SEZ9 commented on issue #11433: URL: https://github.com/apache/seatunnel/issues/11433#issuecomment-5215603997
Thanks for the detailed report. This is a known gap between Doris's COMMITTED and VISIBLE (publish) phases. A possible fix is for DorisCommitter to poll the transaction/load state (e.g., via `SHOW TRANSACTION` or the `get_load_state` API) until it reaches VISIBLE before signaling commit success, ideally behind a config option like `doris.config.wait-for-visible` with a timeout. Could you confirm the Doris version you are using and roughly how large the COMMITTED-to-VISIBLE gap is in your environment? This helps decide reasonable default polling intervals and timeouts. As a temporary workaround, you can add a short delay or a data-availability check in your DolphinScheduler downstream task before querying the loaded data. <!-- streview-comment:63 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
