fishfishfishfishaa commented on PR #8671: URL: https://github.com/apache/paimon/pull/8671#issuecomment-4990482646
Regarding the ITCase failure in batch mode: due to the asynchronous architecture of the coordinator, the batch job finishes first, and close() interrupts the final commit. I've adjusted it to use synchronous blocking for batch EndInput events. Since this is not waiting for the last EndInput event — it only waits for in-memory operations — the blocking time is minimal. Moreover, EndInput itself is in the "finalization" phase. Compared to other approaches, I believe this design is simpler and more consistent with the original logic. (Other alternatives include: 1. Modifying close() to wait for all pending commits; 2. Explicitly waiting only for the last one; 3. Adding an ack mechanism.) Additionally, the coordinator Context initialization issue has also been fixed. In streaming mode, the last snapshot phase was too strict, so end-input.watermark=12345 has been removed. In this scenario, the original CommitterOperator does not promise to create an empty snapshot for writing the watermark either — instead, it relies on committer.forceCreatingSnapshot(). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
