hussein-awala commented on issue #31795: URL: https://github.com/apache/airflow/issues/31795#issuecomment-1627712416
@zoid-w @ikholodkov Do you have multiple triggerers running simultaneously with the `AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC` value set to 30 or higher? Regarding #31803, it addresses an issue when there is a single triggerer under pressure. Previously, it could take some time to process sent events, allowing the triggerer to read another message in the meantime. However, after this PR, that scenario is no longer possible. It's important to note that the triggerer does not provide an exactly-once guarantee for running triggers; its guarantee is at least-once. This means it is possible to have the same trigger running on multiple triggerers simultaneously. We encountered an issue with the HA mode, which I have fixed in #32123. So, when version 2.6.3 is released, it is advisable to upgrade to it. While #31803 and #32123 can significantly reduce the probability of this problem occurring to less than 1%, it is still a possibility. I am currently exploring ways to synchronize between the triggerers when necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
