how2how2how2-arch commented on issue #76735: URL: https://github.com/apache/airflow/issues/76735#issuecomment-6735609041
The nonce as described covers the case where the response is lost and the same worker retries, and I think it is worth being explicit about the one it does not cover, because the two are easy to read as one. If the worker dies after the server handled the exchange, the successor takes the message off the broker without a nonce, gets 409, and the callback is marked failed without running. That is unchanged by this proposal rather than made worse, but it means "safe to retry" holds only for the process that already exchanged. Whether you want that second cell decides where the nonce lives. Kept in the worker, the coverage is one process. Written to the callback row in the same transaction as QUEUED to RUNNING, a successor can present it, and the exchange becomes idempotent per callback run instead of per process. That seems to cost nothing in what you are defending: what stops a leaked broker copy is that the nonce is absent from the message, not that it is absent from the database. One thing I could not answer from the issue: a callback that is RUNNING with the exchange done and no worker left to run it. The only timeout I found is `[callbacks] callback_execution_timeout`, which reads to me as supervising the callback subprocess from the client side, so it would not fire when there is no client. If nothing moves that row to a terminal state, this proposal shifts the failure from "marked failed without running", which is visible, to a row that stays RUNNING, which is not — the worse of the two. I read that off `ACTIVE_STATES` including RUNNING while `TERMINAL_STATES` is SUCCESS and FAILED, so a sweep living outside `models/callback.py` is easy for me to have missed; if there is one, ignore that paragraph. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
