ColtenOuO opened a new pull request, #76824: URL: https://github.com/apache/airflow/pull/76824
On Airflow versions before 3.3, durable execution keeps its journal in Object Storage. If writing a step to the journal failed (Ex. full disk), the error propagated and failed the task even though the LLM call or tool call had already succeeded. The retry then ran that step again, paying for the LLM call twice and repeating any side effects of the tool. Durable execution exists to prevent exactly that. `DurableStorageProtocol` says a backend may skip a failed write without failing the step, and the task state store backend used on Airflow 3.3+ already does so. The Object Storage backend skipped only entries it could not encode. - A failed write is skipped with a warning and `save_step` returns `False`, the same as an entry that cannot be encoded. The journal then runs that step again on retry. - The in-memory cache goes back to what the file still holds, so a later successful write cannot persist an entry the journal was told was not recorded. When the save was overwriting an existing key, the old value is kept instead of being dropped. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
