raphaelauv commented on PR #75658: URL: https://github.com/apache/airflow/pull/75658#issuecomment-6725162329
Thanks @shahar1 for the review :) what's fix : - A failed Job writes its pod logs before failing, honouring `container_logs`, and a pod whose logs can't be read no longer fails the task. The `parallelism < 1` check is back. - The trigger retries pod listing on transient errors, skips a pod on a status-0 exec error only if the pod was deleted, and treats an invalid `return.json` as unread. - The XCom loss after a triggerer restart is documented. `execution_timeout` now also bounds the deferred wait (Airflow 3 doesn't apply it while deferred), and the Job is deleted when it's reached. - Test mocks use `spec`/`autospec`. It was tested on kind with Kueue v0.20.5 and Airflow 3.4.0 : - preemption in the middle of the Job in sync and deferrable modes (XCom of the succeeded pods only, logs of both pods), a partial `return.json`, and a deferred timeout on a missing queue. I've been using for few weeks in production this https://gist.github.com/raphaelauv/7ddbf1ad6816ca864be97c1a9b86eab3#file-kueuejoboperator-py it have way more features , but I want to contribute them to apache-airflow in multiple PR ( end_from_trigger , start_from_trigger , on_finish_action. live-logs ... ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
