johnhoran opened a new pull request, #58684: URL: https://github.com/apache/airflow/pull/58684
closes: #58490 When KPO first starts if does a call to either adopt an existing pod or create a new one. If the pod already exists, then its state will be part of the kubernetes reply, and if the pod has already completed then there is little benefit to handing off to the triggerer and back to the operator again. I ran into an issue where a DAG using kubernetes pod operator deffered somehow got into a state where the task was marked as complete before it could transition from the triggerer back to the operator when the main container in the pod had completed. The pod in question was using a sidecar to keep the container alive. On task retries airflow would run and hand off to the trigger component, but when trying to hand back to the operator again, it would notice the state mismatch and the task try would fail. Upon clearing the task, the operator was able to adopt the pod, so it marked the task as successful and cleaned everything up. I feel there is an underlying issue here, but I haven't fully diagnosed it yet, this is an attempt to sidestep it. <!-- Please keep an empty line above the dashes. --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
