vchiapaikeo opened a new issue, #37298:
URL: https://github.com/apache/airflow/issues/37298

   ### Apache Airflow version
   
   2.8.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When KPO and KPO subclasses (like the GKEStartPodOperator) are set to 
deferrable=True and do_xcom_push=True, they enter the 
[execute_complete()](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/7.14.0/airflow/providers/cncf/kubernetes/operators/pod.py#L663)
 method after the triggerer detects that the base container has reached a 
terminal status. When pods fail, we enter [this 
block](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/7.14.0/airflow/providers/cncf/kubernetes/operators/pod.py#L675-L683).
 Eventually, an AirflowException is raised and the call to 
[pod_manager.await_pod_completion()](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/7.14.0/airflow/providers/cncf/kubernetes/operators/pod.py#L697)
 is made. However, when defined with the above arguments, this call never 
completes. 
   
   The reason for this is because the xcom container continues to run. See the 
following screenshot.
   
   <img width="1019" alt="image" 
src="https://github.com/apache/airflow/assets/9200263/099bb5f6-be7d-4c52-a52d-c327e8776530";>
   
   As a result, this is what you see in the logs -
   
   <img width="1372" alt="image" 
src="https://github.com/apache/airflow/assets/9200263/263faebe-e128-4c84-8702-71bf677a05db";>
   
   Despite the pod having already failed:
   
   <img width="940" alt="image" 
src="https://github.com/apache/airflow/assets/9200263/bb5af252-98e3-4045-a40f-ce9c76b0af9d";>
   
   
   ### What you think should happen instead?
   
   We should call self.extract_xcom in the failure scenario just as we do in 
the [success 
scenario](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/7.14.0/airflow/providers/cncf/kubernetes/operators/pod.py#L689-L691)
 to kill the xcom container. There might not be any values to extract but based 
on the docstring, extract_xcom also has the 
[side-effect](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/7.14.0/airflow/providers/cncf/kubernetes/operators/pod.py#L539-L547)
 of killing the xcom container.
   
   ### How to reproduce
   
   Create failing dag with deferrable=True and do_xcom_push=True and observe 
hanging task on execute_complete. 
   
   Sample dag:
   
   ```py
   from airflow import DAG
   
   from airflow.providers.google.cloud.operators.kubernetes_engine import (
       GKEStartPodOperator,
   )
   
   
   DEFAULT_TASK_ARGS = {
       "owner": "gcp-data-platform",
       "start_date": "2021-04-20",
       "retries": 0,
       "retry_delay": 60,
   }
   
   with DAG(
       dag_id="test_gke_op",
       schedule_interval="@daily",
       max_active_runs=1,
       max_active_tasks=5,
       catchup=False,
       default_args=DEFAULT_TASK_ARGS,
   ) as dag:
   
       _ = GKEStartPodOperator(
           task_id="fail",
           name="fail",
           cmds=["bash"],
           arguments=["-xc", "sleep 2 && exit 1"],
           image="gcr.io/google.com/cloudsdktool/cloud-sdk:slim",
           project_id="redacted-project-id",
           namespace="airflow-default",
           location="us-central1",
           cluster_name="airflow-gke-cluster",
           service_account_name="default",
           deferrable=True,
           do_xcom_push=True,
       )
   
   ```
   
   ### Operating System
   
   debian11
   
   ### Versions of Apache Airflow Providers
   
   providers-cncf-kubernetes/7.14.0
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   NOTE: if using ADC credentials, this PR needs to be reverted which breaks 
ADC auth flow: https://github.com/apache/airflow/pull/37081
   
   cc @Lee-W @pankajkoti @dirrao @hussein-awala 
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to