hanna-liashchuk opened a new issue, #31183: URL: https://github.com/apache/airflow/issues/31183
### Apache Airflow version 2.6.0 ### What happened After upgrading to 2.6.0 version, pipelines with SparkKubernetesOperator -> SparkKubernetesSensor stopped working correctly. [this PR](https://github.com/apache/airflow/pull/29977) introduces some enhancement into Spark Kubernetes logic, now SparkKubernetesOperator receives the log from spark pods (which is great), but it doesn't monitor the status of a pod, which means if spark application fails - a task in Airflow finishes successfully. On the other hand, using previous pipelines (Operator + Sensor) is impossible now, cause SparkKubernetesSensor fails with `jinja2.exceptions.UndefinedError: 'None' has no attribute 'metadata'` as SparkKubernetesOperator is no longer pushing info to xcom. ### What you think should happen instead Old pipelines should be compatible with Airflow 2.6.0, even though the log would be retrieved in two places - operator and sensor. OR remove the sensor completely and implement all the functionality in the operator (log + status) ### How to reproduce Create a DAG with two operators ``` t1 = SparkKubernetesOperator( kubernetes_conn_id='common/kubernetes_default', task_id=f"task-submit", namespace="namespace", application_file="spark-applications/app.yaml", do_xcom_push=True, dag=dag, ) t2 = SparkKubernetesSensor( task_id=f"task-sensor", namespace="namespace", application_name=f"{{{{ task_instance.xcom_pull(task_ids='task-submit')['metadata']['name'] }}}}", dag=dag, attach_log=True, ) ``` ### Operating System Debian GNU/Linux 10 (buster) ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==8.0.0 apache-airflow-providers-apache-spark==4.0.1 apache-airflow-providers-celery==3.1.0 apache-airflow-providers-cncf-kubernetes==6.1.0 apache-airflow-providers-common-sql==1.4.0 apache-airflow-providers-docker==3.6.0 apache-airflow-providers-elasticsearch==4.4.0 apache-airflow-providers-ftp==3.3.1 apache-airflow-providers-google==10.0.0 apache-airflow-providers-grpc==3.1.0 apache-airflow-providers-hashicorp==3.3.1 apache-airflow-providers-http==4.3.0 apache-airflow-providers-imap==3.1.1 apache-airflow-providers-microsoft-azure==6.0.0 apache-airflow-providers-microsoft-mssql==3.3.2 apache-airflow-providers-microsoft-psrp==2.2.0 apache-airflow-providers-microsoft-winrm==3.1.1 apache-airflow-providers-mysql==5.0.0 apache-airflow-providers-odbc==3.2.1 apache-airflow-providers-oracle==3.6.0 apache-airflow-providers-postgres==5.4.0 apache-airflow-providers-redis==3.1.0 apache-airflow-providers-sendgrid==3.1.0 apache-airflow-providers-sftp==4.2.4 apache-airflow-providers-slack==7.2.0 apache-airflow-providers-snowflake==4.0.5 apache-airflow-providers-sqlite==3.3.2 apache-airflow-providers-ssh==3.6.0 apache-airflow-providers-telegram==4.0.0 ### Deployment Other 3rd-party Helm chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
