truonglac2603 opened a new issue, #39184:
URL: https://github.com/apache/airflow/issues/39184
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.3
### What happened?
I'm having an issue when using SparkKubernetesOperator to execute Spark job.
Whenever i set do_xcom_push to True, the driver pod is created and run
perfectly, but the xcom sidecar pod is nowhere to be found. Airflow log is thus
stuck with this message:
> [2024-04-23, 02:47:38 UTC] {custom_object_launcher.py:301} WARNING - Spark
job submitted but not yet started. job_id: spark-custome-task-5r3pt2vk
> [2024-04-23, 02:47:48 UTC] {pod_manager.py:529} ERROR - container base
whose logs were requested not found in the pod
spark-custome-task-5r3pt2vk-driver
> [2024-04-23, 02:47:48 UTC] {pod_manager.py:718} INFO - Checking if xcom
sidecar container is started.
> [2024-04-23, 02:47:48 UTC] {pod_manager.py:724} WARNING - The xcom sidecar
container is not yet started.
> [2024-04-23, 02:51:24 UTC] {local_task_job_runner.py:296} WARNING - DagRun
timed out after 0:05:03.242442.
> [2024-04-23, 02:51:29 UTC] {local_task_job_runner.py:296} WARNING - DagRun
timed out after 0:05:08.319418.
> [2024-04-23, 02:51:29 UTC] {local_task_job_runner.py:302} WARNING - State
of this instance has been externally set to skipped. Terminating instance.
> [2024-04-23, 02:51:29 UTC] {process_utils.py:131} INFO - Sending 15 to
group 30. PIDs of all processes in the group: [30]
> [2024-04-23, 02:51:29 UTC] {process_utils.py:86} INFO - Sending the signal
15 to group 30
> [2024-04-23, 02:51:29 UTC] {taskinstance.py:2483} ERROR - Received
SIGTERM. Terminating subprocesses.
> [2024-04-23, 02:51:29 UTC] {process_utils.py:79} INFO - Process
psutil.Process(pid=30, status='terminated', exitcode=0, started='02:46:43')
(30) terminated with exit code 0
In my opinion, there should be something wrong with xcom push for this
operator. Any help would be much appreciated. Thanks in advance.
### What you think should happen instead?
xcom pushed perfectly with any case of spark, maybe there will be
placeholder for spark job with no return value or whatsoever
### How to reproduce
submit a basic spark application by using SparkKubernetesOperator to K8s
cluster and set do_xcom_push to True
### Operating System
Ubuntu 22.04.4 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==8.0.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
