ESUBLXX opened a new issue #19552:
URL: https://github.com/apache/airflow/issues/19552
### Apache Airflow version
2.0.2
### Operating System
Linux
### Versions of Apache Airflow Providers
Linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I am using KubernetesPodOperator to deploy my Spark Application and I am
doing SparkSubmit to deploy below is the syntax . But once the spark driver
gets completed , the task is still running and not moving to Completed State in
the UI. the dependent tasks are not starting since the existing task is not
getting completed. Please let me know if this is a known issue and we have any
workaround for this.
node_data_load_task = KubernetesPodOperator(
namespace='isp-airflow',
image='sekidocker.rnd.ki.sw.ericsson.se/proj-gaia-dev/gaia-isp-batch-job:v1',
name='node-data-load-task',
task_id='node_data_load_task',
in_cluster=True,
get_logs=True,
startup_timeout_seconds=STARTUP_TIMEOUT_SECONDS,
#secrets=[PG_PASSWORD],
cmds = [],
arguments=[
'/opt/spark/bin/spark-submit',
'--master',
'k8s://https://aksispprodeus05-6ab4d922.hcp.eastus.azmk8s.io:443',
'--deploy-mode', 'cluster',
'--name', 'isp-node-config-load',
'--conf', 'spark.executor.instances=2',
'--conf',
'spark.kubernetes.authenticate.driver.serviceAccountName=spark-airflow-serviceaccount',
'--conf', 'spark.kubernetes.namespace=isp-airflow',
'--conf',
'spark.kubernetes.container.image=sekidocker.rnd.ki.sw.ericsson.se/proj-gaia-dev/gaia-isp-batch-job:v1',
'--conf',
'spark.kubernetes.driver.secretKeyRef.PG_PASSWORD=crash-pipeline-secret:PG_PASSWORD',
'--conf',
'spark.kubernetes.driver.secretKeyRef.PG_HOST=crash-pipeline-secret:PG_HOST',
'--conf',
'spark.kubernetes.driver.volumes.persistentVolumeClaim.isp-spark-job-pvc.options.claimName=isp-spark-job-pvc',
'--conf',
'spark.kubernetes.driver.volumes.persistentVolumeClaim.isp-spark-job-pvc.mount.path=/mnt/isp-spark-batch',
'--conf',
'spark.kubernetes.driver.volumes.persistentVolumeClaim.isp-spark-job-pvc.mount.readOnly=false',
'--conf',
'spark.kubernetes.executor.volumes.persistentVolumeClaim.isp-spark-job-pvc.options.claimName=isp-spark-job-pvc',
'--conf',
'spark.kubernetes.executor.volumes.persistentVolumeClaim.isp-spark-job-pvc.mount.path=/mnt/isp-spark-batch',
'--conf',
'spark.kubernetes.executor.volumes.persistentVolumeClaim.isp-spark-job-pvc.mount.readOnly=false',
'--conf',
'spark.kubernetes.container.image.pullPolicy=Always',
'--conf',
'spark.cassandra.connection.host=prod-new-cassandra.isp-prod-new.svc.cluster.local',
'--conf', 'spark.executor.cores=1',
'--conf', 'spark.executor.memory=2g',
'local:///home/gaia/app/run.py',
'--job isp_input_data_load_job',
'--config-file configs/isp_config.json',
'--job-args load-input-table=node-config
daily-node-info-flag=true'
],
dag=dag
)
### What happened
_No response_
### What you expected to happen
_No response_
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
