Mike Prior created AIRFLOW-6244:
-----------------------------------

             Summary: spark operator fails tasks for kubernetes jobs, 
eventhough the job is successful.
                 Key: AIRFLOW-6244
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6244
             Project: Apache Airflow
          Issue Type: Bug
          Components: operators
    Affects Versions: 1.10.6
            Reporter: Mike Prior


When spark jobs are run using the 'spark operator', the job will complete 
successfully, but the task is marked as failed. This only happens for 'Dynamic 
Spark' jobs to k8s clusters. The Error code is 0.  If I change the 
'spark_submit_hook.py', so that the only check, is the spark exit code, the 
task is successful. This part of the validation is failing '_*In Kubernetes 
mode, also check the value # of exit code in the log, as it may differ'***_ I 
cannot see any non 0 exit code in the logs.


# Check spark-submit return code. In Kubernetes mode, also check the value

# of exit code in the log, as it may differ.

*### if returncode or (self._is_kubernetes and self._spark_exit_code != 0):*

*if returncode != 0:*

raise AirflowException(

"Cannot execute: {}. Error code is: {}.".format(

spark_submit_cmd, returncode

 

{{}}

 
--------------------------------------------------------------------------------------------

{{[2019-12-09 12:00:27,480] \{{taskinstance.py:1058}} ERROR - Cannot execute: 
['/opt/spark/bin/spark-submit', '--master', 'k8s://https://10.1.1.1:6443', 
'--conf', 'spark.executor.instances=2', '--conf', 'spark.executor.memory=4G', 
'--conf', 'spark.kubernetes.driver.label.app=pyspark243-k8suser', '--conf', 
'spark.kubernetes.executor.label.app=pyspark243-k8suser', '--conf', 
'spark.kubernetes.local.dirs.tmpfs=true', '--conf', 
'spark.kubernetes.driver.limit.cores=2', '--conf', 
'spark.kubernetes.executor.limit.cores=4', '--conf', 
'spark.kubernetes.container.image.pullPolicy=Always', '--conf', 
'spark.kubernetes.container.image=myimage-spark:latest', '--conf', 
'spark.kubernetes.namespace=u2cradle', '--conf', 
'spark.kubernetes.authenticate.caCertFile=/opt/certs/e.crt', '--conf', 
'spark.kubernetes.authenticate.driver.serviceAccountName=u2cradle-devsa', 
'--conf', 
'spark.kubernetes.authenticate.oauthToken=eyJhbGciOiJSUzI1NiIsImtpZCI6I', 
'--conf', '--num-executors', '2', '--total-executor-cores', '1', 
'--executor-cores', '1', '--executor-memory', '2g', '--driver-memory', '1g', 
'--name', 'airflowspark-pi', '--class', 'org.apache.spark.examples.SparkPi', 
'--verbose', '--deploy-mode', 'client', 
'/app/airflow/test/spark-examples_2.11-2.4.4.jar']. Error code is: 0.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", 
line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File 
"/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py",
 line 181, in execute
    self._hook.submit(self._application)
  File 
"/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py",
 line 359, in submit}}{{ }}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to