t oo created AIRFLOW-7052: ----------------------------- Summary: spark 3.0.0 does not work with sparksubmitoperator Key: AIRFLOW-7052 URL: https://issues.apache.org/jira/browse/AIRFLOW-7052 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: 1.10.9 Reporter: t oo
from slack: If anyone runs into this in the future I've found out where the issue is in the spark_submit_hook.py. Line 419 ``match_exit_code = re.search(r'\s*Exit code: (\d+)', line)``` in spark 3.0 the line that prints the exit code is actually lower case E on "Exit code:" so this re.search will never find that value. To fix this you can simply switch the line to this ```match_exit_code = re.search(r'\s*Exit code: (\d+)', line, re.IGNORECASE)``` Which should also be backwards compatible. MattD Having some difficulty understanding why my spark-submit task is being marked as failed even though the spark job has completed successfully, I see these logs at the end of the job, exit code: 0 termination reason: Completed But then it also right after displays this, Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 187, in execute self._hook.submit(self._application) File "/usr/local/lib/python3.7/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 403, in submit self._mask_cmd(spark_submit_cmd), returncode airflow.exceptions.AirflowException: Cannot execute: spark-submit (spark submit args would be here) Error code is: 0. I took a look at spark_submit_hook.py line 403 and it shows that it shouldn't be throwing that exception if the error code is 0. Anyone have any ideas? I'm only finding this happens now that I've switched to using spark 3.0, never ran into this with spark 2.4.5. *Also running 1.10.9 now -- This message was sent by Atlassian Jira (v8.3.4#803005)