tooptoop4 opened a new issue #8453: Spark version 3 does not work URL: https://github.com/apache/airflow/issues/8453 If anyone runs into this in the future I've found out where the issue is in the spark_submit_hook.py. Line 419 in spark submit hook match_exit_code = re.search(r'\s*Exit code: (\d+)', line) in spark 3.0 the line that prints the exit code is actually lower case E on "Exit code:" so this re.search will never find that value. To fix this you can simply switch the line to this match_exit_code = re.search(r'\s*Exit code: (\d+)', line, re.IGNORECASE) Which should also be backwards compatible.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
