Noam Shemesh created SPARK-48217:
------------------------------------
Summary: Spark stdout and stderr getting removed at end of spark
job triggered from cloudera hue workflow
Key: SPARK-48217
URL: https://issues.apache.org/jira/browse/SPARK-48217
Project: Spark
Issue Type: Improvement
Components: Spark Submit
Affects Versions: 1.6.0
Reporter: Noam Shemesh
Hello,
we are running spark job triggered from cloudera hue workflow
and spark printing stdout and stderr logs during execution as expected:
e.g.
!image-2024-05-09-12-55-55-477.png|width=638,height=332!
*But stdout and stderr logs getting cleaned when workflows finished/status
succeeded*
!image-2024-05-09-12-57-12-144.png!
following is spark-submit command workflow is triggering:
_/usr/bin/spark-submit \_
_--master yarn-client \_
_--driver-memory 4g \_
_--executor-memory 16g \_
_--executor-cores 4 \_
_--class tst \_
_--files `ls -m *.conf | tr -d '\n '` \_
_--conf "spark.dynamicAllocation.maxExecutors=4" \_
_--conf "spark.kryoserializer.buffer.max=1024" \_
_tst.jar $*_
does someone familiar with this spark job behavior or can advise ideas to fix
it?
Thanks in advance
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
