[
https://issues.apache.org/jira/browse/SPARK-48217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noam Shemesh updated SPARK-48217:
---------------------------------
Attachment: workflow_running_logs_printed.png
workflow_succeeded_logs_cleaned.png
> Spark stdout and stderr getting removed at end of spark job triggered from
> cloudera hue workflow
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-48217
> URL: https://issues.apache.org/jira/browse/SPARK-48217
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 1.6.0
> Reporter: Noam Shemesh
> Priority: Major
> Attachments: workflow_running_logs_printed.png,
> workflow_succeeded_logs_cleaned.png
>
>
> Hello,
> we are running spark job triggered from cloudera hue workflow
> and spark printing stdout and stderr logs during execution as expected:
> e.g.
> !image-2024-05-09-12-55-55-477.png|width=638,height=332!
> *But stdout and stderr logs getting cleaned when workflows finished/status
> succeeded*
> !image-2024-05-09-12-57-12-144.png!
>
> following is spark-submit command workflow is triggering:
> _/usr/bin/spark-submit \_
> _--master yarn-client \_
> _--driver-memory 4g \_
> _--executor-memory 16g \_
> _--executor-cores 4 \_
> _--class tst \_
> _--files `ls -m *.conf | tr -d '\n '` \_
> _--conf "spark.dynamicAllocation.maxExecutors=4" \_
> _--conf "spark.kryoserializer.buffer.max=1024" \_
> _tst.jar $*_
>
>
> does someone familiar with this spark job behavior or can advise ideas to fix
> it?
>
> Thanks in advance
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
