[
https://issues.apache.org/jira/browse/SPARK-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-15891.
------------------------------------
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 2.1.0
> Make YARN logs less noisy
> -------------------------
>
> Key: SPARK-15891
> URL: https://issues.apache.org/jira/browse/SPARK-15891
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 2.1.0
>
>
> Spark can generate a lot of logs when running in YARN mode. The problem is
> already a little bad with normal configuration, but is even worse with
> dynamic allocation on.
> The first problem is that for every executor Spark launches, it will print
> the whole command and all the env variables it's setting, even though those
> are exactly the same for every executor. That's not too bad with a handful of
> executors, but get annoying pretty soon after that. Dynamic allocation makes
> that problem worse since there executors constantly being started and then
> going away.
> Also, there's a lot of logging generated by the dynamic allocation backend
> code in the YARN module. We should audit those and make sure they all make
> sense, and whether / how to reduce the amount of logging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
