[
https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-21321.
-----------------------------------
Resolution: Fixed
Assignee: Jong Yoon Lee
Fix Version/s: 2.3.0
2.2.1
> Spark very verbose on shutdown confusing users
> ----------------------------------------------
>
> Key: SPARK-21321
> URL: https://issues.apache.org/jira/browse/SPARK-21321
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Jong Yoon Lee
> Assignee: Jong Yoon Lee
> Priority: Minor
> Fix For: 2.2.1, 2.3.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> On shutdown, spark can be very verbose and can spit out errors that causes
> the user to be confused.
> If possible, we should not print those out and just ignore them by changing
> the changing the log level from WARNING to DEBUG.
> Also shutdown of spark can take a long time because of the backlog of events
> in the event queue ( warnings like: Message $message dropped.
> ${e.getMessage}).
> This happens more with dynamic allocation on.
> I am suggesting to
> 1. change the log level when the shutdown is happening and the RPC
> connections are closed(RpcEnvStoppedException).
> 2. Clear the event queue when RPC module is stopped and spark is shutting
> down.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
