[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531080#comment-15531080 ] Dhruve Ashar commented on SPARK-17649: -- I get the idea. While you certainly cannot filter out a specific type of message. You could increase the heartbeat interval to reduce the executor metrics update events to an acceptable degree. This would be significant while running with a large no. of executors. The stats would the effect of it on the event bus. > Log how many Spark events got dropped in LiveListenerBus > > > Key: SPARK-17649 > URL: https://issues.apache.org/jira/browse/SPARK-17649 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Minor > Fix For: 1.6.3, 2.0.2, 2.1.0 > > > Log how many Spark events got dropped in LiveListenerBus so that the user can > get insights on how to set a correct event queue size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530301#comment-15530301 ] Shixiong Zhu commented on SPARK-17649: -- My motivation of logging the dropped events is for helping find out a reasonable queue size. Not sue if more stat information is helpful. You cannot drop only some types of messages. > Log how many Spark events got dropped in LiveListenerBus > > > Key: SPARK-17649 > URL: https://issues.apache.org/jira/browse/SPARK-17649 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Minor > Fix For: 1.6.3, 2.0.2, 2.1.0 > > > Log how many Spark events got dropped in LiveListenerBus so that the user can > get insights on how to set a correct event queue size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526426#comment-15526426 ] Dhruve Ashar commented on SPARK-17649: -- [~zsxwing] Since we are logging the drop event count, it would be beneficial to provide some stats into how many were dropped of a specific type. >From what I have seen is majority of the dropped events are Executor Metrics >update. It would make the log message a bit longer, but the information will >be helpful. We could add it as a debug as well. Let me know your thoughts. > Log how many Spark events got dropped in LiveListenerBus > > > Key: SPARK-17649 > URL: https://issues.apache.org/jira/browse/SPARK-17649 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Minor > Fix For: 1.6.3, 2.0.2, 2.1.0 > > > Log how many Spark events got dropped in LiveListenerBus so that the user can > get insights on how to set a correct event queue size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517949#comment-15517949 ] Apache Spark commented on SPARK-17649: -- User 'zsxwing' has created a pull request for this issue: https://github.com/apache/spark/pull/15226 > Log how many Spark events got dropped in LiveListenerBus > > > Key: SPARK-17649 > URL: https://issues.apache.org/jira/browse/SPARK-17649 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Minor > > Log how many Spark events got dropped in LiveListenerBus so that the user can > get insights on how to set a correct event queue size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517248#comment-15517248 ] Apache Spark commented on SPARK-17649: -- User 'zsxwing' has created a pull request for this issue: https://github.com/apache/spark/pull/15220 > Log how many Spark events got dropped in LiveListenerBus > > > Key: SPARK-17649 > URL: https://issues.apache.org/jira/browse/SPARK-17649 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Minor > > Log how many Spark events got dropped in LiveListenerBus so that the user can > get insights on how to set a correct event queue size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
