GitHub user dirkraft opened a pull request:
https://github.com/apache/spark/pull/18850
Remove dropping bus logic.
Log warning when event uptake takes too long, but block anyways. Since
dropping events is no longer possible, removed that code.
## What changes were proposed in this pull request?
LiveListenerBus events are now never dropped. I believe that critical
components communicate through this bus. I will also add the argument that a
buggy spark UI (because of dropped events) is just as useless, and so I am
unaware of any kind of event which can be lossy.
## How was this patch tested?
Basically we ran it in our production environment. Our past couple spark
runs now reliably complete with this change since crucial events never get
dropped, but it is unclear how much lag this change might be contributing to
the overall run time.
We have partition counts in the thousands and executors in the hundreds.
----
Perhaps the events can be tagged as critical (`queue.put`) or not
(`queue.offer`), but this small change is meant to get spark stable again. We
have turned up the eventqueue size to 1,000,000 (and fiddled with all available
settings), but it still isn't enough. With some probability, enough crucial
events are dropped and leaves the spark job hung indefinitely unable to recover
(sometimes it does). The large eventqueue also maxed out our driver process
with 64GB of memory, so that's pretty much untenable.
With this change and over tens (hundreds?) of millions of events flowing
through this bus, only 1100 triggered the slow warning, usually around 20ms
with a max of 100ms. The current working hypothesis is that the events tend to
arrive in bursts and so quickly overwhelm the queue and then quickly empty out.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dirkraft/spark dont-drop-events-ever
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18850.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18850
----
commit 56c4c1d87716b6f629be83fff8197b36a607a9ae
Author: Jason Dunkelberger <[email protected]>
Date: 2017-08-04T23:21:02Z
Remove droppy logic. Log warning when event uptake takes too long, but
block anyways.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
