GitHub user sitalkedia opened a pull request:
https://github.com/apache/spark/pull/16291
[SPARK-18838] Use separate executor service for each event listener
## What changes were proposed in this pull request?
Currently we are observing the issue of very high event processing delay in
driver's `ListenerBus` for large jobs with many tasks. Many critical component
of the scheduler like `ExecutorAllocationManager`, `HeartbeatReceiver` depend
on the `ListenerBus` events and these delay is causing job failure. For
example, a significant delay in receiving the `SparkListenerTaskStart` might
cause `ExecutorAllocationManager` manager to remove an executor which is not
idle. The event processor in `ListenerBus` is a single thread which loops
through all the Listeners for each event and processes each event synchronously
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala#L94.
The single threaded processor often becomes the bottleneck for large jobs.
In addition to that, if one of the Listener is very slow, all the listeners
will pay the price of delay incurred by the slow listener.
To solve the above problems, we plan to have a per listener single threaded
executor service and separate event queue. That way we are not bottlenecked by
the single threaded event processor and also critical listeners will not be
penalized by the slow listeners. The downside of this approach is separate
event queue per listener will increase the driver memory footprint.
## How was this patch tested?
Tested by running the job on the cluster and the average event processing
latency dropped from 2 minutes to 3 milliseconds.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sitalkedia/spark skedia/event_log_processing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16291.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16291
----
commit b4af82f0a95487cd099432d17864b3cfac2780bb
Author: Sital Kedia <[email protected]>
Date: 2016-12-14T23:38:58Z
[SPARK-18838] Use a separate executor service for each event listener
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
