Github user sitalkedia commented on the issue:
https://github.com/apache/spark/pull/16291
@markhamstra, @vanzin thanks for your comments!
>> so it is entirely possible. for example, for one ListenerEventExecutor
to process a task end event for a particular task before another
ListenerEventExecutor has worked sufficiently through its eventQueue to have
even seen the corresponding task start event. That is quite a bit different
than the prior ordering guarantee implied by the comment "onPostEvent is
guaranteed to be called in the same thread for all listeners."
That is true. If a listener is slow in processing the events than it might
not see the task start event where as other listeners might already have
processed the corresponding task end event. I am not sure why would that cause
any problem though. Each listener should in theory could work independent of
each other and we should only guarantee ordered processing of the events within
a listener. This is the only way I can think of to scale the driver for large
workloads.
>> but I wonder if a system where listeners can "opt in" to this behavior
wouldn't be better. e.g., EventLoggingListener can slow down the bus, having it
run in a separate thread would benefit all the others.
I think that will be an over-kill. Having a separate thread per listener
simplifies the design. Also, for user added listeners, it will be hard to
decide if we want to have a shared or separate thread per listener. The only
downside of having a separate thread per listener is increased memory footprint
in worst case. But I would argue that since the average event processing
latency is decreased by multi threaded event processor, the backlog on the
event queues will clear faster which actually might decrease the driver memory
footprint on average case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
