Github user bOOm-X commented on the issue:
https://github.com/apache/spark/pull/18253
@vanzin Ok it is ready now.
> why SynchronousListenerBus? It doesn't seem to add anything interesting
over the existing interface.
I removed it.
> the whole "group of listeners" abstraction seems unnecessary. As far as I
see it can be folded into the "listener queue" concept - a queue feeds events
to a collection of listeners.
I simplified it and add usage in the other commits. It is basically usefull
to hold the metrics, and I need a common way to add a group of dependent
listeners to the LiveListenerBus and the ReplayBus.
> Changing "post" to "postToAll" as part of this change also is adding a
lot of unnecessary noise. I'm not a fan of the current class hierarchy of the
listener bus and I think that change makes sense, but at the same time it
should be done separately since it's distracting here
100% agree. I removed it
> I'd also like to see better justification for your custom queue
implementation. Have you identified the use of BlockingQueue as a hotspot in
the current code?
This implementation had 2 advantages: it is a 1 producer - 1 consumer queue
whereas the BlockingQueue is a n producers - m consumers. So it use much less
synchronization. The other advantage (the main one) is that no object is
created for each message added to the queue. So it produces a lot less garbage.
More independent queues we have, more it is significant.
> My main worry is the sleep; if you're unlucky, your queue will always be
20ms behind in processing events and may suffer if there's a sudden burst while
it's sleeping.
I change it to 1 ms instead of 20ms. This time is much less than the
average processing time of the fastest listener (around 5 ms for the
HeartbeatListener). It is just to force the consumer thread to escape in case
of empty queue to give more chance to the producer thread to be scheduled. I
can remove it if you want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
