Github user bOOm-X commented on the issue:
https://github.com/apache/spark/pull/18083
I think it is quite interesting to have performance counters on the
dequeing process in the LiveListenerBus.
I am not sure that monitoring (with real metrics) the number of dropped
events really worth it. You just want to know if messages have been dropped
(and having the number in the log is fine).
I am agree that having the number of messages in the queue is important.
For the number of processed events, it is, in my mind, as for the number of
dropped events not so important.
For the execution time of message processing it is very interesting, but
not having the by listener or by event type breakdowns (just the global timing)
will not allow to do a fine grained analysis and so not to do improvements
I think that it will be better to have this timings for each listener
individually (More than the dequeing process itself, the performance
improvements will be achieved at the listener level).
So putting the counters in ListenerBus is more appropriate for me. This
will allows to not only monitor the LiveListenerBus, but the other one too
(like: StreamingQueryListenerBus, StreamingListenerBus, ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
