Github user bOOm-X commented on the issue:
https://github.com/apache/spark/pull/18004
> Wait, which comment?
You mention the issue with the external listener as I did not mention this
case. But I clearly explain what I think is the best option for this potential
issue: log very clearly which external listener is causing the issue. It is
the responsibility of the user / and or the writer of the custom listener for
its performance and not the spark one. But helping to quickly identifying the
issue is I think a good idea and should not be very expensive.
Then you mentioned the UI listener issue. If you quoted my full sentences,
I said that there is a issue for the event logging listener and the UI
listeners but it is fine for the OTHER ones.
Then you mentioned the synchronization issue of the UI listener. But it is
clearly unrelated to the listener bus. Putting immutable collections in them
will directly solve this synchronization issue. But allocation much more
memory, so maybe introducing an other issue - even if this is not so certain
given the frequency of the events which change the number of entry in the maps.
> you current design has limitations
I am the one that propose to NOT change the global design of listener bus.
You want to change it. This design (reactor pattern) make its proof in many
other softwares (nodejs, vertx.Ix) with one known limitation, listener should
be carefully implemented. It is why I want to improve thing at the listener
level. Are we agree on that or are we still discussing the design with multiple
duplicated queues ?
> I noticed at least one race after a very quick look
Please point me out the race condition that you find
For the concrete implementation of the asynchronous event logging listener
I preferred to factorized it with the synchronous one instead of creating an
upper abstract asynchronous listener. This choice is indeed disputable, but you
omitted this factorization in your comment.
This current implementation is quite simple due to the absence of a
(generic) dispatch (message type) mechanism and message type filtering. With
what you propose, the implementation will be much more complex and I think that
it is not valuable yet while there is no concrete need of this mechanism. I
would rather go with a simple implementation here and refactor it to a much
more generic one when it is implemented in a second listener, and see only at
this moment what is needed to be factorized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
