Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18004
> But I clearly explain what I think is the best option for this potential
issue
Well, you stated your opinion, and I disagree with it. I've tried to
explain why I disagree with it and propose solutions that I consider better,
but all you've done so far is refuse to accept any feedback.
> It is the responsibility of the user / and or the writer of the custom
listener for its performance
That's unrealistic. I'll go back again to my favorite example, the dynamic
allocation listener. Even if all listeners behave well and don't block for
arbitrary periods of time, you still have a problem: the more listeners you
add, by definition, the more expensive the listener bus becomes, and having
everything in a single thread will eventually lead to issues.
And that goes back to why I've been suggesting the different approach in
the first place. It's bad design to force all these constraints of a whole set
of code, some of which is not even controlled by Spark, just so that one single
listener can operate well. Instead, why not have a system where that single
listener - and any other really - can be treated especially and avoid all of
those issues?
Allowing arbitrary listeners to be asynchronous (basically making your code
generic, or implementing the solution in the other PR I mentioned) solves that
issue. The dynamic allocation listener extends whatever class makes it
asynchronous, or register itself to the bus with whatever API gives it a
separate "event queue", and now it's not affected by other listeners.
> synchronization issue of the UI listeners. But it is clearly unrelated to
the listener bus.
*Of course* it's related. Those synchronized methods are being called by
the listener bus; meaning they can block the listener bus and cause events to
be dropped if they're arriving at a high rate.
> I am the one that propose to NOT change the global design of listener
bus. You want to change it.
How is that related to the limitations of your current design? I've given
you example after example of why fixing just the event logging listener is the
wrong approach but for some reason you don't see that.
> Please point me out the race condition that you find
As I said, I don't want to review the code because I disagree with your
current approach. Pointing out individual issues with the code will not help at
this point in time. Once we can agree on how to actually solve the problem
you're trying to solve in a more generic and maintainable manner, then we can
look at the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
