Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18004
> not block the dequeing process ! and this is the responsibility of each
listener unfortunately
That's exactly why the other approach is better. It's more complex and may
use more memory, but it allows you to choose "special" listeners that cannot
miss events and place them in a separate queue. For example, if the dynamic
allocation listener loses events, things go south pretty quick. And a
user-installed listener that misbehaves may negatively affect that.
Multiple queues mitigate that issue; the user listener will have limited
effect on the dynamic allocation listeners (it will still fight for CPU, most
most probably things will behave better).
> I agree that other listeners should become asynchronous too
Then why are you against the other approach? It's basically making
listeners "asynchronous" without having to bake that logic into every listener.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
