Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/13482
@andrewor14, I think we should consider two problems here: the fact that
the thread will sleep for less than the min interval if something triggers it
and whatever is currently triggering it. We should certainly fix the loss
reason request that is currently triggering this behavior, but I still think
that this patch is a good solution to the first problem in case there are other
situations that cause it as well.
There's not a good reason to sleep for less than the min interval if it can
cause the application to become unstable. We could look at a more complicated
strategy -- like an exponentially increasing min interval up to the current min
-- but the important thing right now is to ensure nothing can cause this
instability.
To be clear, I don't consider this a complete fix for both of those
problems. We should definitely avoid the `askWithRetry`, only signal the
allocator thread when necessary, etc. But as a safety precaution, I think this
patch is a good start.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
