Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4420#issuecomment-73566542
@mingyukim we can't just remove the check because this is a hot code path.
Otherwise we'll try to access a synchronized block inside `tryToAcquire` every
single element (not partition) and that's really expensive. I think a threshold
that looks something like `currentMemory - myMemoryThreshold` might make sense
as a simple fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
