Github user mingyukim commented on the pull request:
https://github.com/apache/spark/pull/4420#issuecomment-74828351
@pwendell, I mentioned above, but at a high level, wouldn't it be better to
control the frequency of spills by how much memory you acquire from the shuffle
memory manager at a time than by how often you check if spill is needed?
We can test out your proposal in the specific case we have if that's a less
risky option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
