Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16387
@samkum can you reply to the questions above? This PR is basically blocked
on your feedback that it makes things worse.
> @samkum did you try this patch without the "spill map on every call"
behavior (basically returning "false" instead of spilling the in-memory map
when the iterator doesn't exist)?
>
> If that case also results in increased gc, maybe your app just needs more
memory. That would be the case where the assert would trigger before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
