Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4887#issuecomment-155892902
@suyanNone When we try to unroll `MEMORY_AND_DISK` there are two cases: (1)
it succeeds, so the values remain in memory, or (2) it fails, so the values are
dropped to disk. In the case of (1), we can't release anything because the
in-memory buffer will be consumed directly by the caller. In (2), however, we
don't need the in-memory buffer anymore so we can just release it. That's what
SPARK-6157 is, and actually an earlier version of this patch fixes it correctly.
Does that make sense? Are we talking about different issues?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
