Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/3629#issuecomment-69122238
@suyanNone Having taken a detailed look over the patch, I believe the
correct thing to do here is to just remove the [memory
release](https://github.com/apache/spark/blob/d345ebebd554ac3faa4e870bd7800ed02e89da58/core/src/main/scala/org/apache/spark/storage/MemoryStore.scala#L290)
in `unrollSafely`. Even after we successfully unroll the entire iterator into
an array, we still hold onto the array so it doesn't make sense (in my original
code) to release the memory prematurely. For this reason, I believe much of the
changes in this PR is unnecessary and adds further complexity to the already
complex logic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
