Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/6423#issuecomment-113244059
@massie thanks for updating the tests. It's still a little concerning to
me that we don't explicitly check that the iterator returned from
HashShuffleReader releases all memory. In theory, someone could change
HashShuffleReader to not use asKeyValueIterator (and not use a NextIterator)
such the the stream didn't get closed, and we wouldn't have any unit tests for
that (so could end up with a memory leak). I realize it's annoying to add a
test for that though, and it sounds like @rxin and @JoshRosen are fine with
this as-is; given that I'm fine to sign off as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
