srowen commented on pull request #28769: URL: https://github.com/apache/spark/pull/28769#issuecomment-641672923
Lots of the paths actually do close the iterator explicitly. And I'm still not sure how .toSeq doesn't consume it. Recall that the underlying iterator closes itself when it consumes all elements. The one at line 42 is the only obvious issue here, which was at least 2 of the 3 ones encountered. I do wonder how far that goes to fixing it in practice, esp if we can think of a another way to write stageList et al. We can't get at the iterator that .asScala produces but we might be able to add a close() method to the view or something, but that's getting ugly ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
