srowen commented on pull request #28769: URL: https://github.com/apache/spark/pull/28769#issuecomment-641683180
@HeartSaVioR Have a look at, for example, `AppStatusStore.lastStageAttempt`. It retrieves a `LevelDBIterator` (when using LevelDB) via `.closeableIterator()` but then does close it with try-finally. Look at `LevelDBIterator:116`. It closes when there are no more elements. It does not use CompletionIterator but does the same thing. I'm not suggesting leaks are OK? I'm suggesting figuring out and fixing the leaks directly, not cleaning them up later indirectly if possible. Being `Closeable` isn't the issue per se, nor `.asScala`, but getting them closed in all cases may be tricky. But I don't know if we'll find and fix all of them; we can fix one at least, even if we also have to resort to the cleanup method as well in this PR to catch the other unknown paths. @zhli1142015 I'm not sure how that latest one happens, but it doesn't seem to be a leak; the underlying LevelDBIterator never got created. Looks like a request happened after the DB was closed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
