viirya commented on issue #26105: [SPARK-26570][SQL] Reduce memory consumption when transforming very many filestatus in InMemoryFileIndex.bulkListLeafFiles URL: https://github.com/apache/spark/pull/26105#issuecomment-541838731 First from the stacktrace, I was thinking it is possible that the reason to user OOM is that SerializableFileStatus is too many. Then it is possible to cause OOM when transforming them back to Status. This PR was created for that. But from users latest posts, from jmap logs, looks like SerializableFileStatus do not hold too much memory. For now I am not sure SerializableFileStatus is the cause of the OOM, so close this first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
