Kimahriman commented on pull request #35504: URL: https://github.com/apache/spark/pull/35504#issuecomment-1043163373
> Can I ask what use cases this is targeting? There's no specific use case I'm trying to use more off heap memory that's not specifically off-heap spark features. I've just noticed more than 10% extra memory being used for our normal jobs (that don't use off heap features). I have no idea why or what's using this memory, but it is. I definitely have one job that has a memory leak on the driver side, and is currently sitting at 100g reserved memory with a 32g heap, and haven't figured out why, but we also tried turning on strict memory enforcement in yarn and constantly had executors killed for using too much memory. I haven't investigated these much either, but currently we're calculating a higher memoryOverhead based on the executor memory to give it a little more breathing room, but this makes it easier to just set a factor across the board versus manually tuning memoryOverhead for each job. I feel like I've heard G1GC uses more off-heap memory than other garbage collectors, don't know if that's true? But this is just a quality of life improvement if other people run into similar issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
