HeartSaVioR commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-624964828
The idea is similar with HistoryServerDiskManager so makes sense in general. We may need to get concrete answers for these questions to go forward: 1. How we will guarantee these area of memory is used only for Hybrid KV store to prevent OOM? (Or no guard and end users have to deal with providing enough memory on heap?) 2. How to calculate approximate memory usage? I guess it would be safe to assume the approximate size as event log file size, but it would take over huge memory for single app. (That may not be a problem on safety perspective, but pretty less efficient.) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
