HeartSaVioR commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-655125113
@baohe-zhang Thanks for the update. This is really helpful. So the small event log file shows there's a chance the ratio can be beyond 1/2. Personally it seems OK to simply allow such case, as these small event logs can be migrated to the LevelDB KV store fast enough, and the memory usage would hold shortly. That said, it'd be simpler if we can just go through with LevelDB KV store for small event logs, no need to run background thread per app log. Either is fine for me. For latter the lower bound should be configured against uncompressed log, and we should do the similar estimation on compressed log before applying. That's another tricky one, but probably good to have. (I wouldn't run background thread for event log which can be loaded in a couple of seconds) @tgravescs WDYT? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
