weixiuli commented on PR #36088: URL: https://github.com/apache/spark/pull/36088#issuecomment-1096199260
> The size should be bounded by `spark.shuffle.service.index.cache.size`; with the cache evicting entries based on LRU. Did you try configuring that to control the memory footprint instead ? It may not be a good ideat to control the memory footprint by the `spark.shuffle.service.index.cache.size` and `spark.shuffle.push.server.mergedIndexCacheSize`. Since if we make them very small it might NOT avoid open/close the index files, if we make them very large it might waste memory. With this pr, it can automatically remove an expired indexFilePath from them. @mridulm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
