weixiuli opened a new pull request, #36088: URL: https://github.com/apache/spark/pull/36088
### What changes were proposed in this pull request? Currently, we use Guava Cache either in the ExternalShuffleBlockResolver or the RemoteBlockPushResolver to cache index file information so that we can avoid open/close the index files for each request. However, it will take up a fixed amount of memory, even though we no longer need any cached index files, which is a waste of memory. In our production，we set spark.shuffle.service.index.cache.size to be 400m，and found above problem. So, we should automatically remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.  ### Why are the changes needed? To save memory ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing unittests and Passed CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
