saurabhd336 opened a new pull request, #3739: URL: https://github.com/apache/celeborn/pull/3739
### What changes were proposed in this pull request? A lot of times, across multiple Spark apps hosting shuffle data on Celeborn, multiple stale shuffle dependencies can be cleaned up to reduce shuffle load. However, it is not practical to trigger a GC on each app's Driver manually. This change allows Master to notify all connected apps to trigger a gc() when under load. This can give the cluster some breathing room by cleaning up stale shuffle. In order to avoid repeated GCs, once triggered, app side triggers gc only after a configurable delay (5m by default) and the feature on the master side is guarded behind a feature flag (flase by default) ### Why are the changes needed? Allows aggressive cleanup of stale shuffle dependencies during heavy load ### Does this PR resolve a correctness bug? No ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UTs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
