nileshrathi345 opened a new pull request, #533: URL: https://github.com/apache/livy/pull/533
## What changes were proposed in this pull request? **Problem** Livy was calling YarnClient.getApplications(SPARK), which asks the ResourceManager for all Spark applications and then filters client-side by application tag. On clusters with many Spark apps, this produces large RM responses, adds unnecessary load on YARN, and can contribute to RM memory pressure (especially during session startup polling and leaked-app GC). **Solution** Use a tag-filtered GetApplicationsRequest so the RM returns only matching applications: - **getAppIdFromTag()** — when resolving a session’s YARN app ID by tag, query with the lowercase session tag instead of scanning all Spark apps. - **leakedAppsGCThread** — when cleaning up leaked apps, batch all pending leaked tags into one filtered request per GC cycle instead of fetching the full Spark app list. Jira: https://issues.apache.org/jira/browse/LIVY-866 ## How was this patch tested? Updated and ran SparkYarnAppSpec, including: - "should get App Id" — verifies tag-based app ID lookup uses the filtered getApplications request. - "should delete leak app when timeout" — verifies the leaked-app GC path uses the filtered request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
