Akshat Bordia created LIVY-866:
----------------------------------
Summary: Optimize Yarn GetApplications Query to prevent additional
load on Yarn and Livy
Key: LIVY-866
URL: https://issues.apache.org/jira/browse/LIVY-866
Project: Livy
Issue Type: Improvement
Components: Server
Reporter: Akshat Bordia
Currently Livy queries Yarn applications by applicationType : Spark. This will
put heavy load on Yarn clusters if there are thousands or more Spark
application in all states (running, finished, failed, queued etc.).
A better approach would be to query the applications by tags in addition to job
type since Livy only needs to track application with certain application tags.
However, YarnClient does not expose any API to query applications by tags.
We can extend the YarnClientImpl which would allow us to use protected rmClient
in YarnClient and we can implement getApplications method with
GetApplicationsRequest where we can include tags.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
