Eric Yang created YARN-7399:
-------------------------------
Summary: Yarn services metadata storage improvement
Key: YARN-7399
URL: https://issues.apache.org/jira/browse/YARN-7399
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn-native-services
Reporter: Eric Yang
In Slider, metadata is stored in user's home directory. Slider command line
interface interacts with HDFS directly to list deployed applications and invoke
YARN API or HDFS API to provide information to user. This design works for a
single user manage his/her own applications. When this design has been ported
to Yarn services, it becomes apparent that this design is difficult to list all
deployed applications on Hadoop cluster for administrator to manage
applications. Resource Manager needs to crawl through every user's home
directory to compile metadata about deployed applications. This can trigger
high load on namenode to list hundreds or thousands of list directory calls
owned by different users. Hence, it might be best to centralize the metadata
storage to Solr or HBase to reduce number of IO calls to namenode for manage
applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
