[
https://issues.apache.org/jira/browse/YARN-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317705#comment-16317705
]
Eric Yang commented on YARN-7605:
---------------------------------
[~jianhe] {quote}
IIUC, the code is load if app is not running. This can be that app is pending
or finished, it may still hit hdfs a lot. For getStatus kinda API, the consumer
is always calling it in a loop. This will get very bad if hdfs is down or
during failover, the api call will be spinning and quickly overload RM. IMO, we
may need a better solution for serving persistent app status. The current
approach may create a bottleneck accidentally.
{quote}
What api would you recommend to retrieve the status and spec combined object.
The code used to create a new service object, and only set service name, status
and time remaining. There is a lot of missing information, which can confuse
user. This is the reason that I changed it to return the spec from HDFS. If
spec was stored in solr or hbase with memory caching, it will reduce the
bottleneck. However, those proposed ideas was not well received during the
early implementation. Therefore, there isn't many options here to make this
more robust at this time. I am open to suggestions.
{quote}
How is it retrieving it from AM ? Only RM knows the app's remaining time
{quote}
It appears that the application master stores the information to update the
status of the service. I am not sure how it was computed. Remaining time is
also updated. One could argue that it may have delay in comparison to current
time, if this is the case, I can put the code back. However, I didn't find the
remaining time to be not current when retrieved via AMproxy.
> Implement doAs for Api Service REST API
> ---------------------------------------
>
> Key: YARN-7605
> URL: https://issues.apache.org/jira/browse/YARN-7605
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: yarn-native-services
>
> Attachments: YARN-7605.001.patch, YARN-7605.004.patch,
> YARN-7605.005.patch, YARN-7605.006.patch, YARN-7605.007.patch,
> YARN-7605.008.patch, YARN-7605.009.patch, YARN-7605.010.patch,
> YARN-7605.011.patch, YARN-7605.012.patch, YARN-7605.013.patch,
> YARN-7605.014.patch
>
>
> In YARN-7540, all client entry points for API service is centralized to use
> REST API instead of having direct file system and resource manager rpc calls.
> This change helped to centralize yarn metadata to be owned by yarn user
> instead of crawling through every user's home directory to find metadata.
> The next step is to make sure "doAs" calls work properly for API Service.
> The metadata is stored by YARN user, but the actual workload still need to be
> performed as end users, hence API service must authenticate end user kerberos
> credential, and perform doAs call when requesting containers via
> ServiceClient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
