Zhijie Shen commented on YARN-3332:

It sounds a great proposal, thanks Vinod! I quick thought about the publishing 
channel of the collected statistics. I'm not sure how different the access 
pattern would be, but just thinking it out loudly, is it possible reuse the 
timeline service to distribute the node statistics, getting rid of maintaining 
different but similar interfaces (or multiple data flow channels). On step 
further, we can make the timeline service the main bus to transmit metrics from 
A to B.

> [Umbrella] Unified Resource Statistics Collection per node
> ----------------------------------------------------------
>                 Key: YARN-3332
>                 URL: https://issues.apache.org/jira/browse/YARN-3332
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: Design - UnifiedResourceStatisticsCollection.pdf
> Today in YARN, NodeManager collects statistics like per container resource 
> usage and overall physical resources available on the machine. Currently this 
> is used internally in YARN by the NodeManager for only a limited usage: 
> automatically determining the capacity of resources on node and enforcing 
> memory usage to what is reserved per container.
> This proposal is to extend the existing architecture and collect statistics 
> for usage b​eyond​ the existing use­cases.
> Proposal attached in comments.

This message was sent by Atlassian JIRA

Reply via email to