Junping Du commented on YARN-3039:

Thanks [~sjlee0]! 
Providing an end-to-end flow below which could be helpful for your review:
- When AM get launched, NM auxiliary service will add a new aggregator service 
to aggregatorCollection (per Node) for necessary binding work. 
aggregatorCollection also has a client for AggregatorNodeManagerProtocol to 
notify NM on new app aggregator registered and detailed address.
- When NM get notified, it will update registeredAggregators list (for all 
local app aggregators), and notify RM in next heartbeat.
- RM received registeredAggregators from NM, it will update its aggregators 
- Next time, when other NMs and AM heartbeat with RM, it will provide 
aggregatorInfo in heartbeat response (for AM, it is through AllocationResponse).
- AM of DS has AMRMClientAsync which heartbeat with RM so can receive updated 
aggregator address periodically. With registered a callback for listening 
aggregator address update, it can update address of TimelineClient in a 
thread-safe way. 
- AM call timeline operations in a non-blocking way (for not hanging there as 
deadlock), currently is wrapping with a new thread but will be improved later 
(in another JIRA) for saving resource of threads.
- TimelineClient (consuming v2 service) is looping in retry logic until get 
correct address that being set by AM.

> [Aggregator wireup] Implement ATS app-appgregator service discovery
> -------------------------------------------------------------------
>                 Key: YARN-3039
>                 URL: https://issues.apache.org/jira/browse/YARN-3039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: Service Binding for applicationaggregator of ATS 
> (draft).pdf, Service Discovery For Application Aggregator of ATS (v2).pdf, 
> YARN-3039-no-test.patch, YARN-3039-v2-incomplete.patch, 
> YARN-3039-v3-core-changes-only.patch, YARN-3039-v4.patch
> Per design in YARN-2928, implement ATS writer service discovery. This is 
> essential for off-node clients to send writes to the right ATS writer. This 
> should also handle the case of AM failures.

This message was sent by Atlassian JIRA

Reply via email to