Li Lu updated YARN-5638:
    Attachment: YARN-5638-trunk.v2.patch

Thanks for the review [~rohithsharma]! I addressed most of you comments except 
those two:

bq. Can happensBefore comparison method name can be changed something 
meaningful ? May we can define comparator method itself.
Happens-before has very concrete meaning in distributed system theory, as 
defined in Lamport's "Time, Clocks, and the Ordering of Events in a Distributed 
Here, we're assigning each collector data a timestamp, and then we use 
timestamps to reason about happens before order in the system. Personally I'd 
prefer using this formal definition to capture our use case here. 

bq. In stamped method, I think check for both rmIdentifiers && version?
Did not quite get the point here... I think we're checking both fields? 

About the design question, a pull (IIUC, pulling collector data from the RM) 
based method works. The current approach offloads the burden of deciding where 
collectors should run from the RM. At the early stage we can also reuse some 
well established mechanisms with heartbeat. I believe most them are for 
engineering reasons, though. 

One thing to note is that we will not send known collector information from NMs 
to the RM in *every* heartbeat. A collector's information got sent only when it 
needs registration to the RM, or the RM needs resync. On the other hand, the RM 
will only send related collector's information to each of the NMs. 

> Introduce a collector timestamp to uniquely identify collectors creation 
> order in collector discovery
> -----------------------------------------------------------------------------------------------------
>                 Key: YARN-5638
>                 URL: https://issues.apache.org/jira/browse/YARN-5638
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-5638-trunk.v1.patch, YARN-5638-trunk.v2.patch
> As discussed in YARN-3359, we need to further identify timeline collectors' 
> creation order to rebuild collector discovery data in the RM. This JIRA 
> proposes to use <rm_timestamp, logical_version_number> to order collectors 
> for each application in the RM. This timestamp can then be used when a 
> standby RM becomes active and rebuild collector discovery data. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to