[
https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Lu updated YARN-5638:
------------------------
Attachment: YARN-5638-trunk.v2.patch
Thanks for the review [~rohithsharma]! I addressed most of you comments except
those two:
bq. Can happensBefore comparison method name can be changed something
meaningful ? May we can define comparator method itself.
Happens-before has very concrete meaning in distributed system theory, as
defined in Lamport's "Time, Clocks, and the Ordering of Events in a Distributed
System"
(http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf)
Here, we're assigning each collector data a timestamp, and then we use
timestamps to reason about happens before order in the system. Personally I'd
prefer using this formal definition to capture our use case here.
bq. In stamped method, I think check for both rmIdentifiers && version?
Did not quite get the point here... I think we're checking both fields?
About the design question, a pull (IIUC, pulling collector data from the RM)
based method works. The current approach offloads the burden of deciding where
collectors should run from the RM. At the early stage we can also reuse some
well established mechanisms with heartbeat. I believe most them are for
engineering reasons, though.
One thing to note is that we will not send known collector information from NMs
to the RM in *every* heartbeat. A collector's information got sent only when it
needs registration to the RM, or the RM needs resync. On the other hand, the RM
will only send related collector's information to each of the NMs.
> Introduce a collector timestamp to uniquely identify collectors creation
> order in collector discovery
> -----------------------------------------------------------------------------------------------------
>
> Key: YARN-5638
> URL: https://issues.apache.org/jira/browse/YARN-5638
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Li Lu
> Attachments: YARN-5638-trunk.v1.patch, YARN-5638-trunk.v2.patch
>
>
> As discussed in YARN-3359, we need to further identify timeline collectors'
> creation order to rebuild collector discovery data in the RM. This JIRA
> proposes to use <rm_timestamp, logical_version_number> to order collectors
> for each application in the RM. This timestamp can then be used when a
> standby RM becomes active and rebuild collector discovery data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
