Naganarasimha G R commented on YARN-3045:

Hi [~djp]
bq. 1.what we want to differentiate here is what kind of events are critical 
(so writer client in TimelineCollector could flush to backend storage after 
written them) and what kinds of events are not so critical
Well was aware that priority was not to differentiate the containers but for 
the events of it, but i thought you mentioned for the purpose of better 
querying rather than the purpose of writing it. I have not gone through the 
writer code completely but is there any caching which you want to flush if the 
event priority is high ? Also was thinking whether we need to change the 
Writer/Collector API to mention the criticality of the event being published?

bq. From an initiative thinking, some important app/container events include: 
unimportant events could include: APPLICATION_INITED, 
So from NM side we want to publish events for ApplicationEntity and 
ContainerEntity, but based on the title of this jira i thought scope of this 
jira is to handle only ContainerEntities from NM side, is it better to handle 
events related Application entities specific to a given NM in another Jira? but 
i can try to ensure required foundation is done in NM side in this jira as part 
of your other comments, Thoughts? 
Also event has just id but NM related Application events will have the same 
event ID in different NM's so would it be something like 

bq. 2. We should have some handy method to turn these app/container events to 
TimelineEvent and publish these events in a consensus way rather than publish 
one type of event with one method.
bq. 3. We don't need to create new container events but should log existing 
YARN app/container events that happen in NM. If we really think some important 
events are missing in YARN, we can have futher discussions later after timeline 
service v2 in good shape.
+1 for this thought, had the same initial hitch as in future if we add more 
events than unnecessary create event and methods in publisher, but for the 
initial version thought will have approach similar to RM and ATSV1. But i feel 
better to handle now than refactor later on. But i can think of couple of 
approaches here 
# Approach as you mentioned inside the app/container transitions in the NM side 
publish the event containing the container/app information. May be in some 
cases like creation of app or container caller can publish the events (like 
Container created so as to capture the creation time rather than )
# In ContainerEventDispatcher,ApplicationEventDispatcher & rsrcLocalizationSrvc 
after handling it can by default call different handlers of 
NMTimeLinePublisher(inner classes) to handle the respective events. Specific 
req events can be handled and others can be just ignored.
# Source itself can create the entity and the event object and 
NMTimelinePublisher can expose a method to take timeline objects add it to 
Async Dispatcher and event handler will just call the client to publish the 

bq. 4. It looks like NMTimelinePublisher should be used by ContainerManager, 
Container, ResourceLocalizationService and Log Handler. Move it to NMContext 
should be convenient to use for other components.
Will take care based on the approach we take as per prev step.

bq. 5. Container Resource Usage event may not be necessary given we already 
have metrics update and will do aggregation according to metrics update.bq. 
1.what we want to differentiate here is what kind of events are critical (so 
writer client in TimelineCollector could flush to backend storage after written 
them) and what kinds of events are not so critical
Was not clear about the comment, IIRC Zhijjie in the meeting also mentioned 
that i am handling removing threaded model of publishing container metrics 
statistics as part of this jira. May be i am missing some other jira which you 
are already working on, may be can you englighten me about it ?

> [Event producers] Implement NM writing container lifecycle events to ATS
> ------------------------------------------------------------------------
>                 Key: YARN-3045
>                 URL: https://issues.apache.org/jira/browse/YARN-3045
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3045-YARN-2928.002.patch, 
> YARN-3045-YARN-2928.003.patch, YARN-3045-YARN-2928.004.patch, 
> YARN-3045-YARN-2928.005.patch, YARN-3045-YARN-2928.006.patch, 
> YARN-3045.20150420-1.patch
> Per design in YARN-2928, implement NM writing container lifecycle events and 
> container system metrics to ATS.

This message was sent by Atlassian JIRA

Reply via email to