Naganarasimha G R commented on YARN-3995:

Thanks for the comments [~sjlee0], 
IIUC 2nd point is continuation of the first idea right ?
bq. I am not too knowledgeable about the NM and so not sure if this is 
{{PerNodeTimelineCollectorsAuxService}} can take this responsibility so i don't 
see any problem to it with NM, right ?

I can think of little modification on top of your idea,
* Once NM notifies the Auxillary service that the app is finished (by container 
finished call in the existing way), {{PerNodeTimelineCollectorsAuxService}} can 
add move this collector to a  zombie collector Map. 
* This map stores the last event published time for the zombie collector. 
* We can have one thread running to check which zombie collector is inactive 
for configurable time period and then remove it

 Thus none of the events are lost till the end. like we can keep this period as 
2 mins and if the collector in the zombie list not active for 2 mins then 
remove it and close it   ?

> Some of the NM events are not getting published due race condition when AM 
> container finishes in NM 
> ----------------------------------------------------------------------------------------------------
>                 Key: YARN-3995
>                 URL: https://issues.apache.org/jira/browse/YARN-3995
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
> As discussed in YARN-3045:  While testing in TestDistributedShell found out 
> that few of the container metrics events were failing as there will be race 
> condition. When the AM container finishes and removes the collector for the 
> app, still there is possibility that all the events published for the app by 
> the current NM and other NM are still in pipeline, 

This message was sent by Atlassian JIRA

Reply via email to