[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[
https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493667#comment-14493667
]
Hadoop QA commented on YARN-3390:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725163/YARN-3390.1.patch
against trunk revision b5a0b24.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7327//console
This message is automatically generated.
> RMTimelineCollector should have the context info of each app
>
>
> Key: YARN-3390
> URL: https://issues.apache.org/jira/browse/YARN-3390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-3390.1.patch
>
>
> RMTimelineCollector should have the context info of each app whose entity
> has been put
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493659#comment-14493659 ] Zhijie Shen commented on YARN-3390: --- Thanks for sharing your idea, Naga! Thinking about RM side use case again, an RMTimelineCollectorManager is better than an RMTimelineCollector with a map of appId -> collector context info. RMTimelineCollectorManager is similar to what we have to NM, i.e., TimelineCollectorManager. However, it doesn't need IPC interface and web server, but maintain the collection of app-level collector an RM side. The benefit is that we can reuse most of the code of app-level collector as well as collector manager, uniforming the way we write the timeline entities. The approach is to make TimelineCollectorManager abstract and contain the common logic of managing collector lifecycles, and have NMTimelineCollectorManager extend TimelineCollectorManager by adding the IRP interface and start the web server, and have RMTimelineCollectorManager extend TimelineCollectorManager by sourcing RMContext for the context info and coupling with RMApp to add/remove the app-level collector. I created patch for this approach. > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: YARN-3390.1.patch > > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[
https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394059#comment-14394059
]
Naganarasimha G R commented on YARN-3390:
-
Thanks for the feedback [~zjshen] & [~sjlee0],
bq. either pass in the context per call or have a map of app id to context. I
would favor the latter approach because it'd be easier on the perspective of
callers of putEntities().
I too agree it will be easier easier on the perspective of callers of
putEntities() but if we favor for map of {{app id to context}}
* implicit assumption would be that {{putEntities(TimelineEntities ) }} will be
for same appId(/will have have the same context)
* TimelineEntities as such do not have appID explicitly, so planning to modify
{{TimelineCollector.getTimelineEntityContext()}} to
{{TimelineCollector.getTimelineEntityContext(TimelineEntity.Identifier id)}}
and subclasses of TimelineCollector can take care of mapping the Id to the
Context (via AppId) if required.
* code of {{putEntities(TimelineEntities)}} would look something like
{code}
Iterator iterator = entities.getEntities().iterator();
TimelineEntity next = (iterator.hasNext())?iterator.next():null;
if(null!=next) {
TimelineCollectorContext context =
getTimelineEntityContext(next.getIdentifier());
return writer.write(context.getClusterId(), context.getUserId(),
context.getFlowId(), context.getFlowRunId(), context.getAppId(),
entities);
}
{code}
If its ok then shall i work on it ?
> RMTimelineCollector should have the context info of each app
>
>
> Key: YARN-3390
> URL: https://issues.apache.org/jira/browse/YARN-3390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>
> RMTimelineCollector should have the context info of each app whose entity
> has been put
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393643#comment-14393643 ] Zhijie Shen commented on YARN-3390: --- bq. I would favor the latter approach +1 > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393615#comment-14393615 ] Sangjin Lee commented on YARN-3390: --- I think we need to either pass in the context per call or have a map of app id to context. I would favor the latter approach because it'd be easier on the perspective of callers of putEntities(). > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393139#comment-14393139 ] Zhijie Shen commented on YARN-3390: --- I think it makes sense to generalize TimelineEntityContext from a single app's context to the app -> context map. Reader may need this map too. I'll fix the problem after YARN-3391 is done. > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[
https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393095#comment-14393095
]
Naganarasimha G R commented on YARN-3390:
-
Hi [~zjshen], [~djp] & [~sjlee0]
{{TimelineCollector.getTimelineEntityContext()}} interface will not be suitable
for the RMTimelineCollector as it will be posting/putting entities for multiple
apps, appattempts and containers. Hence was initially planning to modify this
method to take a {{TimelineEntity.Identifier}} as a parameter and @
RMTimelineCollector planning to hold a map of {{TimelineEntity.Identifier to
AppId}} and another Map to hold {{AppId to TimelineEntityContext}} (required as
context is created per app when appCreatedEvent occurs).
But one more conflict which i can see is AppLevelTimelineCollector is specific
for a app, so invoking {{getTimelineEntityContext}} in
{{getTimelineEntityContext(TimelineEntities ,Ugi)}} is fine because all the
entities which are posted can be assumed to have same context as they belong to
a single app but in a general case (like RMTimelineCollector) its not
guaranteed that all TimelineEntities belong to same app(i.e. TimelineEntities
might have diff context). so would it be better to change the interface of
{{TimelineCollector.putEntities)}} to accept the {{TimelineEntityContext}} as
parameter and remove {{TimelineCollector.getTimelineEntityContext()}} method
from interface ? please share your opinion...
> RMTimelineCollector should have the context info of each app
>
>
> Key: YARN-3390
> URL: https://issues.apache.org/jira/browse/YARN-3390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>
> RMTimelineCollector should have the context info of each app whose entity
> has been put
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376684#comment-14376684 ] Zhijie Shen commented on YARN-3390: --- It shouldn't. Storage layer implementations only depends on the writer interface, which is covered in YARN-3040. > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376670#comment-14376670 ] Li Lu commented on YARN-3390: - Hi [~zjshen], could you please confirm that this JIRA will also block all storage layer implementations? Or we can proceed after YARN-3040 is in? Thanks! > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3390) RMTimelineCollector should have the context info of each app
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376323#comment-14376323 ] Naganarasimha G R commented on YARN-3390: - Hi [~zjshen], Shall i work on this jira ? as i can utilize the same in YARN-3044 ? > RMTimelineCollector should have the context info of each app > > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
