[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329671#comment-14329671 ] Li Lu commented on YARN-3033: - Hi [~sjlee0], thanks for the comments! I thing I used the wrong name Application Level Aggregator inside RM here. I think a more appropriate name would be aggregator inside RM for app data. I agree that we should strictly limit the total number of this type of aggregators (one per RM seems to be reasonable for now). We may want to reuse the implementation for web server/data storage layer in aggregator collection for this aggregator as well, by simply wrap it to an aggregator collection? For the second point, yes, we can (and should always) reuse the same hbase client for all app level aggregators on the same node. [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Li Lu Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329748#comment-14329748 ] Sangjin Lee commented on YARN-3033: --- Agreed. Thanks for the clarification. In summary, I just want to make sure that we do not impose an app-level context for the RM's aggregator. [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Li Lu Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328365#comment-14328365 ] Junping Du commented on YARN-3033: -- Thanks [~gtCarrera9] for delivering a proposal here which looks pretty good. A couple of comments here: bq. In the below discussion, we start with a notion of AMs/NMs not making any assumptions on the physical locations of the aggregators an aggregator can sit next to the AM in a separate container, it can side on the local node etc. I think we should mention our best-effort to place logical app aggregator to be on the same physical node with AM containers, given the assumption that AM could consume aggregator service heavier so we want to keep data transferred in local as much as possible. The other cases that AM don't stay with logical aggregator is AM get failed over to other node but NM physical aggregator service is still alive so logic aggregator keep the same place. We'd better to mention here. bq. Depending on the number of running applications ( N_app ) and the number of nodes ( N_node ) or on the size of individual applications, we may choose different mapping policies (implemented as different aggregatorcollections). For the reason I mentioned above, we should do our best effort to keep logical aggregator stay with AM container and we don't have to have different policies for different cases. Even for the case N_app N_node, we should make sure AMs doesn't get skewed (aggregated) across the cluster. Also I didn't see the discussion on launching (or we call binding - to physical aggregator) the logical aggregator (by who and how). I assume this is the scope of this JIRA. Isn't it? [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Li Lu Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[
https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328453#comment-14328453
]
Sangjin Lee commented on YARN-3033:
---
Thanks for the well-written proposal [~gtCarrera9]! It looks fine except for
one thing IMO: whether the RM's aggregator needs to use the app-level
aggregators.
I'm not convinced that an Application Level Aggregator inside RM is needed or
beneficial. The main use case of RM writing application-related data is writing
application life-cycle events. This doesn't represent much volume for each app
(at most a few events per app). Furthermore, it does not require any
batching/aggregation of metrics of any kind. But by having the per-app
aggregators it would retain a lot of memory for the duration the apps are
alive. And it could be a significant amount of memory pressure for a big/busy
cluster. IMO, it would be a superfluous abstraction with little benefit. Does
the RM aggregator have to use it? Do you see it being a useful abstraction? If
so, how?
In my opinion, it would be far simpler and also perform better if the RM
aggregator writes data to the storage outside the app-level context.
{quote}
If N_app N_node or N_app N_node, we may consider to launch a constant
number of aggregators inside each NodeManager, so the total aggregator entities
is
bounded by the number of NMs. The reason we’d like to avoid running too many
aggregators is the pressure on the storage too many writers writing to say
HBase
RegionServers. We can override the aggregator mapping in this case.
{quote}
+1 with Junping's comment to keep the model simple. This can also be handled by
a different manner if it is for HBase. One can use a single shared HBase client
for all app level aggregators on a per-node aggregator, which would mitigate
that concern. If app level aggregators are separate processes, it's a different
story of course. Also, it's been my observation that the nodes usually
outnumber the active apps unless the apps are real tiny.
[Aggregator wireup] Implement NM starting the ATS writer companion
--
Key: YARN-3033
URL: https://issues.apache.org/jira/browse/YARN-3033
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Li Lu
Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf
Per design in YARN-2928, implement node managers starting the ATS writer
companion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328375#comment-14328375 ] Li Lu commented on YARN-3033: - Thanks [~djp]! Some quick comments: bq. I think we should mention our best-effort to place logical app aggregator to be on the same physical node with AM containers, given the assumption that AM could consume aggregator service heavier so we want to keep data transferred in local as much as possible. bq. For the reason I mentioned above, we should do our best effort to keep logical aggregator stay with AM container and we don't have to have different policies for different cases. I agree. We should make every effort to improve the locality here. Will cover this later. bq. I didn't see the discussion on launching (or we call binding - to physical aggregator) the logical aggregator (by who and how). I assume this is the scope of this JIRA. Isn't it? Yes. Let's focus on launching aggregators for the whole flow in this JIRA. We may then consider if we need a separate JIRA for the standalone mode. [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Li Lu Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320469#comment-14320469 ] Li Lu commented on YARN-3033: - OK, thanks! [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Li Lu Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318833#comment-14318833 ] Li Lu commented on YARN-3033: - Hi [~sjlee0] and [~devaraj.k], I've already started thinking on this, would you mind if I take this Jira over? Thanks! [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Devaraj K Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3033) [Aggregator wireup] Implement NM starting the ATS writer companion
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319595#comment-14319595 ] Devaraj K commented on YARN-3033: - Please take it if you have already started working on it. [Aggregator wireup] Implement NM starting the ATS writer companion -- Key: YARN-3033 URL: https://issues.apache.org/jira/browse/YARN-3033 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Devaraj K Per design in YARN-2928, implement node managers starting the ATS writer companion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
