[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369782#comment-15369782 ] Hudson commented on YARN-3034: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-3034. Implement RM starting its timeline collector. Contributed by (sjlee: rev 5e3d9a477b5a7f8978a93d99fc0b4e8869193ad0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/timelineservice/RMTimelineCollector.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java > [Collector wireup] Implement RM starting its timeline collector > --- > > Key: YARN-3034 > URL: https://issues.apache.org/jira/browse/YARN-3034 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Naganarasimha G R > Fix For: YARN-2928 > > Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, > YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, > YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch > > > Per design in YARN-2928, implement resource managers starting their own ATS > writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377920#comment-14377920 ] Junping Du commented on YARN-3034: -- Thanks [~Naganarasimha] for updating the patch! Latest patch LGTM. [~zjshen] and [~sjlee0], do you have further comments? If not, I will go ahead and commit it today. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377589#comment-14377589
]
Hadoop QA commented on YARN-3034:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706868/YARN-3024.20150324-1.patch
against trunk revision c6c396f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7089//console
This message is automatically generated.
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch,
YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch,
YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377604#comment-14377604 ] Naganarasimha G R commented on YARN-3034: - [~zjshen], I have uploaded the patch with the changes which you mentioned for the configuration. Please review. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378364#comment-14378364 ] Zhijie Shen commented on YARN-3034: --- bq. so that ATS V1 and V2 are less coupled and removal of SMP once completely deprecated is smoother Exactly. The last patch looks good to me. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378549#comment-14378549 ] Naganarasimha G R commented on YARN-3034: - Thanks for reviewing [~djp],[~zjshen],[~sjlee0],[~vinodkv] [~gtCarrera9] and [~djp] for committing the patch. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Fix For: YARN-2928 Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378238#comment-14378238 ] Sangjin Lee commented on YARN-3034: --- LGTM. Thanks! [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376150#comment-14376150 ] Sangjin Lee commented on YARN-3034: --- LGTM. Let's wait to hear from Zhijie. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376200#comment-14376200 ] Junping Du commented on YARN-3034: -- Thanks [~Naganarasimha] for updating the patch! bq. Also, we should add a warning message log if user put something illegal here or it just get silent without any warn. This i feel is not required as we don't do this for any other configuration and also we have clearly captured the possible values in the yarn-default.xml. Most configurations get loaded as boolean value or int number. Some String configuration is for loading class, so ClassNotFound will get throw immediately if name is wrong. Here it belongs a different case, so I still suggest to add some check and warning here. For context info, [~zjshen], can we put that work on your patch in YARN-3040? Or you suggest something else? [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376415#comment-14376415 ] Naganarasimha G R commented on YARN-3034: - Also [~zjshen], earlier thought process to expose RMTimelineCollector to RM and it's context, was to gradually replace SystemMetricsPublisher with RMTimelineCollector, as i felt once we deprecate completely remove ATSV1, we might not require much of the functionality of SystemMetricsPublisher and it will be just delegating the calls to RMTimelineCollector. your thoughts ? [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376498#comment-14376498
]
Naganarasimha G R commented on YARN-3034:
-
Thanks for the comments [~zjshen],
bq. and in this approach, I don't think we should couple RMTimelineCollector
and SystemMetricsPublisher. Keeping SystemMetricsPublisher separate, we can
easily deprecate and even remove it from the code base later.
May be i am missing something here , If RM or RM context is not aware then only
way RMTimelineCollector can be invoked is through SystemMetricsPublisher's
(SMP) public methods like appCreated, appFinished,appAttemptRegistered or
RMTimelineCollector can have its own event handler and during initialization
SMP can select the event handler present in its class or of
RMTimelineCollector. But still there will be dependency of event source calling
public methods of SMP . So i feel it will not be smoother to deprecate and
remove SystemMetricsPublisher's as it will have code for creation of
RMTimelineCollector, sending events to RMTimelineCollector to publish to ATS V2.
bq. Moreover, we can keep the existing config as what it is now, and create a
new config to control starting v2 RM writing data stack.
IMHO i feel the current config is better because in ATS V2 container events are
planned to be moved to NM side (YARN-3045), So in NM side too we req a
configuration but we cannot use the existing one
{{yarn.resourcemanager.system-metrics-publisher.enabled}} as it indicates
more like RM side configuration only
Approach specified in the patch uses a single config for both NM and RM
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch,
YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376446#comment-14376446 ] Zhijie Shen commented on YARN-3034: --- bq. so i think its not an incompatible change. Please provide your opinion on the same. Sorry, I missed that piece. bq. IIUC SystemMetricsPublisher.publish*Event methods can determine which version of ATS to publish and can post it accordingly ? I meant in the current approach SystemMetricsPublisher can be self contained. RMTimelineCollector can be a private stuff in SystemMetricsPublisher, constructed and started there. It's not necessary to be visible in RM and its context objects. bq. we might not require much of the functionality of SystemMetricsPublisher and it will be just delegating the calls to RMTimelineCollector. I'm not sure about if there's previous discussion about the way for RM to put entities, but this approach sound cleaner, and in this approach, I don't think we should couple RMTimelineCollector and SystemMetricsPublisher. Keeping SystemMetricsPublisher separate, we can easily deprecate and even remove it from the code base later. Moreover, we can keep the existing config as what it is now, and create a new config to control starting v2 RM writing data stack. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376575#comment-14376575
]
Zhijie Shen commented on YARN-3034:
---
bq. then only way RMTimelineCollector can be invoked is through
SystemMetricsPublisher's (SMP) public methods
Oh, probably I misunderstood your intention. I used to think this is way that
you want to do put the data into RMTimelineCollector. So in this case, we could
put RMTimelineCollector inside SystemMetricsPublisher, and whereas we invoke
timeline client, we call RMTimelineCollector for v2.
According to this comments, it seems that you want to create a separate stack
to put entities into RMTimelineCollector, right? If so, the current design
makes sense.
bq. So in NM side too we req a configuration but we cannot use the existing one
I meant we keep {{yarn.resourcemanager.system-metrics-publisher.enabled}} for
v1 SystemMetricsPublisher. For v2, both RM and NM reads
{{yarn.system-metrics-publisher.enabled}}? No need to have v1/v2 flag?
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch,
YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376229#comment-14376229
]
Naganarasimha G R commented on YARN-3034:
-
Thanks [~sjlee0] [~djp] for the reviews,
{{ so I still suggest to add some check and warning here.}} : well currently
i have logged a warning message as {{RMTimelineCollector has not been
configured to publish System Metrics in ATS V2}} if it is not configured to
publish system metrics for ATS v2. will that suffice ?
bq. Zhijie Shen, can we put that work on your patch in YARN-3040? Or you
suggest something else?
We can do it in 2 ways ,
* as Zhijie suggested
[earlier|https://issues.apache.org/jira/browse/YARN-3034?focusedCommentId=14372342page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14372342],
we can handle it in a separate jira
* can handle as part of YARN-3044 (which i am working on )
I would prefer for the former one as it would be simpler to review. Please
provide your opinion
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch,
YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376228#comment-14376228 ] Zhijie Shen commented on YARN-3034: --- Let me elaborate my previous comments. In YARN-3040, I'm working on the issue to make the context info available in app-level collector, such that when we use timeline client to put entity inside AM and NM, the entity will be automatically associated to this context. This jiar is to create RM collector. To achieve the similar thing, RM collector should have the context info available too. RM has all this information available (should be inside RMApp), such that RM collector needs to make sure this information is available in some when when putting an entity. I'm okay if you want to exclude this work here, and I'll file a separate jira for it. However, I want to exclude it from YARN-3040 to prevent the patch there growing even bigger. That one is required to unblock the framework to write their specific data, and I wish it could get in asap. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376349#comment-14376349
]
Naganarasimha G R commented on YARN-3034:
-
Thanks for your comments [~zjshen]
bq. RM_SYSTEM_METRICS_PUBLISHER_ENABLED - SYSTEM_METRICS_PUBLISHER_ENABLED is
an incompatible change. :
This i incorporated based on [~vinodkv]'s
[comment|https://issues.apache.org/jira/browse/YARN-3034?focusedCommentId=14360797page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14360797],
And also i have added old keys as part of {{addDeprecatedKeys}}, so i think
its not an incompatible change. Please provide your opinion on the same.
bq. RMTimelineCollector doesn't need to be exposed to RM and it's context. It
seems to be enough to construct it inside SystemMetricsPublisher only.
IIUC SystemMetricsPublisher.publish*Event methods can determine which version
of ATS to publish and can post it accordingly ?
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch,
YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376256#comment-14376256 ] Zhijie Shen commented on YARN-3034: --- Some comments about the patch: 1. RM_SYSTEM_METRICS_PUBLISHER_ENABLED - SYSTEM_METRICS_PUBLISHER_ENABLED is an incompatible change. 2. RMTimelineCollector doesn't need to be exposed to RM and it's context. It seems to be enough to construct it inside SystemMetricsPublisher only. bq. I would prefer for the former one as it would be simpler to review. Please provide your opinion I filed a separate Jira: YARN-3390 [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377196#comment-14377196 ] Naganarasimha G R commented on YARN-3034: - Hi [~zjshen], bq. According to this comments, it seems that you want to create a separate stack to put entities into RMTimelineCollector, right? If so, the current design makes sense. yes I wanted to create a separate stack similar to SystemMetricsPublisher, so that ATS V1 and V2 are less coupled and removal of SMP once completely deprecated is smoother bq. yarn.resourcemanager.system-metrics-publisher.enabled for v1 SystemMetricsPublisher. For v2, both RM and NM reads yarn.system-metrics-publisher.enabled? No need to have v1/v2 flag? On after thoughts i feel this approach is better as once we deprecate SMP then there will be unnecessary additional configuration of which version type, which would not be required. If all are fine then will move back to the approach as mentioned by Zhijie. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372342#comment-14372342 ] Zhijie Shen commented on YARN-3034: --- Kindly raise one issue that's not covered in YARN-3040: RMTimelineCollector needs to have the context info setup when being constructed and started. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372468#comment-14372468 ] Naganarasimha G R commented on YARN-3034: - Ok [~zjshen], will have a look at YARN-3040 and raise the issue, but can you or [~djp] review the current patch and commit if fine. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371812#comment-14371812
]
Hadoop QA commented on YARN-3034:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705983/YARN-3034.20150320-1.patch
against trunk revision a6a5aae.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7048//console
This message is automatically generated.
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch,
YARN-3034.20150320-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371680#comment-14371680 ] Naganarasimha G R commented on YARN-3034: - Thanks [~sjlee0] surprisingly yest faced problem it dint update, but today seems to be working fine, [~djp], ??Also, we should add a warning message log if user put something illegal here or it just get silent without any warn.?? This i feel is not required as we don't do this for any other configuration and also we have clearly captured the possible values in the yarn-default.xml instead will log if RMTimelineCollector is configured, ur opinion? For other comments, will include them in the patch [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370299#comment-14370299 ] Sangjin Lee commented on YARN-3034: --- [~Naganarasimha], I do see the commit, and I'm able to pull it (dda84085cabd8fdf143b380e54e1730802fd9912). You might want to try again. [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369872#comment-14369872 ] Junping Du commented on YARN-3034: -- I have commit YARN- in. [~Naganarasimha], would you mind rebase the patch and address my minor comments? Thanks! [Collector wireup] Implement RM starting its timeline collector --- Key: YARN-3034 URL: https://issues.apache.org/jira/browse/YARN-3034 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Naganarasimha G R Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch Per design in YARN-2928, implement resource managers starting their own ATS writers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[
https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370041#comment-14370041
]
Naganarasimha G R commented on YARN-3034:
-
Hi [~djp], thanks for informing, i was trying to update from YARN-2928 branch
but unfortunately was not getting the modifications from YARN-,
tried using {{git pull -v --progress origin}} and also pulling through tools
like git-cola. Anyway will try to clone the branch tomorrow and try again, mean
while can you check once whether you are able to get the modifications in
YARN-2928 branch ?
[Collector wireup] Implement RM starting its timeline collector
---
Key: YARN-3034
URL: https://issues.apache.org/jira/browse/YARN-3034
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Naganarasimha G R
Attachments: YARN-3034-20150312-1.patch, YARN-3034.20150205-1.patch,
YARN-3034.20150316-1.patch, YARN-3034.20150318-1.patch
Per design in YARN-2928, implement resource managers starting their own ATS
writers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
