[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-16 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313322
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 16/Sep/19 22:05
Start Date: 16/Sep/19 22:05
Worklog Time Spent: 10m 
  Work Description: asfgit commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 313322)
Time Spent: 3h  (was: 2h 50m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-16 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313307
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 16/Sep/19 21:30
Start Date: 16/Sep/19 21:30
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit 
container health metrics when running in cluster mode
URL: 
https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1)
 Report
   > Merging 
[#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc)
 will **increase** coverage by `0.08%`.
   > The diff coverage is `70.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2729  +/-   ##
   
   + Coverage 44.99%   45.08%   +0.08% 
   - Complexity 8742 8759  +17 
   
 Files  1884 1886   +2 
 Lines 7029570377  +82 
 Branches   7715 7718   +3 
   
   + Hits  3162931726  +97 
   + Misses3573535709  -26 
   - Partials   2931 2942  +11
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=)
 | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | |
   | 
[...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=)
 | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==)
 | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.35% <50%> (-0.91%)` | `28 <1> (-1)` | |
   | 
[...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==)
 | `78.57% <78.57%> 

[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-16 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313297
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 16/Sep/19 20:54
Start Date: 16/Sep/19 20:54
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r324880686
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerHealthMetricsService.java
 ##
 @@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.cluster;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.MemoryMXBean;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.google.common.util.concurrent.AbstractScheduledService;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.sun.management.OperatingSystemMXBean;
+import com.typesafe.config.Config;
+
+import org.apache.gobblin.metrics.ContextAwareGauge;
+import org.apache.gobblin.metrics.RootMetricContext;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/**
+ * A utility class that periodically emits system level metrics that report 
the health of the container.
+ * Reported metrics include CPU/Memory usage of the JVM, system load, file 
descriptors used etc.
 
 Review comment:
   Fixed the javadoc.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 313297)
Time Spent: 2h 40m  (was: 2.5h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-16 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313294
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 16/Sep/19 20:48
Start Date: 16/Sep/19 20:48
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r324775801
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerHealthMetricsService.java
 ##
 @@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.cluster;
+
+import java.lang.management.ManagementFactory;
+import java.lang.management.MemoryMXBean;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.google.common.util.concurrent.AbstractScheduledService;
+import com.google.common.util.concurrent.AtomicDouble;
+import com.sun.management.OperatingSystemMXBean;
+import com.typesafe.config.Config;
+
+import org.apache.gobblin.metrics.ContextAwareGauge;
+import org.apache.gobblin.metrics.RootMetricContext;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/**
+ * A utility class that periodically emits system level metrics that report 
the health of the container.
+ * Reported metrics include CPU/Memory usage of the JVM, system load, file 
descriptors used etc.
 
 Review comment:
   Which parameters are concerning with file descriptors?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 313294)
Time Spent: 2.5h  (was: 2h 20m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-13 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=312178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312178
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 16:53
Start Date: 13/Sep/19 16:53
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit 
container health metrics when running in cluster mode
URL: 
https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1)
 Report
   > Merging 
[#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc)
 will **increase** coverage by `0.09%`.
   > The diff coverage is `70.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2729  +/-   ##
   
   + Coverage 44.99%   45.08%   +0.09% 
   - Complexity 8742 8758  +16 
   
 Files  1884 1886   +2 
 Lines 7029570377  +82 
 Branches   7715 7718   +3 
   
   + Hits  3162931730 +101 
   + Misses3573535705  -30 
   - Partials   2931 2942  +11
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=)
 | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | |
   | 
[...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=)
 | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==)
 | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | |
   | 
[...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==)
 | `78.57% <78.57%> 

[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309523
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 04:31
Start Date: 10/Sep/19 04:31
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322548013
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
 ##
 @@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
 this.services.addAll(suite.getServices());
 
 this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config, 
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED, 
false)) {
 
 Review comment:
   Sorry I should not mentioned `TaskRunnerSuiteBase`. What I meant to say is, 
does it make more sense to add this service inside `getServices` method in 
`GobblinTaskRunner` ?  
   
   The comment of `getServices` is :
   ```Creates and returns a {@link List} of additional {@link Service}s that 
should be run in this {@link GobblinTaskRunner}. Sub-classes that need 
additional {@link Service}s to run, should override this method```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309523)
Time Spent: 2h 10m  (was: 2h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309402
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:22
Start Date: 10/Sep/19 00:22
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322507069
 
 

 ##
 File path: 
gobblin-cluster/src/test/java/org/apache/gobblin/cluster/ContainerHealthMetricsServiceTest.java
 ##
 @@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.cluster;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class ContainerHealthMetricsServiceTest {
+
+  @Test
+  public void testRunOneIteration() throws Exception {
+Config config = ConfigFactory.empty();
+ContainerHealthMetricsService service = new 
ContainerHealthMetricsService(config);
+service.runOneIteration();
+long processCpuTime1 = service.processCpuTime.get();
+Thread.sleep(10);
+service.runOneIteration();
+long processCpuTime2 = service.processCpuTime.get();
+Assert.assertTrue(processCpuTime1 < processCpuTime2);
+  }
+}
 
 Review comment:
   Added newline.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309402)
Time Spent: 2h  (was: 1h 50m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309401
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:22
Start Date: 10/Sep/19 00:22
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322507066
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
 ##
 @@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs, 
Path destFilePath, Loca
 
 return environmentVariableMap;
   }
+
 
 Review comment:
   Added javadoc.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309401)
Time Spent: 1h 50m  (was: 1h 40m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309399
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:21
Start Date: 10/Sep/19 00:21
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506877
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
 ##
 @@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
 this.services.addAll(suite.getServices());
 
 this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config, 
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED, 
false)) {
 
 Review comment:
   Fixed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309399)
Time Spent: 1h 40m  (was: 1.5h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309398
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:20
Start Date: 10/Sep/19 00:20
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506849
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
 ##
 @@ -167,4 +167,5 @@
 
   public static final String HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 
GOBBLIN_CLUSTER_PREFIX + "job.stoppingStateTimeoutSeconds";
   public static final long DEFAULT_HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 
300;
+  public static final String CONTAINER_HEALTH_METRICS_SERVICE_ENABLED = 
GOBBLIN_CLUSTER_PREFIX + "container.health.metrics.service.enabled" ;
 }
 
 Review comment:
   Added a default.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309398)
Time Spent: 1.5h  (was: 1h 20m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309396
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:20
Start Date: 10/Sep/19 00:20
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506776
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
 ##
 @@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
 this.services.addAll(suite.getServices());
 
 this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config, 
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED, 
false)) {
 
 Review comment:
   The TaskRunnerSuiteBase is an abstract class with two implementations, 
process model and thread model. Wanted to leave getServices() in 
TaskRunnerSuiteBase as an abstract method so as not to change the contract of 
the class.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309396)
Time Spent: 1h 10m  (was: 1h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309397
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:20
Start Date: 10/Sep/19 00:20
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506814
 
 

 ##
 File path: 
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java
 ##
 @@ -954,4 +954,5 @@
*/
   public static final String AVRO_SCHEMA_CHECK_STRATEGY = 
"avro.schema.check.strategy";
   public static final String AVRO_SCHEMA_CHECK_STRATEGY_DEFAULT = 
"org.apache.gobblin.util.schema_check.AvroSchemaCheckDefaultStrategy";
+
 
 Review comment:
   Fixed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309397)
Time Spent: 1h 20m  (was: 1h 10m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309389
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 10/Sep/19 00:06
Start Date: 10/Sep/19 00:06
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322504103
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
 ##
 @@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs, 
Path destFilePath, Loca
 
 return environmentVariableMap;
   }
+
+  public static String getContainerNum(String containerId) {
+return "container-" + containerId.substring(containerId.lastIndexOf("_") + 
1);
 
 Review comment:
   We want the substring starting from the char immediately following the last 
"_". e.g. if containerId = "container_e94_1567552810874_2132400_01_01", we 
want to return 
   "container-01". Added javadoc to make the behavior clear.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309389)
Time Spent: 1h  (was: 50m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309316
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m 
  Work Description: shirshanka commented on pull request #2729: 
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481749
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
 ##
 @@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs, 
Path destFilePath, Loca
 
 return environmentVariableMap;
   }
+
 
 Review comment:
   javadoc missing. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309316)
Time Spent: 40m  (was: 0.5h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309317
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m 
  Work Description: shirshanka commented on pull request #2729: 
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481788
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
 ##
 @@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs, 
Path destFilePath, Loca
 
 return environmentVariableMap;
   }
+
+  public static String getContainerNum(String containerId) {
+return "container-" + containerId.substring(containerId.lastIndexOf("_") + 
1);
 
 Review comment:
   why are we doing + 1?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309317)
Time Spent: 40m  (was: 0.5h)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309320
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m 
  Work Description: shirshanka commented on pull request #2729: 
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322477750
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
 ##
 @@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
 this.services.addAll(suite.getServices());
 
 this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config, 
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED, 
false)) {
 
 Review comment:
   Create a single static constant for the default value for this config, so 
you don't have to say false in two places?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309320)
Time Spent: 50m  (was: 40m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309318
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m 
  Work Description: shirshanka commented on pull request #2729: 
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322474783
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
 ##
 @@ -167,4 +167,5 @@
 
   public static final String HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 
GOBBLIN_CLUSTER_PREFIX + "job.stoppingStateTimeoutSeconds";
   public static final long DEFAULT_HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 
300;
+  public static final String CONTAINER_HEALTH_METRICS_SERVICE_ENABLED = 
GOBBLIN_CLUSTER_PREFIX + "container.health.metrics.service.enabled" ;
 }
 
 Review comment:
   no default?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309318)
Time Spent: 50m  (was: 40m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309319
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m 
  Work Description: shirshanka commented on pull request #2729: 
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481849
 
 

 ##
 File path: 
gobblin-cluster/src/test/java/org/apache/gobblin/cluster/ContainerHealthMetricsServiceTest.java
 ##
 @@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.cluster;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class ContainerHealthMetricsServiceTest {
+
+  @Test
+  public void testRunOneIteration() throws Exception {
+Config config = ConfigFactory.empty();
+ContainerHealthMetricsService service = new 
ContainerHealthMetricsService(config);
+service.runOneIteration();
+long processCpuTime1 = service.processCpuTime.get();
+Thread.sleep(10);
+service.runOneIteration();
+long processCpuTime2 = service.processCpuTime.get();
+Assert.assertTrue(processCpuTime1 < processCpuTime2);
+  }
+}
 
 Review comment:
   missing newline at end of file. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309319)
Time Spent: 50m  (was: 40m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309134
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 18:02
Start Date: 09/Sep/19 18:02
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit 
container health metrics when running in cluster mode
URL: 
https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1)
 Report
   > Merging 
[#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `71.64%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2729  +/-   ##
   
   + Coverage 44.99%   45.02%   +0.03% 
   - Complexity 8742 8750   +8 
   
 Files  1884 1886   +2 
 Lines 7029570359  +64 
 Branches   7715 7717   +2 
   
   + Hits  3162931680  +51 
   - Misses3573535746  +11 
   - Partials   2931 2933   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=)
 | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `65.11% <0%> (-0.15%)` | `29 <0> (ø)` | |
   | 
[...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=)
 | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==)
 | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: |
   | 
[...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==)
 | `78.57% 

[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309132
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 18:01
Start Date: 09/Sep/19 18:01
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322371601
 
 

 ##
 File path: 
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java
 ##
 @@ -954,4 +954,5 @@
*/
   public static final String AVRO_SCHEMA_CHECK_STRATEGY = 
"avro.schema.check.strategy";
   public static final String AVRO_SCHEMA_CHECK_STRATEGY_DEFAULT = 
"org.apache.gobblin.util.schema_check.AvroSchemaCheckDefaultStrategy";
+
 
 Review comment:
   Empty line, another one in`GobblinApplicationMaster.java`, line 116
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309132)
Time Spent: 20m  (was: 10m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309133
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 18:01
Start Date: 09/Sep/19 18:01
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322377318
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
 ##
 @@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
 this.services.addAll(suite.getServices());
 
 this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config, 
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED, 
false)) {
 
 Review comment:
   Should we package this service as part of `TaskRunnerSuiteBase` instead of 
having a `this.services.addAll(getServices)` plus another service which is used 
for metric-reporting outside? 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309133)
Time Spent: 20m  (was: 10m)

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-09 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309091
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 09/Sep/19 17:27
Start Date: 09/Sep/19 17:27
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: 
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-875
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   This task implements a service that emits CPU/Memory health metrics from the 
JVM when running in the cluster mode. 
   
   
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Added unit test in ContainerHealthMetricsServiceTest.
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 309091)
Remaining Estimate: 0h
Time Spent: 10m

> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the 
> JVM when running in the cluster mode. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)