[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313307 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 16/Sep/19 21:30 Start Date: 16/Sep/19 21:30 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1) Report > Merging [#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc) will **increase** coverage by `0.08%`. > The diff coverage is `70.42%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2729 +/- ## + Coverage 44.99% 45.08% +0.08% - Complexity 8742 8759 +17 Files 1884 1886 +2 Lines 7029570377 +82 Branches 7715 7718 +3 + Hits 3162931726 +97 + Misses3573535709 -26 - Partials 2931 2942 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=) | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==) | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.35% <50%> (-0.91%)` | `28 <1> (-1)` | | | [...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==) | `78.57% <78.57%>
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313297 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 16/Sep/19 20:54 Start Date: 16/Sep/19 20:54 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r324880686 ## File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerHealthMetricsService.java ## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.cluster; + +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryMXBean; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +import com.google.common.util.concurrent.AbstractScheduledService; +import com.google.common.util.concurrent.AtomicDouble; +import com.sun.management.OperatingSystemMXBean; +import com.typesafe.config.Config; + +import org.apache.gobblin.metrics.ContextAwareGauge; +import org.apache.gobblin.metrics.RootMetricContext; +import org.apache.gobblin.util.ConfigUtils; + + +/** + * A utility class that periodically emits system level metrics that report the health of the container. + * Reported metrics include CPU/Memory usage of the JVM, system load, file descriptors used etc. Review comment: Fixed the javadoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313297) Time Spent: 2h 40m (was: 2.5h) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=313294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313294 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 16/Sep/19 20:48 Start Date: 16/Sep/19 20:48 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r324775801 ## File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerHealthMetricsService.java ## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.cluster; + +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryMXBean; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +import com.google.common.util.concurrent.AbstractScheduledService; +import com.google.common.util.concurrent.AtomicDouble; +import com.sun.management.OperatingSystemMXBean; +import com.typesafe.config.Config; + +import org.apache.gobblin.metrics.ContextAwareGauge; +import org.apache.gobblin.metrics.RootMetricContext; +import org.apache.gobblin.util.ConfigUtils; + + +/** + * A utility class that periodically emits system level metrics that report the health of the container. + * Reported metrics include CPU/Memory usage of the JVM, system load, file descriptors used etc. Review comment: Which parameters are concerning with file descriptors? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313294) Time Spent: 2.5h (was: 2h 20m) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=312178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312178 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 13/Sep/19 16:53 Start Date: 13/Sep/19 16:53 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1) Report > Merging [#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc) will **increase** coverage by `0.09%`. > The diff coverage is `70.42%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2729 +/- ## + Coverage 44.99% 45.08% +0.09% - Complexity 8742 8758 +16 Files 1884 1886 +2 Lines 7029570377 +82 Branches 7715 7718 +3 + Hits 3162931730 +101 + Misses3573535705 -30 - Partials 2931 2942 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=) | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==) | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | | | [...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==) | `78.57% <78.57%>
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309523
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 10/Sep/19 04:31
Start Date: 10/Sep/19 04:31
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322548013
##
File path:
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
##
@@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
this.services.addAll(suite.getServices());
this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config,
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED,
false)) {
Review comment:
Sorry I should not mentioned `TaskRunnerSuiteBase`. What I meant to say is,
does it make more sense to add this service inside `getServices` method in
`GobblinTaskRunner` ?
The comment of `getServices` is :
```Creates and returns a {@link List} of additional {@link Service}s that
should be run in this {@link GobblinTaskRunner}. Sub-classes that need
additional {@link Service}s to run, should override this method```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309523)
Time Spent: 2h 10m (was: 2h)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309402
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 10/Sep/19 00:22
Start Date: 10/Sep/19 00:22
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322507069
##
File path:
gobblin-cluster/src/test/java/org/apache/gobblin/cluster/ContainerHealthMetricsServiceTest.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.cluster;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class ContainerHealthMetricsServiceTest {
+
+ @Test
+ public void testRunOneIteration() throws Exception {
+Config config = ConfigFactory.empty();
+ContainerHealthMetricsService service = new
ContainerHealthMetricsService(config);
+service.runOneIteration();
+long processCpuTime1 = service.processCpuTime.get();
+Thread.sleep(10);
+service.runOneIteration();
+long processCpuTime2 = service.processCpuTime.get();
+Assert.assertTrue(processCpuTime1 < processCpuTime2);
+ }
+}
Review comment:
Added newline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309402)
Time Spent: 2h (was: 1h 50m)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309399
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 10/Sep/19 00:21
Start Date: 10/Sep/19 00:21
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506877
##
File path:
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
##
@@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
this.services.addAll(suite.getServices());
this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config,
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED,
false)) {
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309399)
Time Spent: 1h 40m (was: 1.5h)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309398 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 10/Sep/19 00:20 Start Date: 10/Sep/19 00:20 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506849 ## File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java ## @@ -167,4 +167,5 @@ public static final String HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = GOBBLIN_CLUSTER_PREFIX + "job.stoppingStateTimeoutSeconds"; public static final long DEFAULT_HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 300; + public static final String CONTAINER_HEALTH_METRICS_SERVICE_ENABLED = GOBBLIN_CLUSTER_PREFIX + "container.health.metrics.service.enabled" ; } Review comment: Added a default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309398) Time Spent: 1.5h (was: 1h 20m) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309396
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 10/Sep/19 00:20
Start Date: 10/Sep/19 00:20
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506776
##
File path:
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
##
@@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
this.services.addAll(suite.getServices());
this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config,
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED,
false)) {
Review comment:
The TaskRunnerSuiteBase is an abstract class with two implementations,
process model and thread model. Wanted to leave getServices() in
TaskRunnerSuiteBase as an abstract method so as not to change the contract of
the class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309396)
Time Spent: 1h 10m (was: 1h)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309397 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 10/Sep/19 00:20 Start Date: 10/Sep/19 00:20 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322506814 ## File path: gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java ## @@ -954,4 +954,5 @@ */ public static final String AVRO_SCHEMA_CHECK_STRATEGY = "avro.schema.check.strategy"; public static final String AVRO_SCHEMA_CHECK_STRATEGY_DEFAULT = "org.apache.gobblin.util.schema_check.AvroSchemaCheckDefaultStrategy"; + Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309397) Time Spent: 1h 20m (was: 1h 10m) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309389
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 10/Sep/19 00:06
Start Date: 10/Sep/19 00:06
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322504103
##
File path:
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
##
@@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs,
Path destFilePath, Loca
return environmentVariableMap;
}
+
+ public static String getContainerNum(String containerId) {
+return "container-" + containerId.substring(containerId.lastIndexOf("_") +
1);
Review comment:
We want the substring starting from the char immediately following the last
"_". e.g. if containerId = "container_e94_1567552810874_2132400_01_01", we
want to return
"container-01". Added javadoc to make the behavior clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309389)
Time Spent: 1h (was: 50m)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309316 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 09/Sep/19 22:31 Start Date: 09/Sep/19 22:31 Worklog Time Spent: 10m Work Description: shirshanka commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481749 ## File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java ## @@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs, Path destFilePath, Loca return environmentVariableMap; } + Review comment: javadoc missing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309316) Time Spent: 40m (was: 0.5h) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 40m > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309317
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m
Work Description: shirshanka commented on pull request #2729:
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481788
##
File path:
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnHelixUtils.java
##
@@ -133,4 +131,8 @@ public static void addFileAsLocalResource(FileSystem fs,
Path destFilePath, Loca
return environmentVariableMap;
}
+
+ public static String getContainerNum(String containerId) {
+return "container-" + containerId.substring(containerId.lastIndexOf("_") +
1);
Review comment:
why are we doing + 1?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309317)
Time Spent: 40m (was: 0.5h)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309320
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m
Work Description: shirshanka commented on pull request #2729:
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322477750
##
File path:
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
##
@@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
this.services.addAll(suite.getServices());
this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config,
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED,
false)) {
Review comment:
Create a single static constant for the default value for this config, so
you don't have to say false in two places?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309320)
Time Spent: 50m (was: 40m)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309318 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 09/Sep/19 22:31 Start Date: 09/Sep/19 22:31 Worklog Time Spent: 10m Work Description: shirshanka commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322474783 ## File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java ## @@ -167,4 +167,5 @@ public static final String HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = GOBBLIN_CLUSTER_PREFIX + "job.stoppingStateTimeoutSeconds"; public static final long DEFAULT_HELIX_JOB_STOPPING_STATE_TIMEOUT_SECONDS = 300; + public static final String CONTAINER_HEALTH_METRICS_SERVICE_ENABLED = GOBBLIN_CLUSTER_PREFIX + "container.health.metrics.service.enabled" ; } Review comment: no default? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309318) Time Spent: 50m (was: 40m) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 50m > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309319
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 22:31
Start Date: 09/Sep/19 22:31
Worklog Time Spent: 10m
Work Description: shirshanka commented on pull request #2729:
GOBBLIN-875: Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322481849
##
File path:
gobblin-cluster/src/test/java/org/apache/gobblin/cluster/ContainerHealthMetricsServiceTest.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.cluster;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import lombok.extern.slf4j.Slf4j;
+
+
+@Slf4j
+public class ContainerHealthMetricsServiceTest {
+
+ @Test
+ public void testRunOneIteration() throws Exception {
+Config config = ConfigFactory.empty();
+ContainerHealthMetricsService service = new
ContainerHealthMetricsService(config);
+service.runOneIteration();
+long processCpuTime1 = service.processCpuTime.get();
+Thread.sleep(10);
+service.runOneIteration();
+long processCpuTime2 = service.processCpuTime.get();
+Assert.assertTrue(processCpuTime1 < processCpuTime2);
+ }
+}
Review comment:
missing newline at end of file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309319)
Time Spent: 50m (was: 40m)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309134 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 09/Sep/19 18:02 Start Date: 09/Sep/19 18:02 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1) Report > Merging [#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `71.64%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2729 +/- ## + Coverage 44.99% 45.02% +0.03% - Complexity 8742 8750 +8 Files 1884 1886 +2 Lines 7029570359 +64 Branches 7715 7717 +2 + Hits 3162931680 +51 - Misses3573535746 +11 - Partials 2931 2933 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.11% <0%> (-0.15%)` | `29 <0> (ø)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=) | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==) | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: | | [...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==) | `78.57%
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309132 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 09/Sep/19 18:01 Start Date: 09/Sep/19 18:01 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322371601 ## File path: gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java ## @@ -954,4 +954,5 @@ */ public static final String AVRO_SCHEMA_CHECK_STRATEGY = "avro.schema.check.strategy"; public static final String AVRO_SCHEMA_CHECK_STRATEGY_DEFAULT = "org.apache.gobblin.util.schema_check.AvroSchemaCheckDefaultStrategy"; + Review comment: Empty line, another one in`GobblinApplicationMaster.java`, line 116 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309132) Time Spent: 20m (was: 10m) > Emit container health metrics when running in cluster mode > -- > > Key: GOBBLIN-875 > URL: https://issues.apache.org/jira/browse/GOBBLIN-875 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This task implements a service that emits CPU/Memory health metrics from the > JVM when running in the cluster mode. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309133
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 18:01
Start Date: 09/Sep/19 18:01
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729#discussion_r322377318
##
File path:
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java
##
@@ -196,6 +196,11 @@ public GobblinTaskRunner(String applicationName,
this.services.addAll(suite.getServices());
this.services.addAll(getServices());
+
+if (ConfigUtils.getBoolean(this.config,
GobblinClusterConfigurationKeys.CONTAINER_HEALTH_METRICS_SERVICE_ENABLED,
false)) {
Review comment:
Should we package this service as part of `TaskRunnerSuiteBase` instead of
having a `this.services.addAll(getServices)` plus another service which is used
for metric-reporting outside?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309133)
Time Spent: 20m (was: 10m)
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=309091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309091
]
ASF GitHub Bot logged work on GOBBLIN-875:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 17:27
Start Date: 09/Sep/19 17:27
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2729: GOBBLIN-875:
Emit container health metrics when running in cluster mode
URL: https://github.com/apache/incubator-gobblin/pull/2729
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-875
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This task implements a service that emits CPU/Memory health metrics from the
JVM when running in the cluster mode.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Added unit test in ContainerHealthMetricsServiceTest.
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309091)
Remaining Estimate: 0h
Time Spent: 10m
> Emit container health metrics when running in cluster mode
> --
>
> Key: GOBBLIN-875
> URL: https://issues.apache.org/jira/browse/GOBBLIN-875
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This task implements a service that emits CPU/Memory health metrics from the
> JVM when running in the cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
