[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687897#comment-16687897
]
ASF GitHub Bot commented on FLINK-10634:
zentol closed pull request #7056: [FLINK-10634][metrics][rest] Add metrics
availability e2e test
URL: https://github.com/apache/flink/pull/7056
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-end-to-end-tests/flink-metrics-availability-test/pom.xml
b/flink-end-to-end-tests/flink-metrics-availability-test/pom.xml
new file mode 100644
index 000..eea4ef8458d
--- /dev/null
+++ b/flink-end-to-end-tests/flink-metrics-availability-test/pom.xml
@@ -0,0 +1,93 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.8-SNAPSHOT
+
+
+ 4.0.0
+
+ flink-metrics-availability-test
+ 1.8-SNAPSHOT
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+ test
+
+
+ org.apache.flink
+
flink-runtime_${scala.binary.version}
+ ${project.version}
+ test
+
+
+ org.apache.flink
+ flink-end-to-end-tests-common
+ ${project.version}
+ test
+
+
+ org.apache.flink
+ flink-test-utils-junit
+ ${project.version}
+ test
+
+
+
+
+
+ e2e-metric-availability
+
+
+ e2e-metrics
+
+
+
+
+
+
org.apache.maven.plugins
+
maven-surefire-plugin
+
+
+
e2e-metric-availability
+
integration-test
+
+
test
+
+
+
+
**/*ITCase.*
+
+
+
+
+
+
+
+
+
+
+
diff --git
a/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
b/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
new file mode 100644
index 000..28a21354e5d
--- /dev/null
+++
b/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686342#comment-16686342 ] ASF GitHub Bot commented on FLINK-10634: zentol commented on issue #7056: [FLINK-10634][metrics][rest] Add metrics availability e2e test URL: https://github.com/apache/flink/pull/7056#issuecomment-438618845 > Also it is worth adding a line in the README for the Prometheus and this test that states that you can run them locally with mvn clean verify -De2e-metrics -DdistDir=YOUR_DIR I don't really want to add these instructions right now since they are quite disconnected from the actual test. How the profile is called is determined in the pom, what the distDir property is called in the `FlinkDistribution`. I'd like to avoid them becoming out-dated, i.e. simply wrong. I have some ideas on how to better document how to run them, so that if the test is not executed you get a message saying what was missing. E.g. "Test was skipped since property 'e2e-metrics' was not specified." This way you would simply _try_ running them and get the infos on how to do it right there. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684968#comment-16684968 ] ASF GitHub Bot commented on FLINK-10634: zentol commented on issue #7056: [FLINK-10634][metrics][rest] Add metrics availability e2e test URL: https://github.com/apache/flink/pull/7056#issuecomment-438207344 The test passed in the linked build. The second build encountered a timeout since the cache hat do be rebuilt, the first failed in a subsequent bash e2e test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683483#comment-16683483
]
ASF GitHub Bot commented on FLINK-10634:
kl0u commented on a change in pull request #7056: [FLINK-10634][metrics][rest]
Add metrics availability e2e test
URL: https://github.com/apache/flink/pull/7056#discussion_r232591919
##
File path:
flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.pache.flink.metrics.tests;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter;
+import org.apache.flink.runtime.rest.RestClient;
+import org.apache.flink.runtime.rest.RestClientConfiguration;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsMessageParameters;
+import
org.apache.flink.runtime.rest.messages.job.metrics.MetricCollectionResponseBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsMessageParameters;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerInfo;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersHeaders;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersInfo;
+import org.apache.flink.tests.util.FlinkDistribution;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.SupplierWithException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+/**
+ * End-to-end test for the availability of metrics.
+ */
+public class MetricsAvailabilityITCase extends TestLogger {
+
+ private static final String HOST = "localhost";
+ private static final int PORT = 8081;
+
+ @Rule
+ public final FlinkDistribution dist = new FlinkDistribution();
+
+ @Nullable
+ private static ScheduledExecutorService scheduledExecutorService = null;
+
+ @BeforeClass
+ public static void startExecutor() {
+ scheduledExecutorService = Executors.newScheduledThreadPool(4);
+ }
+
+ @AfterClass
+ public static void shutdownExecutor() {
+ if (scheduledExecutorService != null) {
+ scheduledExecutorService.shutdown();
+ }
+ }
+
+ @Test
+ public void testReporter() throws Exception {
+ dist.startFlinkCluster();
+
+ final RestClient restClient = new
RestClient(RestClientConfiguration.fromConfiguration(new Configuration()),
scheduledExecutorService);
+
+ checkJobManagerMetricAvailability(restClient);
+
+ final Collection taskManagerIds =
getTaskManagerIds(restClient);
+
+ for (final ResourceID taskManagerId : taskManagerIds) {
+ checkTaskManagerMetricAvailability(restClient,
taskManagerId);
+ }
+ }
+
+ private static void
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683491#comment-16683491 ] ASF GitHub Bot commented on FLINK-10634: kl0u commented on issue #7056: [FLINK-10634][metrics][rest] Add metrics availability e2e test URL: https://github.com/apache/flink/pull/7056#issuecomment-437821227 BTW the Travis build you link to has failed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683488#comment-16683488
]
ASF GitHub Bot commented on FLINK-10634:
kl0u commented on a change in pull request #7056: [FLINK-10634][metrics][rest]
Add metrics availability e2e test
URL: https://github.com/apache/flink/pull/7056#discussion_r232591615
##
File path:
flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.pache.flink.metrics.tests;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter;
+import org.apache.flink.runtime.rest.RestClient;
+import org.apache.flink.runtime.rest.RestClientConfiguration;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsMessageParameters;
+import
org.apache.flink.runtime.rest.messages.job.metrics.MetricCollectionResponseBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsMessageParameters;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerInfo;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersHeaders;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersInfo;
+import org.apache.flink.tests.util.FlinkDistribution;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.SupplierWithException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+/**
+ * End-to-end test for the availability of metrics.
+ */
+public class MetricsAvailabilityITCase extends TestLogger {
+
+ private static final String HOST = "localhost";
+ private static final int PORT = 8081;
+
+ @Rule
+ public final FlinkDistribution dist = new FlinkDistribution();
+
+ @Nullable
+ private static ScheduledExecutorService scheduledExecutorService = null;
+
+ @BeforeClass
+ public static void startExecutor() {
+ scheduledExecutorService = Executors.newScheduledThreadPool(4);
+ }
+
+ @AfterClass
+ public static void shutdownExecutor() {
+ if (scheduledExecutorService != null) {
+ scheduledExecutorService.shutdown();
+ }
+ }
+
+ @Test
+ public void testReporter() throws Exception {
+ dist.startFlinkCluster();
+
+ final RestClient restClient = new
RestClient(RestClientConfiguration.fromConfiguration(new Configuration()),
scheduledExecutorService);
+
Review comment:
Do we need the `checkJobManagerMetricAvailability ` given that later we also
ask the JM for the `TaskManagerIds`? If not, the we can remove that method and
rename the `getTaskManagerIds ` to sth like `verifyJMMetricsAndFetch...`.
This is an automated message from the Apache Git Service.
To
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683484#comment-16683484
]
ASF GitHub Bot commented on FLINK-10634:
kl0u commented on a change in pull request #7056: [FLINK-10634][metrics][rest]
Add metrics availability e2e test
URL: https://github.com/apache/flink/pull/7056#discussion_r232591854
##
File path:
flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.pache.flink.metrics.tests;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter;
+import org.apache.flink.runtime.rest.RestClient;
+import org.apache.flink.runtime.rest.RestClientConfiguration;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsMessageParameters;
+import
org.apache.flink.runtime.rest.messages.job.metrics.MetricCollectionResponseBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsMessageParameters;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerInfo;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersHeaders;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersInfo;
+import org.apache.flink.tests.util.FlinkDistribution;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.SupplierWithException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+/**
+ * End-to-end test for the availability of metrics.
+ */
+public class MetricsAvailabilityITCase extends TestLogger {
+
+ private static final String HOST = "localhost";
+ private static final int PORT = 8081;
+
+ @Rule
+ public final FlinkDistribution dist = new FlinkDistribution();
+
+ @Nullable
+ private static ScheduledExecutorService scheduledExecutorService = null;
+
+ @BeforeClass
+ public static void startExecutor() {
+ scheduledExecutorService = Executors.newScheduledThreadPool(4);
+ }
+
+ @AfterClass
+ public static void shutdownExecutor() {
+ if (scheduledExecutorService != null) {
+ scheduledExecutorService.shutdown();
+ }
+ }
+
+ @Test
+ public void testReporter() throws Exception {
+ dist.startFlinkCluster();
+
+ final RestClient restClient = new
RestClient(RestClientConfiguration.fromConfiguration(new Configuration()),
scheduledExecutorService);
+
+ checkJobManagerMetricAvailability(restClient);
+
+ final Collection taskManagerIds =
getTaskManagerIds(restClient);
+
+ for (final ResourceID taskManagerId : taskManagerIds) {
+ checkTaskManagerMetricAvailability(restClient,
taskManagerId);
+ }
+ }
+
+ private static void
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683485#comment-16683485
]
ASF GitHub Bot commented on FLINK-10634:
kl0u commented on a change in pull request #7056: [FLINK-10634][metrics][rest]
Add metrics availability e2e test
URL: https://github.com/apache/flink/pull/7056#discussion_r232591615
##
File path:
flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.pache.flink.metrics.tests;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter;
+import org.apache.flink.runtime.rest.RestClient;
+import org.apache.flink.runtime.rest.RestClientConfiguration;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsMessageParameters;
+import
org.apache.flink.runtime.rest.messages.job.metrics.MetricCollectionResponseBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsMessageParameters;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerInfo;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersHeaders;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersInfo;
+import org.apache.flink.tests.util.FlinkDistribution;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.SupplierWithException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+/**
+ * End-to-end test for the availability of metrics.
+ */
+public class MetricsAvailabilityITCase extends TestLogger {
+
+ private static final String HOST = "localhost";
+ private static final int PORT = 8081;
+
+ @Rule
+ public final FlinkDistribution dist = new FlinkDistribution();
+
+ @Nullable
+ private static ScheduledExecutorService scheduledExecutorService = null;
+
+ @BeforeClass
+ public static void startExecutor() {
+ scheduledExecutorService = Executors.newScheduledThreadPool(4);
+ }
+
+ @AfterClass
+ public static void shutdownExecutor() {
+ if (scheduledExecutorService != null) {
+ scheduledExecutorService.shutdown();
+ }
+ }
+
+ @Test
+ public void testReporter() throws Exception {
+ dist.startFlinkCluster();
+
+ final RestClient restClient = new
RestClient(RestClientConfiguration.fromConfiguration(new Configuration()),
scheduledExecutorService);
+
Review comment:
Do we need the `checkJobManagerMetricAvailability ` given that later we also
ask the JM for the `TaskManagerIds`? If not, the we can remove that method and
rename the `getTaskManagerIds ` to sth like `verifyJMMetricsAndFetch...`.
This is an automated message from the Apache Git Service.
To
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[
https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683486#comment-16683486
]
ASF GitHub Bot commented on FLINK-10634:
kl0u commented on a change in pull request #7056: [FLINK-10634][metrics][rest]
Add metrics availability e2e test
URL: https://github.com/apache/flink/pull/7056#discussion_r232592152
##
File path:
flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java
##
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.pache.flink.metrics.tests;
+
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter;
+import org.apache.flink.runtime.rest.RestClient;
+import org.apache.flink.runtime.rest.RestClientConfiguration;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.JobManagerMetricsMessageParameters;
+import
org.apache.flink.runtime.rest.messages.job.metrics.MetricCollectionResponseBody;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsHeaders;
+import
org.apache.flink.runtime.rest.messages.job.metrics.TaskManagerMetricsMessageParameters;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagerInfo;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersHeaders;
+import org.apache.flink.runtime.rest.messages.taskmanager.TaskManagersInfo;
+import org.apache.flink.tests.util.FlinkDistribution;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.function.SupplierWithException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+/**
+ * End-to-end test for the availability of metrics.
+ */
+public class MetricsAvailabilityITCase extends TestLogger {
+
+ private static final String HOST = "localhost";
+ private static final int PORT = 8081;
+
+ @Rule
+ public final FlinkDistribution dist = new FlinkDistribution();
+
+ @Nullable
+ private static ScheduledExecutorService scheduledExecutorService = null;
+
+ @BeforeClass
+ public static void startExecutor() {
+ scheduledExecutorService = Executors.newScheduledThreadPool(4);
+ }
+
+ @AfterClass
+ public static void shutdownExecutor() {
+ if (scheduledExecutorService != null) {
+ scheduledExecutorService.shutdown();
+ }
+ }
+
+ @Test
+ public void testReporter() throws Exception {
+ dist.startFlinkCluster();
+
+ final RestClient restClient = new
RestClient(RestClientConfiguration.fromConfiguration(new Configuration()),
scheduledExecutorService);
+
+ checkJobManagerMetricAvailability(restClient);
+
+ final Collection taskManagerIds =
getTaskManagerIds(restClient);
+
+ for (final ResourceID taskManagerId : taskManagerIds) {
+ checkTaskManagerMetricAvailability(restClient,
taskManagerId);
+ }
+ }
+
+ private static void
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679576#comment-16679576 ] ASF GitHub Bot commented on FLINK-10634: zentol opened a new pull request #7056: [FLINK-10634][metrics][rest] Add metrics availability e2e test URL: https://github.com/apache/flink/pull/7056 ## What is the purpose of the change This PR adds an end-to-end test to ensure that JM/TM metrics are available via the REST API. This test primarily covers the `MetricQueryService`, but could effectively be used in the future as a general metrics-availability test. ## Brief change log * make several MessageParameter fields public as intended * added new module `flink-metrics-availability-test` containing test ## Verifying this change Ran the test on travis: https://travis-ci.org/zentol/flink/builds/452275270 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678125#comment-16678125 ] Biao Liu commented on FLINK-10634: -- [~Zentol] Not yet. Please feel free to take it. > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Biao Liu >Priority: Major > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677828#comment-16677828 ] Chesnay Schepler commented on FLINK-10634: -- [~SleePy] Did you already start on this? If not I'd like to do it. > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Biao Liu >Priority: Major > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658681#comment-16658681 ] Till Rohrmann commented on FLINK-10634: --- If we access JM and TM metrics via the REST API then I'm fine to close this issue [~Zentol]. Otherwise I would like to make sure that also TM metrics are properly reported. Please name the e2e tests which do this when closing this issue. > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.7.0 > > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10634) End-to-end test: Metrics accessible via REST API
[ https://issues.apache.org/jira/browse/FLINK-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658666#comment-16658666 ] Chesnay Schepler commented on FLINK-10634: -- I don't believe this test to be necessary as several E2E tests already use the metric system (both via reporters and the REST API). > End-to-end test: Metrics accessible via REST API > > > Key: FLINK-10634 > URL: https://issues.apache.org/jira/browse/FLINK-10634 > Project: Flink > Issue Type: Sub-task > Components: E2E Tests, Metrics, REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.7.0 > > > Verify that Flink's metrics can be accessed via the REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
