[GitHub] flink pull request #4772: [FLINK-7707] [flip6] Add TaskCheckpointStatisticDe...

2017-10-11 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4772


---

[GitHub] flink pull request #4772: [FLINK-7707] [flip6] Add TaskCheckpointStatisticDe...

2017-10-06 Thread tillrohrmann 
Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4772#discussion_r143157550
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/checkpoints/TaskCheckpointStatisticDetailsHandler.java
 ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.job.checkpoints;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.checkpoint.AbstractCheckpointStats;
+import org.apache.flink.runtime.checkpoint.MinMaxAvgStats;
+import org.apache.flink.runtime.checkpoint.SubtaskStateStats;
+import org.apache.flink.runtime.checkpoint.TaskStateStats;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.rest.handler.HandlerRequest;
+import org.apache.flink.runtime.rest.handler.RestHandlerException;
+import org.apache.flink.runtime.rest.handler.legacy.ExecutionGraphCache;
+import 
org.apache.flink.runtime.rest.handler.legacy.checkpoints.CheckpointStatsCache;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.runtime.rest.messages.JobVertexIdPathParameter;
+import org.apache.flink.runtime.rest.messages.MessageHeaders;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.MinMaxAvgStatistics;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.SubtaskCheckpointStatistics;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.TaskCheckpointMessageParameters;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.TaskCheckpointStatisticsWithSubtaskDetails;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+
+import 
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+
+/**
+ * REST handler which serves checkpoint statistics for subtasks.
+ */
+public class TaskCheckpointStatisticDetailsHandler extends 
AbstractCheckpointHandler {
+
+   public TaskCheckpointStatisticDetailsHandler(
+   CompletableFuture localRestAddress,
+   GatewayRetriever 
leaderRetriever,
+   Time timeout,
+   MessageHeaders 
messageHeaders,
+   ExecutionGraphCache executionGraphCache,
+   Executor executor,
+   CheckpointStatsCache checkpointStatsCache) {
+   super(localRestAddress, leaderRetriever, timeout, 
messageHeaders, executionGraphCache, executor, checkpointStatsCache);
+   }
+
+   @Override
+   protected TaskCheckpointStatisticsWithSubtaskDetails 
handleCheckpointRequest(
+   HandlerRequest request,
+   AbstractCheckpointStats checkpointStats) throws 
RestHandlerException {
+
+   final JobVertexID jobVertexId = 
request.getPathParameter(JobVertexIdPathParameter.class);
+
+   final TaskStateStats taskStatistics = 
checkpointStats.getTaskStateStats(jobVertexId);
+
+   if (taskStatistics != null) {
+
+   final 
TaskCheckpointStatisticsWithSubtaskDetails.Summary summary = createSummary(
+   taskStatistics.getSummaryStats(),
+   checkpointStats.getTriggerTimestamp());
+
+   final List 
subtaskCheckpointStatistics = createSubtaskCheckpointStatistics(
+   taskStatistics.getSubtaskStats(),
+   checkpointStats.getTriggerTimestamp());
+

[GitHub] flink pull request #4772: [FLINK-7707] [flip6] Add TaskCheckpointStatisticDe...

2017-10-05 Thread zentol 
Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4772#discussion_r142949904
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/checkpoints/TaskCheckpointStatisticDetailsHandler.java
 ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.handler.job.checkpoints;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.checkpoint.AbstractCheckpointStats;
+import org.apache.flink.runtime.checkpoint.MinMaxAvgStats;
+import org.apache.flink.runtime.checkpoint.SubtaskStateStats;
+import org.apache.flink.runtime.checkpoint.TaskStateStats;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.rest.handler.HandlerRequest;
+import org.apache.flink.runtime.rest.handler.RestHandlerException;
+import org.apache.flink.runtime.rest.handler.legacy.ExecutionGraphCache;
+import 
org.apache.flink.runtime.rest.handler.legacy.checkpoints.CheckpointStatsCache;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.runtime.rest.messages.JobVertexIdPathParameter;
+import org.apache.flink.runtime.rest.messages.MessageHeaders;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.MinMaxAvgStatistics;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.SubtaskCheckpointStatistics;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.TaskCheckpointMessageParameters;
+import 
org.apache.flink.runtime.rest.messages.checkpoints.TaskCheckpointStatisticsWithSubtaskDetails;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+
+import 
org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpResponseStatus;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+
+/**
+ * REST handler which serves checkpoint statistics for subtasks.
+ */
+public class TaskCheckpointStatisticDetailsHandler extends 
AbstractCheckpointHandler {
+
+   public TaskCheckpointStatisticDetailsHandler(
+   CompletableFuture localRestAddress,
+   GatewayRetriever 
leaderRetriever,
+   Time timeout,
+   MessageHeaders 
messageHeaders,
+   ExecutionGraphCache executionGraphCache,
+   Executor executor,
+   CheckpointStatsCache checkpointStatsCache) {
+   super(localRestAddress, leaderRetriever, timeout, 
messageHeaders, executionGraphCache, executor, checkpointStatsCache);
+   }
+
+   @Override
+   protected TaskCheckpointStatisticsWithSubtaskDetails 
handleCheckpointRequest(
+   HandlerRequest request,
+   AbstractCheckpointStats checkpointStats) throws 
RestHandlerException {
+
+   final JobVertexID jobVertexId = 
request.getPathParameter(JobVertexIdPathParameter.class);
+
+   final TaskStateStats taskStatistics = 
checkpointStats.getTaskStateStats(jobVertexId);
+
+   if (taskStatistics != null) {
+
+   final 
TaskCheckpointStatisticsWithSubtaskDetails.Summary summary = createSummary(
+   taskStatistics.getSummaryStats(),
+   checkpointStats.getTriggerTimestamp());
+
+   final List 
subtaskCheckpointStatistics = createSubtaskCheckpointStatistics(
+   taskStatistics.getSubtaskStats(),
+   checkpointStats.getTriggerTimestamp());
+
+

[GitHub] flink pull request #4772: [FLINK-7707] [flip6] Add TaskCheckpointStatisticDe...

2017-10-04 Thread tillrohrmann 
GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4772

[FLINK-7707] [flip6] Add TaskCheckpointStatisticDetailsHandler for new REST 
endpoint

## What is the purpose of the change

Adds the `TaskCheckpointStatisticDetailsHandler` which is the equivalent of 
the `CheckpointStatsDetailsSubtaskHandler` of the old `WebRuntimeMonitor`.

## Verifying this change

This change added tests and can be verified as follows:

- `TaskCheckpointStatisticsTest` and 
`TaskCheckpointStatisticsWithSubtaskDetailsTest` to test the (un)marshalling of 
these classes

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
portTaskCheckpointStatisticsDetailsHandler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4772.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4772


commit c09e579b26752f2ae477d60e8fbb6ccdce315df9
Author: Till Rohrmann 
Date:   2017-09-25T13:29:59Z

[FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers

The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the 
latter, the former
does not expect the AccessExecutionGraph to be the true ExecutionGraph. 
Instead it assumes
it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache 
entries after
a given time to live period. This will trigger requesting the 
AccessExecutionGraph again
and, thus, updating the ExecutionGraph information for the ExecutionGraph 
based REST
handlers.

In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic 
cleanup task
which triggers ExecutionGraphCache.cleanup. This methods releases all cache 
entries which
have exceeded their time to live. Currently it is set to 20 * 
refreshInterval of the
web gui.

This closes #4728.

commit a4f9ef81c02738b40cf0ba375650684b46f5417d
Author: Till Rohrmann 
Date:   2017-09-26T16:39:15Z

[FLINK-7695] [flip6] Add JobConfigHandler for new RestServerEndpoint

This closes #4737.

commit 4259fcc96c7c72644806941bd0df9f508a2f0bcd
Author: Till Rohrmann 
Date:   2017-09-28T16:35:50Z

[FLINK-7708] [flip6] Add CheckpointConfigHandler for new REST endpoint

This commit implements the CheckpointConfigHandler which now returns a
CheckpointConfigInfo object if checkpointing is enabled. In case that
checkpointing is disabled for a job, it will return a 404 response.

commit 6f1756b541ad76521c7b653f834c1032c623f1e6
Author: Till Rohrmann 
Date:   2017-09-29T13:09:06Z

[FLINK-7710] [flip6] Add CheckpointStatisticsHandler for the new REST 
endpoint

This commit also makes the CheckpointStatsHistory object serializable by 
removing the
CheckpointStatsHistoryIterable and replacing it with a static ArrayList.

commit bd8109b0af1a90fc32f76e57261a252762d678eb
Author: Till Rohrmann 
Date:   2017-10-02T17:39:38Z

[FLINK-7709] Add CheckpointStatisticDetailsHandler for new REST endpoint

commit 4d694411e9e4b6c508c258655c9c69cb26ddb6be
Author: Till Rohrmann 
Date:   2017-10-02T17:52:13Z

Disable failing when not all creator properties are known

commit 7c707920934476b5d33b37315a44e45a4e18e0d0
Author: Till Rohrmann 
Date:   2017-10-04T12:41:54Z

[FLINK-7707] [flip6] Add TaskCheckpointStatisticDetailsHandler for new REST 
endpoint




---