[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271116#comment-16271116
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4734


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268760#comment-16268760
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
Hi @tillrohrmann, this PR is now rebased to the latest master, and reworked 
to incorporate your last comments.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241669#comment-16241669
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
@tillrohrmann yes, reusing `DispatcherGateway#requestJobDetails` was an 
option that did occur to me but was also not sure of the redundant cost. But 
sure, we can avoid adding yet another RPC for now.

Thanks for the branch link. I'll rebase onto that.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240176#comment-16240176
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
Maybe you could rebase your work on top of 
https://github.com/tillrohrmann/flink/tree/FLINK-7870. That's the branch where 
I stacked multiple of the pending handler PRs and which I will merge shortly 
after the 1.4 release branch has been cut.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240172#comment-16240172
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r149057297
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import org.apache.flink.runtime.rest.messages.json.JobIDDeserializer;
+import org.apache.flink.runtime.rest.messages.json.JobIDSerializer;
+import 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   @JsonProperty(FIELD_NAME_JOBS_CREATED_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsCreated;
+
+   @JsonProperty(FIELD_NAME_JOBS_RUNNING_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsRunningOrPending;
+
+   @JsonProperty(FIELD_NAME_JOBS_FINISHED_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsFinished;
+
+   @JsonProperty(FIELD_NAME_JOBS_CANCELLING_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsCancelling;
+
+   @JsonProperty(FIELD_NAME_JOBS_CANCELLED_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsCancelled;
+
+   @JsonProperty(FIELD_NAME_JOBS_FAILING_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsFailing;
+
+   @JsonProperty(FIELD_NAME_JOBS_FAILED_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsFailed;
+
+   @JsonProperty(FIELD_NAME_JOBS_RESTARTING_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsRestarting;
+
+   @JsonProperty(FIELD_NAME_JOBS_SUSPENDED_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsSuspended;
+
+   @JsonProperty(FIELD_NAME_JOBS_RECONCILING_IDS)
+   @JsonSerialize(contentUsing = JobIDSerializer.class)
+   private final List jobsReconciling;
--- End diff --

I would actually not group the jobs here. Rather we could 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240154#comment-16240154
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
Thanks a lot @tzulitai. Will take another look at it and merge it if I have 
no other comments.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240110#comment-16240110
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
@tillrohrmann the PR is now rebased. Previous comments have also been 
addressed.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239987#comment-16239987
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
Yes, I'll rebase this now.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238027#comment-16238027
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
I think @zentol is right and we can register the 
`CurrentJobsOverviewHandler` under `jobs/overview`. Thus, we should add this 
handler as well. Could you rebase this handler onto the latest master such that 
we can merge it?


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233794#comment-16233794
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
@tzulitai, I actually want to re-check whether it's indeed not possible to 
register the `CurrentJobsOverviewHandler` under `jobs/overview`. If this should 
indeed be the case, then it will most likely subsume this handler. If not, then 
we'll register this handler under `/jobs`


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233767#comment-16233767
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
Hi @tillrohrmann, just to double check, so does the conclusion mean that 
#4805 subsumes this PR, and this one can be closed? 


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210986#comment-16210986
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
@tzulitai, we actually figured out that we cannot register the 
`CurrentJobsOverviewHandler` under `/jobs/overview` because it conflicts with 
`/jobs/:jobid`. Therefore, we thought that we could register the 
`CurrentJobsOverviewHandler` under `/jobs`. This would then not only return the 
`JobIDs` but also more information about the job. In the future we might add an 
include query parameter with which one can select which information to return.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200216#comment-16200216
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143998745
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+@JsonSerialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewSerializer.class)
+@JsonDeserialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewDeserializer.class)
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   private final List jobsCreated;
+   private final List jobsRunningOrPending;
+   private final List jobsFinished;
+   private final List jobsCancelling;
+   private final List jobsCancelled;
+   private final List jobsFailing;
+   private final List jobsFailed;
+   private final List jobsRestarting;
+   private final List jobsSuspended;
+   private final List jobsReconciling;
+
+   public JobStatusesWithIdsOverview(
+   List jobsCreated,
+   List jobsRunningOrPending,
+   List jobsFinished,
+   List jobsCancelling,
+   List jobsCancelled,
+   List jobsFailing,
+   List jobsFailed,
+   List jobsRestarting,
+   List jobsSuspended,
+   List jobsReconciling) {
+
+   this.jobsCreated = checkNotNull(jobsCreated);
+   this.jobsRunningOrPending = checkNotNull(jobsRunningOrPending);
+   this.jobsFinished = checkNotNull(jobsFinished);
+   this.jobsCancelling = checkNotNull(jobsCancelling);
+   this.jobsCancelled = checkNotNull(jobsCancelled);
+   this.jobsFailing = 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200204#comment-16200204
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143996677
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+@JsonSerialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewSerializer.class)
+@JsonDeserialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewDeserializer.class)
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   private final List jobsCreated;
+   private final List jobsRunningOrPending;
+   private final List jobsFinished;
+   private final List jobsCancelling;
+   private final List jobsCancelled;
+   private final List jobsFailing;
+   private final List jobsFailed;
+   private final List jobsRestarting;
+   private final List jobsSuspended;
+   private final List jobsReconciling;
+
+   public JobStatusesWithIdsOverview(
+   List jobsCreated,
+   List jobsRunningOrPending,
+   List jobsFinished,
+   List jobsCancelling,
+   List jobsCancelled,
+   List jobsFailing,
+   List jobsFailed,
+   List jobsRestarting,
+   List jobsSuspended,
+   List jobsReconciling) {
+
+   this.jobsCreated = checkNotNull(jobsCreated);
+   this.jobsRunningOrPending = checkNotNull(jobsRunningOrPending);
+   this.jobsFinished = checkNotNull(jobsFinished);
+   this.jobsCancelling = checkNotNull(jobsCancelling);
+   this.jobsCancelled = checkNotNull(jobsCancelled);
+   this.jobsFailing = 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200120#comment-16200120
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143983019
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+@JsonSerialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewSerializer.class)
+@JsonDeserialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewDeserializer.class)
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   private final List jobsCreated;
+   private final List jobsRunningOrPending;
+   private final List jobsFinished;
+   private final List jobsCancelling;
+   private final List jobsCancelled;
+   private final List jobsFailing;
+   private final List jobsFailed;
+   private final List jobsRestarting;
+   private final List jobsSuspended;
+   private final List jobsReconciling;
+
+   public JobStatusesWithIdsOverview(
+   List jobsCreated,
+   List jobsRunningOrPending,
+   List jobsFinished,
+   List jobsCancelling,
+   List jobsCancelled,
+   List jobsFailing,
+   List jobsFailed,
+   List jobsRestarting,
+   List jobsSuspended,
+   List jobsReconciling) {
+
+   this.jobsCreated = checkNotNull(jobsCreated);
+   this.jobsRunningOrPending = checkNotNull(jobsRunningOrPending);
+   this.jobsFinished = checkNotNull(jobsFinished);
+   this.jobsCancelling = checkNotNull(jobsCancelling);
+   this.jobsCancelled = checkNotNull(jobsCancelled);
+   this.jobsFailing = 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200085#comment-16200085
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143973267
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java 
---
@@ -242,6 +248,86 @@ public void start() throws Exception {
}
 
@Override
+   public CompletableFuture 
requestJobIdsOverview(@RpcTimeout Time timeout) {
--- End diff --

I think that makes a lot of sense. Will address this.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200087#comment-16200087
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143973446
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java 
---
@@ -242,6 +248,86 @@ public void start() throws Exception {
}
 
@Override
+   public CompletableFuture 
requestJobIdsOverview(@RpcTimeout Time timeout) {
+   final int numJobs = jobManagerRunners.size();
+
+   ArrayList>> 
jobStatuses = new ArrayList<>(numJobs);
+   for (Map.Entry jobManagerRunnerEntry : 
jobManagerRunners.entrySet()) {
+   CompletableFuture jobStatusFuture =
+   
jobManagerRunnerEntry.getValue().getJobManagerGateway().requestJobStatus(timeout);
+
+   jobStatuses.add(jobStatusFuture.thenApply(jobStatus -> 
Tuple2.of(jobManagerRunnerEntry.getKey(), jobStatus)));
+   }
+
+   CompletableFuture>> 
combinedJobStatusesFuture = FutureUtils.combineAll(jobStatuses);
+
+   return combinedJobStatusesFuture.thenApply(
+   jobStatusesWithIds -> {
+   List jobsCreated = new LinkedList<>();
+   List jobsRunning = new LinkedList<>();
+   List jobsFinished = new LinkedList<>();
+   List jobsCancelling = new LinkedList<>();
+   List jobsCancelled = new LinkedList<>();
+   List jobsFailing = new LinkedList<>();
+   List jobsFailed = new LinkedList<>();
+   List jobsRestarting = new LinkedList<>();
+   List jobsSuspended = new LinkedList<>();
+   List jobsReconciling = new 
LinkedList<>();
--- End diff --

I think we might need some micro benchmarking here to be sure, as it may 
depend on the actual list to return. But I could stay with `ArrayList` (I think 
that was what we were using anyway.)


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200083#comment-16200083
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
@tillrohrmann @zentol alright, thanks for the review and inputs. I'll 
address comments and change the `CurrentJobIdsHandler` to return all available 
Job IDs as a simple list as Till suggested.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200077#comment-16200077
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
I just talked to @zentol and we both agreed that we should register the 
`CurrentJobsOverviewHandler` under `jobs/overview`. Moreover, we should change 
the `CurrentJobIdsHandler` such that it simply returns all available job ids as 
a simple list. Maybe we could rename it into `JobIdsHandler` or `JobsHandler`.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198803#comment-16198803
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4734
  
I agree with @zentol. Moreover, I would like to change 
`MultipleJobsDetails` to not split the job details into running and finished. 
Just a collection of `JobDetails`.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198656#comment-16198656
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4734
  
Actually, I'd like to throw out the entire handler and replace it with the 
`CurrentJobsOverviewHandler`. The CurrentJobIdshandler handler is _not_ used by 
the web UI as the listings of jobs in the UI go through `/joboverview` (that i 
would prefer to go through `/jobs`).


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198633#comment-16198633
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143716493
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java 
---
@@ -242,6 +248,86 @@ public void start() throws Exception {
}
 
@Override
+   public CompletableFuture 
requestJobIdsOverview(@RpcTimeout Time timeout) {
+   final int numJobs = jobManagerRunners.size();
+
+   ArrayList>> 
jobStatuses = new ArrayList<>(numJobs);
+   for (Map.Entry jobManagerRunnerEntry : 
jobManagerRunners.entrySet()) {
+   CompletableFuture jobStatusFuture =
+   
jobManagerRunnerEntry.getValue().getJobManagerGateway().requestJobStatus(timeout);
+
+   jobStatuses.add(jobStatusFuture.thenApply(jobStatus -> 
Tuple2.of(jobManagerRunnerEntry.getKey(), jobStatus)));
+   }
+
+   CompletableFuture>> 
combinedJobStatusesFuture = FutureUtils.combineAll(jobStatuses);
+
+   return combinedJobStatusesFuture.thenApply(
+   jobStatusesWithIds -> {
+   List jobsCreated = new LinkedList<>();
+   List jobsRunning = new LinkedList<>();
+   List jobsFinished = new LinkedList<>();
+   List jobsCancelling = new LinkedList<>();
+   List jobsCancelled = new LinkedList<>();
+   List jobsFailing = new LinkedList<>();
+   List jobsFailed = new LinkedList<>();
+   List jobsRestarting = new LinkedList<>();
+   List jobsSuspended = new LinkedList<>();
+   List jobsReconciling = new 
LinkedList<>();
--- End diff --

Not entirely sure, but I think that `ArrayList` would be better here. Even 
though we will most likely only add few elements per list, a linked list is 
probably a bit slower since `LinkedList` uses some more complex internal data 
structures.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198634#comment-16198634
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143718709
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+@JsonSerialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewSerializer.class)
+@JsonDeserialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewDeserializer.class)
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   private final List jobsCreated;
+   private final List jobsRunningOrPending;
+   private final List jobsFinished;
+   private final List jobsCancelling;
+   private final List jobsCancelled;
+   private final List jobsFailing;
+   private final List jobsFailed;
+   private final List jobsRestarting;
+   private final List jobsSuspended;
+   private final List jobsReconciling;
+
+   public JobStatusesWithIdsOverview(
+   List jobsCreated,
+   List jobsRunningOrPending,
+   List jobsFinished,
+   List jobsCancelling,
+   List jobsCancelled,
+   List jobsFailing,
+   List jobsFailed,
+   List jobsRestarting,
+   List jobsSuspended,
+   List jobsReconciling) {
+
+   this.jobsCreated = checkNotNull(jobsCreated);
+   this.jobsRunningOrPending = checkNotNull(jobsRunningOrPending);
+   this.jobsFinished = checkNotNull(jobsFinished);
+   this.jobsCancelling = checkNotNull(jobsCancelling);
+   this.jobsCancelled = checkNotNull(jobsCancelled);
+   this.jobsFailing = 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198635#comment-16198635
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143718904
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobStatusesWithIdsOverview.java
 ---
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.messages.webmonitor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.rest.messages.ResponseBody;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An overview of how many jobs are in which status.
+ */
+@JsonSerialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewSerializer.class)
+@JsonDeserialize(using = 
JobStatusesWithIdsOverview.JobStatusesWithIdsOverviewDeserializer.class)
+public class JobStatusesWithIdsOverview implements ResponseBody, 
InfoMessage {
+
+   private static final long serialVersionUID = -3699051943490133183L;
+
+   public static final String FIELD_NAME_JOBS_CREATED_IDS = "jobs-created";
+   public static final String FIELD_NAME_JOBS_RUNNING_IDS = "jobs-running";
+   public static final String FIELD_NAME_JOBS_FINISHED_IDS = 
"jobs-finished";
+   public static final String FIELD_NAME_JOBS_CANCELLING_IDS = 
"jobs-cancelling";
+   public static final String FIELD_NAME_JOBS_CANCELLED_IDS = 
"jobs-cancelled";
+   public static final String FIELD_NAME_JOBS_FAILING_IDS = "jobs-failing";
+   public static final String FIELD_NAME_JOBS_FAILED_IDS = "jobs-failed";
+   public static final String FIELD_NAME_JOBS_RESTARTING_IDS = 
"jobs-restarting";
+   public static final String FIELD_NAME_JOBS_SUSPENDED_IDS = 
"jobs-suspended";
+   public static final String FIELD_NAME_JOBS_RECONCILING_IDS = 
"jobs-reconciling";
+
+   private final List jobsCreated;
+   private final List jobsRunningOrPending;
+   private final List jobsFinished;
+   private final List jobsCancelling;
+   private final List jobsCancelled;
+   private final List jobsFailing;
+   private final List jobsFailed;
+   private final List jobsRestarting;
+   private final List jobsSuspended;
+   private final List jobsReconciling;
+
+   public JobStatusesWithIdsOverview(
+   List jobsCreated,
+   List jobsRunningOrPending,
+   List jobsFinished,
+   List jobsCancelling,
+   List jobsCancelled,
+   List jobsFailing,
+   List jobsFailed,
+   List jobsRestarting,
+   List jobsSuspended,
+   List jobsReconciling) {
+
+   this.jobsCreated = checkNotNull(jobsCreated);
+   this.jobsRunningOrPending = checkNotNull(jobsRunningOrPending);
+   this.jobsFinished = checkNotNull(jobsFinished);
+   this.jobsCancelling = checkNotNull(jobsCancelling);
+   this.jobsCancelled = checkNotNull(jobsCancelled);
+   this.jobsFailing = 

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198632#comment-16198632
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r143718052
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java 
---
@@ -242,6 +248,86 @@ public void start() throws Exception {
}
 
@Override
+   public CompletableFuture 
requestJobIdsOverview(@RpcTimeout Time timeout) {
--- End diff --

I think it would be better to define `requestJobsOverview` which simply 
returns a `Collection` with `JobOverview` containing the `JobID` 
and the `JobStatus`, for example. All other data processing and data 
restructuring like grouping the individual jobs to make it easier for the Web 
UI to display the content, should then happen on the handler and not in the 
`Dispatcher`. My gut feeling is that this is unnecessary coupling of unrelated 
components.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183982#comment-16183982
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4734
  
nvm, this isn't used by the web UI:


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183968#comment-16183968
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4734
  
The additional job statuses are only new JSON fields. Not sure how the web 
UI handles them, but shouldn't the change be non-breaking?

We could also revert b60466c for now if that will make things easier.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183965#comment-16183965
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r141580505
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/CurrentJobIdsHandler.java
 ---
@@ -68,36 +84,9 @@ public CurrentJobIdsHandler(Executor executor, Time 
timeout) {
 
StringWriter writer = new 
StringWriter();
JsonGenerator gen = 
JsonFactory.JACKSON_FACTORY.createGenerator(writer);
-
-   gen.writeStartObject();
-
-   
gen.writeArrayFieldStart("jobs-running");
-   for (JobID jid : 
overview.getJobsRunningOrPending()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-finished");
-   for (JobID jid : 
overview.getJobsFinished()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-cancelled");
-   for (JobID jid : 
overview.getJobsCancelled()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-failed");
-   for (JobID jid : 
overview.getJobsFailed()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   gen.writeEndObject();
-
+   new 
JobsWithIDsOverview.JobsWithIDsOverviewSerializer().serialize(overview, gen, 
null);
--- End diff --

Will change.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183940#comment-16183940
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4734#discussion_r141575013
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/CurrentJobIdsHandler.java
 ---
@@ -68,36 +84,9 @@ public CurrentJobIdsHandler(Executor executor, Time 
timeout) {
 
StringWriter writer = new 
StringWriter();
JsonGenerator gen = 
JsonFactory.JACKSON_FACTORY.createGenerator(writer);
-
-   gen.writeStartObject();
-
-   
gen.writeArrayFieldStart("jobs-running");
-   for (JobID jid : 
overview.getJobsRunningOrPending()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-finished");
-   for (JobID jid : 
overview.getJobsFinished()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-cancelled");
-   for (JobID jid : 
overview.getJobsCancelled()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   
gen.writeArrayFieldStart("jobs-failed");
-   for (JobID jid : 
overview.getJobsFailed()) {
-   
gen.writeString(jid.toString());
-   }
-   gen.writeEndArray();
-
-   gen.writeEndObject();
-
+   new 
JobsWithIDsOverview.JobsWithIDsOverviewSerializer().serialize(overview, gen, 
null);
--- End diff --

we could store the serializer in a field.


> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182466#comment-16182466
 ] 

ASF GitHub Bot commented on FLINK-7652:
---

GitHub user tzulitai opened a pull request:

https://github.com/apache/flink/pull/4734

[FLINK-7652] [flip6] Port CurrentJobIdsHandler to new REST endpoint

## What is the purpose of the change

This PR ports the existing `CurrentJobIdsHandler` to the new REST endpoint. 
The existing class `JobsWithIdsOverview` is reused as the response message 
(second commit renames the class to `JobStatusesWithIdsOverview`). Introduced 
`CurrentJobIdsHeaders` as part of the porting.

This PR also includes a fix that the previous response message only 
contained job ids for statuses `RUNNING`, `FAILED`, `FINISHED`, and 
`CANCELLED`, whereas there are actually more statuses to differentiate, such as 
`CREATED`, `RESTARTING`, `SUSPENDED`, etc.

## Brief change log

- Let `CurrentJobIdsHandler` implement the `LegacyRestHandler` interface
- Introduced `CurrentJobIdsHeaders`
- Added new methods to `DispatcherGateway` and `JobMasterGateway` to 
facilitate the porting
- Register handler at `DispatcherRestEndpoint`
- Reuse `JobsWithIdsOverview` class as response message (renamed to 
`JobStatusesWithIdsOverview`)
- Let response message differentiate between all possible `JobStatus`.

## Verifying this change

This change is already covered by existing tests.

A new `JobStatusesWithIdsOverviewTest` is added to test the message 
marshalling.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): no
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): no
  - Anything that affects deployment or recovery: no

## Documentation

  - Does this pull request introduce a new feature? no
  - If yes, how is the feature documented? n/a



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tzulitai/flink portCurrentJobIdsHandler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4734.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4734


commit 4f3ad7a5d6566e23bb5138b0115e0863cb445a67
Author: Tzu-Li (Gordon) Tai 
Date:   2017-09-27T11:02:32Z

[FLINK-7652] [flip6] Port CurrentJobIdsHandler to new REST endpoint

commit c8653b1cf78a2695db757b5c9a89b1125c74f1ea
Author: Tzu-Li (Gordon) Tai 
Date:   2017-09-27T11:20:07Z

[FLINK-7652] [flip6] Rename JobsWithIdsOverview message class name

Renamed to JobsStatusesWithIdsOverview, to better respresent the
information that the message holds.

commit 662fe9cb64849b8af7758f9facaaa1dc65e6fb93
Author: Tzu-Li (Gordon) Tai 
Date:   2017-09-27T12:09:43Z

[FLINK-7652] [flip6] Let JobStatusesWithIdsOverview differentiate between 
all possible JobStatuses

Prior to this commit, the JobStatusesWithIdsOverview only differentiates
between RUNNING, FAILED, CANCELLED, and FINISHED. This commit fixes that
to cover all possible JobStatus, including CREATED, CANCELLING, FAILING,
RESTARTING, SUSPENDED, and RECONCILING.




> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
>  Issue Type: Sub-task
>  Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>  Labels: flip-6
> Fix For: 1.4.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)