[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531148#comment-16531148
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5888


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6, pull-request-available
> Fix For: 1.5.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475441#comment-16475441
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5902


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466736#comment-16466736
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user yuqi1129 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r186599431
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -162,7 +166,7 @@ public void run() {
String 
json = archive.getJson();
 
File 
target;
-   if 
(path.equals("/joboverview")) {
--- End diff --

Agree


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465691#comment-16465691
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r186376796
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -162,7 +166,7 @@ public void run() {
String 
json = archive.getJson();
 
File 
target;
-   if 
(path.equals("/joboverview")) {
--- End diff --

this change effectively kills backwards compatibility; i.e. a 1.4 HS cannot 
properly handle 1.5 archives, and a 1.5 HS cannot properly handle 1.4 archives. 
What is missing is a compatibility check for the old URL.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449809#comment-16449809
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r183720759
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -211,7 +215,7 @@ public void run() {
}
}
if (updateOverview) {
-   updateJobOverview(webDir);
+   
updateJobOverview(webOverviewDir, webDir);
}
--- End diff --

yes.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449741#comment-16449741
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user yuqi1129 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r183709218
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -211,7 +215,7 @@ public void run() {
}
}
if (updateOverview) {
-   updateJobOverview(webDir);
+   
updateJobOverview(webOverviewDir, webDir);
}
--- End diff --

In test case, it's quite clear this is make the HS has polled file before 
read, That is to say adding `numFinishedPolls` is just for test purpose, am i 
right ？


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449736#comment-16449736
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r183707159
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -211,7 +215,7 @@ public void run() {
}
}
if (updateOverview) {
-   updateJobOverview(webDir);
+   
updateJobOverview(webOverviewDir, webDir);
}
--- End diff --

We allow passing in a countdown latch for testing purposes. Since the 
fetching happen asynchronously we needed a way to wait until the HS has polled 
the directories.

The `CountdownLatch` is initialized with `1` in `HistoryServerTest`.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449725#comment-16449725
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user yuqi1129 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r183702452
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -211,7 +215,7 @@ public void run() {
}
}
if (updateOverview) {
-   updateJobOverview(webDir);
+   
updateJobOverview(webOverviewDir, webDir);
}
--- End diff --

Could not figure out the function of `numFinishedPolls.countDown();`in line 
224, seems that `numFinishedPolls`   is useless as it was initialized 0


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449724#comment-16449724
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user yuqi1129 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5902#discussion_r183697720
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/JobVertexTaskManagersHandler.java
 ---
@@ -83,6 +90,24 @@ protected JobVertexTaskManagersInfo handleRequest(
throw new NotFoundException(String.format("JobVertex %s 
not found", jobVertexID));
}
 
+   return createJobVertexTaskManagersInfo(jobVertex, jobID, 
metricFetcher);
+   }
+
+   @Override
+   public Collection 
archiveJsonWithPath(AccessExecutionGraph graph) throws IOException {
+   Collection vertices = 
graph.getAllVertices().values();
+   List archive = new ArrayList<>(vertices.size());
+   for (AccessExecutionJobVertex task : vertices) {
+   ResponseBody json = 
createJobVertexTaskManagersInfo(task, graph.getJobID(), null);
+   String path = 
getMessageHeaders().getTargetRestEndpointURL()
+   .replace(':' + JobIDPathParameter.KEY, 
graph.getJobID().toString())
--- End diff --

```
':' + JobIDPathParameter.KEY
':' + JobVertexIdPathParameter.KEY

```
We can make them constant values


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449491#comment-16449491
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/5902

[FLINK-9194][history] Add HistoryServer support to flip6

This PR subsumes #5888.

## What is the purpose of the change

This PR adds HistoryServer support to Flip6, i.e. allows the Dispatcher to 
archive finished jobs. Additionally the HistoryServer was updated to reflect 
the latest changes to the job overview page.


## Brief change log

* fix job overview merging of the HistoryServer (#5888)
* replace some manual JSON reading/writing with mappings to/from 
`MultipleJobDetails`
* add a convenience `ArchivedJson` constructor that accepts a `ResponseBody`
* add `JsonArchivist` argument to `Dispatcher` constructor
  * the archivist creates a `Collection` which is passed to 
`FsJobArchivist` for writing to disk
  * `WebMonitorEndpoint` now implements `JsonArchivist`
* Modify relevant handlers to implement `JsonArchivist`
  * this is different to the existing implementation of using static 
factories
  * archiving handlers are detected by the `WebMonitorEndpoint` which is 
significantly less brittle as the existing approach of separately listing 
archivers
* rework the `HistoryServerTest` to run multiple small jobs against a 
`MiniClusterResource`
  * this better reflects the actual behavior and ensures we don't break 
things again

## Verifying this change

* run `HistoryServerTest`
* manually verify that Flip6 clusters can archive jobs, and ensure that 
HistoryServer can read them.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 9194

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5902.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5902


commit aaf02376a0b4c16e454d0f3b733cee9edd20ef54
Author: hzyuqi1 
Date:   2018-04-24T08:25:10Z

[FLINK-9246][HS] Adjust HistoryServer for job overview changes

commit 722590fa784f36051d8aab0ed01d3cae0c8f8200
Author: zentol 
Date:   2018-04-24T08:26:32Z

[hotfix][history] Read/Write MultipleJobsDetails instead of manual JSON

commit 45bca6b3f996852ca85827903b5a9e90fa2793c1
Author: zentol 
Date:   2018-04-18T12:33:04Z

[FLINK-9194][history] Add archiving routine to Dispatcher

commit de0684c548e5c9e966954082dd963358567997cc
Author: zentol 
Date:   2018-04-18T12:33:16Z

[FLINK-9194][history] Add convenience ArchivedJson constructor

commit 8a0e9e28d429b2cb804dcef485adcd83c58e3b07
Author: zentol 
Date:   2018-04-18T12:34:25Z

[FLINK-9194][history] Adjust handlers

commit 21d0bc0e5fc21d8e03a6c0b92014f0d49b62b0cb
Author: zentol 
Date:   2018-04-23T13:13:41Z

[FLINK-9194][history] Rework and extend the HistoryServer test




> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449492#comment-16449492
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5888
  
see #5902.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449153#comment-16449153
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user yuqi1129 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183392156
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -180,10 +180,22 @@ public void run() {
 
// We 
overwrite existing files since this may be another attempt at fetching this 
archive.
// 
Existing files may be incomplete/corrupt.
+
+   //this 
is to save a job overview information, we put it in the ${path}/jobId folder
+   
//because if put it in ${path}, new job inforamtion will override it and the web
+   //will 
show only one job all the same.
+   if 
(!target.toURI().toURL().getPath().contains(jobID)) {
--- End diff --

Yes, Indeed  change Line 165 will make things more simple


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448092#comment-16448092
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5888
  
I'm currently preparing a PR for adding HistoryServer support to Flip6 and 
will integrate this PR into that one.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447976#comment-16447976
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183355330
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -180,10 +180,22 @@ public void run() {
 
// We 
overwrite existing files since this may be another attempt at fetching this 
archive.
// 
Existing files may be incomplete/corrupt.
+
+   //this 
is to save a job overview information, we put it in the ${path}/jobId folder
+   
//because if put it in ${path}, new job inforamtion will override it and the web
+   //will 
show only one job all the same.
+   if 
(!target.toURI().toURL().getPath().contains(jobID)) {
--- End diff --

this isn't the right fix. This case _was_ handled at Line 165, but was 
rendered ineffective since the URL was changed.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447971#comment-16447971
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r18334
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -180,10 +180,22 @@ public void run() {
 
// We 
overwrite existing files since this may be another attempt at fetching this 
archive.
// 
Existing files may be incomplete/corrupt.
+
+   //this 
is to save a job overview information, we put it in the ${path}/jobId folder
+   
//because if put it in ${path}, new job inforamtion will override it and the web
+   //will 
show only one job all the same.
+   if 
(!target.toURI().toURL().getPath().contains(jobID)) {
+   
Files.deleteIfExists(targetPath);
+   
targetPath = new File(target.getParent() + "/" + jobID + "/" + 
target.getName()).toPath();
+   }

Files.deleteIfExists(targetPath);
 
-   
Files.createFile(target.toPath());
-   try 
(FileWriter fw = new FileWriter(target)) {
+   if 
(!targetPath.toFile().getParentFile().exists()) {
--- End diff --

this change is unnecessary if the above change is reverted, since 
directories are already created at Line 174.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447973#comment-16447973
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183356809
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -231,18 +243,20 @@ public void run() {
 * For the display in the HistoryServer WebFrontend we have to 
combine these overviews.
 */
private static void updateJobOverview(File webDir) {
-   File webOverviewDir = new File(webDir, "overviews");
-   try (JsonGenerator gen = 
jacksonFactory.createGenerator(HistoryServer.createOrGetFile(webDir, 
"joboverview"))) {
+   File webOverviewDir = new File(webDir, "/jobs/");
+   try (JsonGenerator gen = 
jacksonFactory.createGenerator(HistoryServer.createOrGetFile(webDir, 
"/jobs/overview"))) {
gen.writeStartObject();
-   gen.writeArrayFieldStart("running");
-   gen.writeEndArray();
-   gen.writeArrayFieldStart("finished");
+   gen.writeArrayFieldStart("jobs");
 
File[] overviews = new 
File(webOverviewDir.getPath()).listFiles();
if (overviews != null) {
for (File overview : overviews) {
-   JsonNode root = 
mapper.readTree(overview);
-   JsonNode finished = 
root.get("finished");
+   if (overview.isFile()) {
+   continue;
+   }
+   File file = new 
File(overview.getAbsolutePath() + "/overview.json");
+   JsonNode root = mapper.readTree(file);
--- End diff --

A much safer implementation would be to parse these into 
`MultipleJobsDetails`, merge them and write them back out.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447972#comment-16447972
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183355953
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -180,10 +180,22 @@ public void run() {
 
// We 
overwrite existing files since this may be another attempt at fetching this 
archive.
// 
Existing files may be incomplete/corrupt.
+
+   //this 
is to save a job overview information, we put it in the ${path}/jobId folder
+   
//because if put it in ${path}, new job inforamtion will override it and the web
+   //will 
show only one job all the same.
+   if 
(!target.toURI().toURL().getPath().contains(jobID)) {
+   
Files.deleteIfExists(targetPath);
+   
targetPath = new File(target.getParent() + "/" + jobID + "/" + 
target.getName()).toPath();
+   }

Files.deleteIfExists(targetPath);
 
-   
Files.createFile(target.toPath());
-   try 
(FileWriter fw = new FileWriter(target)) {
+   if 
(!targetPath.toFile().getParentFile().exists()) {
+   
Files.createDirectory(targetPath.getParent());
+   }
+
+   
Files.createFile(targetPath);
+   try 
(FileWriter fw = new FileWriter(targetPath.toFile())) {
--- End diff --

this change is unnecessary if the above change is reverted


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447975#comment-16447975
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183357018
  
--- Diff: 
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/history/HistoryServerTest.java
 ---
@@ -96,9 +108,26 @@ public void testFullArchiveLifecycle() throws Exception 
{
String response = getFromHTTP(baseUrl + 
JobsOverviewHeaders.URL);
JsonNode overview = mapper.readTree(response);
 
-   String jobID = 
overview.get("jobs").get(0).get("jid").asText();
-   JsonNode jobDetails = 
mapper.readTree(getFromHTTP(baseUrl + "/jobs/" + jobID));
-   Assert.assertNotNull(jobDetails.get("jid"));
+   JsonNode allJobs = overview.get("jobs");
--- End diff --

safer implementation is now possible by reading this as a 
`MultipleJobsDetails`.


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447974#comment-16447974
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183356576
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -231,18 +243,20 @@ public void run() {
 * For the display in the HistoryServer WebFrontend we have to 
combine these overviews.
 */
private static void updateJobOverview(File webDir) {
-   File webOverviewDir = new File(webDir, "overviews");
-   try (JsonGenerator gen = 
jacksonFactory.createGenerator(HistoryServer.createOrGetFile(webDir, 
"joboverview"))) {
+   File webOverviewDir = new File(webDir, "/jobs/");
+   try (JsonGenerator gen = 
jacksonFactory.createGenerator(HistoryServer.createOrGetFile(webDir, 
"/jobs/overview"))) {
--- End diff --

replace webDir with webOverviewDir and adjust the child path accordingly


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447970#comment-16447970
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5888#discussion_r183356514
  
--- Diff: 
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
 ---
@@ -231,18 +243,20 @@ public void run() {
 * For the display in the HistoryServer WebFrontend we have to 
combine these overviews.
 */
private static void updateJobOverview(File webDir) {
-   File webOverviewDir = new File(webDir, "overviews");
-   try (JsonGenerator gen = 
jacksonFactory.createGenerator(HistoryServer.createOrGetFile(webDir, 
"joboverview"))) {
+   File webOverviewDir = new File(webDir, "/jobs/");
--- End diff --

the slashes are unnecessary, and the directory should be explicitly created 
here


> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446852#comment-16446852
 ] 

ASF GitHub Bot commented on FLINK-9194:
---

GitHub user yuqi1129 opened a pull request:

https://github.com/apache/flink/pull/5888

[FLINK-9194] [histroyserver] Finished jobs are not archived to Histor…


## What is the purpose of the change

This is to solve the problem that history server can only show only one 
completed job even though there exists several completed jobs in the archive 
dir. 


## Brief change log

  - Change the class `HistoryServerArchiveFetcher.JobArchiveFetcherTask`  
and the main change is to place the overview of each job in the folder of its' 
jobId. Final, traverse all the overviews and place them together.


## Verifying this change



This change is already covered by existing tests, such as 
`HistoryServerTest` and we add some test case in this test class.


## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yuqi1129/flink historyserver1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5888.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5888


commit bdbf6c6cba537887ace49b28caf66896722cc0bc
Author: hzyuqi1 
Date:   2018-04-20T16:03:54Z

[FLINK-9194] [histroyserver] Finished jobs are not archived to HistoryServer

This closes #9194




> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445807#comment-16445807
 ] 

Aljoscha Krettek commented on FLINK-9194:
-

Moved this to 1.6.0 to unblock the release, please discuss if this is urgent 
for you.

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.6.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443640#comment-16443640
 ] 

Chesnay Schepler commented on FLINK-9194:
-

[~jstehler] That issue should be unrelated as it is about 1.4.

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Jared Stehler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443357#comment-16443357
 ] 

Jared Stehler commented on FLINK-9194:
--

Could this issue possibly link with 
https://issues.apache.org/jira/browse/FLINK-9030 ?

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[Gary Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441927#comment-16441927
 ] 

Gary Yao commented on FLINK-9194:
-

[~yuqi] I configured {{jobmanager.archive.fs.dir}}, and started the History 
Server as described here: 
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/historyserver.html#overview
It cannot work because the code is not there.

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9194) Finished jobs are not archived to HistoryServer

[yuqi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441846#comment-16441846
 ] 

yuqi commented on FLINK-9194:
-

[~gjy] Could not reproduce this problem with the environment you provide, more 
detail about it?

> Finished jobs are not archived to HistoryServer
> ---
>
> Key: FLINK-9194
> URL: https://issues.apache.org/jira/browse/FLINK-9194
> Project: Flink
>  Issue Type: Bug
>  Components: History Server, JobManager
>Affects Versions: 1.5.0
> Environment: Flink 2af481a
>Reporter: Gary Yao
>Assignee: Chesnay Schepler
>Priority: Blocker
>  Labels: flip-6
>
> In flip6 mode, jobs are not archived to the HistoryServer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)