[
https://issues.apache.org/jira/browse/FLINK-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15899241#comment-15899241
]
ASF GitHub Bot commented on FLINK-1579:
---------------------------------------
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3460#discussion_r104640647
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/MemoryArchivist.scala
---
@@ -183,6 +193,33 @@ class MemoryArchivist(private val max_entries: Int)
}
}
+ private def archiveJsonFiles(graph: ArchivedExecutionGraph) {
+ future {
+ val rootPath = new Path(flinkConfiguration.getString(
+ JobManagerOptions.ARCHIVE_DIR))
+ val fs = rootPath.getFileSystem
+ val tmpArchivePath = new Path(rootPath,
s"tmp_${graph.getJobID.toString}")
+ for (archiver <- WebMonitorUtils.getArchivers) {
+ try {
+ for (archive <- archiver.archiveJsonWithPath(graph).asScala) {
+ val targetPath =
+ new Path(tmpArchivePath, s"${archive.getPath}.json")
+ val out = fs.create(targetPath, false)
+ out.write(archive.getJson.getBytes(StandardCharsets.UTF_8))
+ out.close()
+ }
+ } catch {
+ case ioe: IOException => {
+ log.error("Failed to archive job details.", ioe)
+ }
+ }
+ }
+ if (!fs.rename(tmpArchivePath, new Path(rootPath,
s"${graph.getJobID.toString}"))) {
--- End diff --
yes. Once we write out 1 file this should no longer be an issue.
> Create a Flink History Server
> -----------------------------
>
> Key: FLINK-1579
> URL: https://issues.apache.org/jira/browse/FLINK-1579
> Project: Flink
> Issue Type: New Feature
> Components: Distributed Coordination
> Affects Versions: 0.9
> Reporter: Robert Metzger
> Assignee: Chesnay Schepler
>
> Right now its not possible to analyze the job results for jobs that ran on
> YARN, because we'll loose the information once the JobManager has stopped.
> Therefore, I propose to implement a "Flink History Server" which serves the
> results from these jobs.
> I haven't started thinking about the implementation, but I suspect it
> involves some JSON files stored in HDFS :)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
