[
https://issues.apache.org/jira/browse/FLINK-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538860#comment-16538860
]
ASF GitHub Bot commented on FLINK-9750:
---------------------------------------
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201369555
--- Diff:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java
---
@@ -253,4 +265,39 @@ public CommitRecoverable getRecoverable() {
return recoverable;
}
}
+
+ /**
+ * Called when resuming execution after a failure and waits until the
lease
+ * of the file we are resuming is free.
+ *
+ * <p>The lease of the file we are resuming writing/committing to may
still
+ * belong to the process that failed previously and whose state we are
+ * recovering.
+ *
+ * @param path The path to the file we want to resume writing to.
+ */
+ private boolean waitUntilLeaseIsRevoked(final Path path) throws
IOException {
+ Preconditions.checkState(fs instanceof DistributedFileSystem);
+
+ final DistributedFileSystem dfs = (DistributedFileSystem) fs;
+ dfs.recoverLease(path);
+ boolean isclosed = dfs.isFileClosed(path);
+
+ final StopWatch sw = new StopWatch();
--- End diff --
Let's use `Deadline` from the Flink utils instead to reduce external
dependencies.
> Create new StreamingFileSink that works on Flink's FileSystem abstraction
> -------------------------------------------------------------------------
>
> Key: FLINK-9750
> URL: https://issues.apache.org/jira/browse/FLINK-9750
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming Connectors
> Reporter: Stephan Ewen
> Assignee: Kostas Kloudas
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Using Flink's own file system abstraction means that we can add additional
> streaming/checkpointing related behavior.
> In addition, the new StreamingFileSink should only rely on internal
> checkpointed state what files are possibly in progress or need to roll over,
> never assume enumeration of files in the file system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
