[
https://issues.apache.org/jira/browse/FLINK-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611879#comment-15611879
]
ASF GitHub Bot commented on FLINK-4910:
---------------------------------------
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/2691#discussion_r85334057
--- Diff: flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java
---
@@ -55,6 +60,40 @@
private static final String HADOOP_WRAPPER_SCHEME = "hdwrapper";
+ private static final Logger LOG =
LoggerFactory.getLogger(FileSystem.class);
+
+ /**
+ * Create a SafetyNetCloseableRegistry for a Task. This method should
be called at the beginning of the task's
+ * main thread.
+ */
+ public static void createFileSystemCloseableRegistryForTask() {
+ SafetyNetCloseableRegistry current = REGISTRIES.get();
+ if (null != current) {
+ if (!current.isClosed()) {
+ IOUtils.closeQuietly(current);
+ LOG.warn("Found open
SafetyNetCloseableRegistry. Closed and replaced it.");
+ }
+ }
+ REGISTRIES.set(new SafetyNetCloseableRegistry());
+ }
+
+ /**
+ * Create a SafetyNetCloseableRegistry for a Task. This method should
be called at the end of the task's
+ * main thread or when the task should be canceled.
+ */
+ public static void disposeFileSystemCloseableRegistryForTask() {
+ SafetyNetCloseableRegistry ret = REGISTRIES.get();
+ if (null != ret) {
+ LOG.info("Closing SafetyNetCloseableRegistry.");
--- End diff --
Let's log something like "Ensuring all FileSystem streams are closed".
Users will understand that better ;-)
> Introduce safety net for closing file system streams
> ----------------------------------------------------
>
> Key: FLINK-4910
> URL: https://issues.apache.org/jira/browse/FLINK-4910
> Project: Flink
> Issue Type: Improvement
> Reporter: Stefan Richter
> Assignee: Stefan Richter
>
> Streams that are opened through {{FileSystem}} must be closed at the end of
> their life cycle. However, we found hints that some code forgets to close
> such streams.
> We should introduce i) a mechanism that closes leaking unclosed streams after
> usage and ii) provides logging that helps us to track down and fi the sources
> of such leaks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
