[
https://issues.apache.org/jira/browse/HADOOP-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617369#action_12617369
]
Ankur commented on HADOOP-3818:
-------------------------------
The use case that we hit upon was client flushing data from Apache servers
directly onto hadoop clusters and client dying because of unexpected server
failures in which case the client receives SIGTERM and gets killed. What we
want to do here is simply close our output stream before exiting so that data
isn't lost.
But I can see other use cases where people might want to do additional
operations in the client like renaming, moving etc (besides deleting and
closing).
So I see a couple of approaches for this.
1. Provide additional API's like closeOnExit(..), moveOnExit(...) etc. which I
think would not be so elegant and somewhat limited in the control they provide.
2. Provide additional API in FileSystem, something like
FileSystem.addShutdownEventReciever() that allows clients to register there
custom cleanup code. In the shutdown hook, make sure that all the exit handlers
registered by client are invoked before any code in the shutdown hook. This is
more flexible but relies on the fact there is only one shutdown hook thread
that runs once JVM starts to shutdown which currently is the case, I think.
> Not possible to access a FileSystem from within a ShutdownHook
> --------------------------------------------------------------
>
> Key: HADOOP-3818
> URL: https://issues.apache.org/jira/browse/HADOOP-3818
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.17.1
> Reporter: Rowan Nairn
> Priority: Minor
>
> FileSystem uses addShutdownHook to close all FileSystems at exit. This makes
> it impossible to access a FileSystem from within your own ShutdownHook
> threads, say for deleting incomplete output. Using a pre-existing FileSystem
> object is unsafe since it may be closed by the time the thread executes.
> Using FileSystem.get(...) results in an exception:
> Exception in thread "Thread-10" java.lang.IllegalStateException: Shutdown in
> progress
> at java.lang.Shutdown.add(Shutdown.java:81)
> at java.lang.Runtime.addShutdownHook(Runtime.java:190)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1293)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
