[ 
https://issues.apache.org/jira/browse/HADOOP-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714436#action_12714436
 ] 

Steve Loughran commented on HADOOP-5933:
----------------------------------------

There isn't direct caching for DFSClients, but there is in Filesystem.get(), 
which is how I've been getting DFSClient instances (and those of other 
filesystems)

Up until march I could have different things get filesystems, do some work and 
then close them, but now in SVN_HEAD I'm seeing stack traces when different 
threads try to work with filesystem instances they have been holding on to. So 
one thread running a TaskTracker is happily spinning away, until something does 
a quick check on a different thread that a specific file exists in the 
fileysystem, does a close afterwards. 

My stack traces are here: http://jira.smartfrog.org/jira/browse/SFOS-1208

The semantics of {{FileSystem.get()}} have changed; if I moved my code to the 
new {{FileSystem.newInstance()}} method then things should work again. That 
doesn't mean we dont benefit from tracing who closed the instance, only that 
anyone else doing work in different threads who were getting the filesystem 
clients by way of {{FileSystem.get()}}  are going to encounter the same 
problems. I just saw them first :)

> Make it harder to accidentally close a shared DFSClient
> -------------------------------------------------------
>
>                 Key: HADOOP-5933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5933
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-5933.patch
>
>
> Every so often I get stack traces telling me that DFSClient is closed, 
> usually in {{org.apache.hadoop.hdfs.DFSClient.checkOpen() }} . The root cause 
> of this is usually that one thread has closed a shared fsclient while another 
> thread still has a reference to it. If the other thread then asks for a new 
> client it will get one -and the cache repopulated- but if has one already, 
> then I get to see a stack trace. 
> It's effectively a race condition between clients in different threads. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to