Since FileSystem is a Closeable i would expect code using it to be like
this:
FileSystem fs = path.getFileSystem(conf);
try {
// do something with fs, such as read from the path
} finally {
fs.close()
}
However i have repeatedly gotten into trouble with this approach. In one
situation it turned out that when i closed a FileSystem other operations
that were using their own FileSystems (pointing to the same real-world HDFS
filesystem) also saw their FileSystems closed, leading to very confusing
read and write errors. This led me to believe that FileSystem should never
be closed since it seemed to act like some sort of Singleton. However now
was just looking at some code (Hoop server, to be precise) and noticed that
FileSystems were indeed closed, but they were always threadlocal. Is this
the right approach?
And if FileSystem is threadlocal, is this safe (assuming fs1 and fs2 could
point to the same real-world filesystem)?
FileSystem fs1 = path.getFileSystem(conf);
try {
FileSystem fs2 = path.getFileSystem(conf);
try {
// do something with fs2, such as read from the path
} finally {
fs2.close()
}
// do something with fs1, such as read from the path (note, fs2 is
closed here, and i wouldn't be surprised if fs1 by now is also closed given
my experience)
} finally {
fs1.close()
}
