[
https://issues.apache.org/jira/browse/METAMODEL-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706554#comment-14706554
]
Tomasz Guzialek commented on METAMODEL-179:
-------------------------------------------
I tried to reproduce that in the unit test, but it is not easy. I will post the
code here and hope it is useful despite the fact that this test case fails only
sometimes (race condition):
{code:title=HdfsResourceIntegrationTest.java|borderStyle=solid}
@Test
public void testFileSystemNotBeingClosed() throws IOException {
HdfsResource resourceToRead = null;
try {
resourceToRead = new HdfsResource(_hostname, _port, _filePath);
resourceToRead.write(new Action<OutputStream>() {
@Override
public void run(OutputStream out) throws Exception {
FileHelper.writeString(out, "testFileSystemNotBeingClosed");
}
});
Thread.UncaughtExceptionHandler exceptionHandler = new
Thread.UncaughtExceptionHandler() {
public void uncaughtException(Thread th, Throwable ex) {
Assert.fail("Caught exception in the thread: " + ex);
}
};
for (int i = 0; i < 10; i++) {
final HdfsResource res = new HdfsResource(_hostname, _port,
_filePath);
Thread thread = new Thread(new Runnable() {
@Override
public void run() {
res.read(new Action<InputStream>() {
@Override
public void run(InputStream is) throws Exception {
String readAsString =
FileHelper.readAsString(FileHelper.getReader(is, "UTF-8"));
Assert.assertNotNull(readAsString);
Assert.assertEquals("testFileSystemNotBeingClosed", readAsString);
}
});
}
});
thread.setUncaughtExceptionHandler(exceptionHandler);
thread.start();
}
} finally {
if (resourceToRead != null) {
resourceToRead.getHadoopFileSystem().delete(resourceToRead.getHadoopPath(),
false);
}
}
}
{code}
> FileSystem instance in HdfsResource should not be closed
> --------------------------------------------------------
>
> Key: METAMODEL-179
> URL: https://issues.apache.org/jira/browse/METAMODEL-179
> Project: Apache MetaModel
> Issue Type: Bug
> Reporter: Tomasz Guzialek
> Priority: Critical
>
> In HdfsResource, a FileSystem instance is obtained via get() method.
> According to [1] this method does not return a new instance every time, but
> caches it for the same configuration. Closing such instance may result in
> IOException: Filesystem closed in other, possibly unrelated, places in the
> code also using this FileSystem instance.
> Possible solutions:
> 1) Despite a Java convention to always close Closable objects after use, we
> should leave without closing. Apparently, Hive does it this way (see [1])
> 2) Create instances of FileSystem with newInstance() method, but this is an
> expensive operation.
> [1]
> http://stackoverflow.com/questions/20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close/20061797#20061797
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
