[
https://issues.apache.org/jira/browse/HADOOP-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seb Mo updated HADOOP-13264:
----------------------------
Summary: Hadoop HDFS - DFSOutputStream close method fails to clean up
resources in case no hdfs datanodes are accessible (was: DFSOutputStream
close method fails to clean up resources in case no hdfs datanodes are
accessible )
> Hadoop HDFS - DFSOutputStream close method fails to clean up resources in
> case no hdfs datanodes are accessible
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13264
> URL: https://issues.apache.org/jira/browse/HADOOP-13264
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.7.2
> Reporter: Seb Mo
>
> Using:
> hadoop-hdfs\2.7.2\hadoop-hdfs-2.7.2-sources.jar!\org\apache\hadoop\hdfs\DFSOutputStream.java
> Close method fails when the client can't connect to any data nodes. When
> re-using the same DistributedFileSystem in the same JVM, if all the datanodes
> can't be accessed, then this causes a memory leak as the
> DFSClient#filesBeingWritten map is never cleared after that.
> Here is a test program:
> public static void main(String args[]) throws Exception
> {
> final Configuration conf = new Configuration();
> conf.addResource(new FileInputStream(new
> File("core-site.xml")));
> conf.addResource(new FileInputStream(new
> File("hdfs-site.xml")));
> final DistributedFileSystem newFileSystem =
> (DistributedFileSystem)FileSystem.get(conf);
> OutputStream outputStream = null;
> try
> {
> outputStream = newFileSystem.create(new
> Path("/user/ssmogos", "test1"));
> outputStream.write("test".getBytes());
> }
> catch (IOException e)
> {
> e.printStackTrace();//don't care about this
> }
> finally
> {
> try
> {
> if (outputStream != null)
> outputStream.close();//now this one
> will fail to close the stream
> }
> catch (IOException e)
> {
> e.printStackTrace();//this will list the thrown
> exception from DFSOutputStream->flushInternal->checkClosed
> //TODO the
> DFSOutputStream#close->dfsClient.endFileLease(fileId) is never getting closed
> }
> }
> Field field =
> DFSClient.class.getDeclaredField("filesBeingWritten");
> field.setAccessible(true);
> System.out.print("THIS SHOULD BE EMPTY: " +
> field.get(newFileSystem.getClient()));
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
