[
https://issues.apache.org/jira/browse/HADOOP-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332987#comment-15332987
]
Yiqun Lin commented on HADOOP-13264:
------------------------------------
I think this problem is different with HDFS-9812. HDFS-9812 solved the problem
that {{datastreamer}} thread not closed when failures happened in flushing
data, and these logic was done in {{closeImpl}}. In this probloem, if the
method {{closeImpl}} threw the IOException, the dfsClient.endFileLease(fileId)
will not be called. If we want to fix this, I suggest that we would be better
to keep the synchronized block code, like this:
{code}
public void close() throws IOException {
boolean threwException = false;
synchronized (this) {
try (TraceScope ignored =
dfsClient.newPathTraceScope("DFSOutputStream#close", src)) {
closeImpl();
} catch (IOException ioe) {
threwException = true;
}
}
dfsClient.endFileLease(fileId);
if (threwException) {
throw new IOException("Exception happened in closing the output stream.");
}
}
{code}
Correct me if I am wrong, thanks.
> Hadoop HDFS - DFSOutputStream close method fails to clean up resources in
> case no hdfs datanodes are accessible
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13264
> URL: https://issues.apache.org/jira/browse/HADOOP-13264
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.7.2
> Reporter: Seb Mo
>
> Using:
> hadoop-hdfs\2.7.2\hadoop-hdfs-2.7.2-sources.jar!\org\apache\hadoop\hdfs\DFSOutputStream.java
> Close method fails when the client can't connect to any data nodes. When
> re-using the same DistributedFileSystem in the same JVM, if all the datanodes
> can't be accessed, then this causes a memory leak as the
> DFSClient#filesBeingWritten map is never cleared after that.
> See test program provided by [~sebyonthenet] in comments below.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
