[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741480#comment-13741480
]
Colin Patrick McCabe commented on HDFS-4504:
--------------------------------------------
A few tests needed to be fixed, since they were assuming that a single
{{IOException}} would cause {{DFSOutputStream#close}} to give up immediately,
whereas that is no longer the case. Setting
{{DFSConfigKeys.DFS_CLIENT_CLOSE_TIMEOUT_MS}} to a low value allows the tests
to complete in a reasonable amount of time.
{{FSNamesystem#completeInternal(force=true)}} needed to call
{{internalReleaseLease}} to properly finalize the last block in the file.
Failure to do this could put us in a situation where the NameNode thinks the
block is finalized, but the DataNode(s) do not.
> DFSOutputStream#close doesn't always release resources (such as leases)
> -----------------------------------------------------------------------
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch,
> HDFS-4504.007.patch, HDFS-4504.008.patch, HDFS-4504.009.patch,
> HDFS-4504.010.patch, HDFS-4504.011.patch, HDFS-4504.014.patch,
> HDFS-4504.015.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
