[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737627#comment-13737627
]
Andrew Wang commented on HDFS-4504:
-----------------------------------
Some light review comments, I'd prefer if someone more experienced with the
write path also reviewed this before committing.
{code}
+ // We couldn't contact the NameNode (or it refused to close the file.)
{code}
Nit: period goes outside the parenthesis.
* TreeMaps are memory inefficient, so I'd prefer to use something else unless
we need ordering. I believe {{ConcurrentHashMap}} has safe concurrent iteration
behavior.
* Any reason for the StreamInfo class over a Long?
* TestHdfsClose: should use {{GenericTestUtils#assertExceptionContains}} when
catching an expected IOException.
* Unrelated to this patch, but {{DFSClient#endFileLease}} is poorly named,
since it doesn't actually end the lease. Maybe rename it to
{{stopLeaseRenewal}}?
* I noticed that {{endFileLease}} is now called before {{completeFile}}, could
the lease expire between these two calls? I guess it'll still get cleaned up
okay by the ZSM, but this is extra work.
> DFSOutputStream#close doesn't always release resources (such as leases)
> -----------------------------------------------------------------------
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch,
> HDFS-4504.007.patch, HDFS-4504.008.patch, HDFS-4504.009.patch,
> HDFS-4504.010.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
