[
https://issues.apache.org/jira/browse/HDFS-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4725:
--------------------------------
Attachment: HDFS-4725.1.patch
I'm attaching a patch to address the file handle leaks that I've found. Here
is a summary of the changes:
# Namenode shutdown during an early termination (fatal error) did not guarantee
complete cleanup. I've made changes to {{NameNode}} and {{FSEditLog}} to
guarantee all edits logs get closed and storage lock files get released via
{{FSNamesystem#close}}. Previously, we could skip closing edits logs and
releasing lock files when {{ExitUtil#terminate}} was running in test mode, and
therefore throwing an unchecked exception instead of exiting the process.
# {{OfflineEditsBinaryLoader}} was not closing the edits log it was reading.
I've added a finally block to guarantee that it gets closed.
# {{DFSTestUtil#loadFile}} had a file handle leak, so I added a finally block
to guarantee close.
# Various tests were not calling shutdown on their {{MiniDFSCluster}}
instances. You'll see some big diffs from where I added finally blocks to
guarantee shutdown, but this is just due to the indentation changes.
# {{TestEditLogJournalFailures}} had set up a mock to do nothing when anything
called {{EditLogOutputStream#abort}}. Under normal operation, the file would
get closed in that method, so the mocking would cause tests to leak file
handles. From what I can tell, there is no particular reason that the tests
need to skip the abort, so I just removed that part of the mocking.
I tested this successfully on Mac and Windows. With this patch, we'll stop
seeing failures on Windows in these test suites:
* {{TestEditLogJournalFailures}}
* {{TestStartup}}
* {{TestDatanodeDeath}}
* {{TestDFSClientExcludedNodes}}
* {{TestDistributedFileSystem}}
* {{TestQuota}}
* {{TestOfflineEditsViewer}}
> fix HDFS file handle leaks
> --------------------------
>
> Key: HDFS-4725
> URL: https://issues.apache.org/jira/browse/HDFS-4725
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test, tools
> Affects Versions: 3.0.0
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HDFS-4725.1.patch
>
>
> The scope of this issue is to fix multiple file handle leaks observed from
> recent HDFS test runs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
