[
https://issues.apache.org/jira/browse/MAPREDUCE-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805244#action_12805244
]
Steve Loughran commented on MAPREDUCE-437:
------------------------------------------
the problem with the tests failing if the JT is set to close its filesystem
when shut down is triggered by the FS caching
Two solutions
# loading these tests using the HADOOP-6231 trick with a config with
{{fs.hdfs.impl.disable.cache=true}}. Fixes the tests, leaves the problem
lurking around
# have the JT connect to the filesystem using {{FileSystem.newInstance()}}
instead of {{FileSystem.get()}}
I would favour #2 as I can't see any reason why you'd want to share the
filesystem reference for the JT with anything else running in the same VM,
including JUnit tests, as it only changes system behavior. With a move over to
{{newInstance()}} the JT can close the filesystem on termination without any
concerns about adverse consequences.
> JobTracker may need to close its filesystem when being terminated
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-437
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-437
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Steve Loughran
> Priority: Minor
>
> This is something I've been experimenting with HADOOP-3268; I'm not sure what
> the right action is here.
> -currently, the JobTracker does not close() its filesystem when it is shut
> down. This will cause it to leak filesystem references if JobTrackers are
> started and stopped in the same process.
> -The TestMRServerPorts test explicitly closes the filesystem
> jt.fs.close();
> jt.stopTracker();
> -If you move the close() operation into the stopTracker()/terminate logic,
> the filesystem gets cleaned up, but
> TestRackAwareTaskPlacement and TestMultipleLevelCaching fail with a
> FilesystemClosed error (stack traces to follow)
> Should the JobTracker close its filesystem whenever it is terminated? If so,
> there are some tests that need to be reworked slightly to not expect the
> fileystem to be live after the jobtracker is taken down.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
