BoYang created HADOOP-10606:
-------------------------------
Summary: NodeManager cannot launch container when using
RawLocalFileSystem for fs.file.impl
Key: HADOOP-10606
URL: https://issues.apache.org/jira/browse/HADOOP-10606
Project: Hadoop Common
Issue Type: Bug
Components: fs, io, util
Affects Versions: 2.4.0
Environment: The environment does not matter with this issue. But I
use Windows 8 64bits, Open JDK 7.0.
Reporter: BoYang
Priority: Critical
Fix For: 2.4.0
Node manager failed to launch container when I set fs.file.impl to
org.apache.hadoop.fs.RawLocalFileSystem in core-site.xml.
The log is:
WARN ContainersLauncher #11
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch
- Failed to launch container.
java.lang.ClassCastException: org.apache.hadoop.fs.RawLocalFileSystem cannot be
cast to org.apache.hadoop.fs.LocalFileSystem
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:270)
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLogPathForWrite(LocalDirsHandlerService.java:307)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:185)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
The issue is in
hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\fs\LocalDirAllocator.java.
It invokes FileSystem.getLocal(), which tries to convert the FileSystem to
LocalFileSystem and will fail when FileSystem object is RawLocalFileSystem
(RawLocalFileSystem is not a sub class of LocalFileSystem).
public static LocalFileSystem getLocal(Configuration conf)
throws IOException {
return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
}
The fix for LocalDirAllocator.java seems to be invoking LocalFileSystem.get()
instead of LocalFileSystem.getLocal()?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
