[
https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257676#comment-13257676
]
Dave Latham commented on HADOOP-7682:
-------------------------------------
Here's my understanding of this issue from digging around a bit in case it's
helpful for others. I could be mistaken on some of the details.
Some time ago, RawLocalFileSystem.setPermission used to use a shell exec
command to fork a process to alter permissions of a file.
Somewhere along the 0.20 branch (I believe in the 0.20.security branch) it was
decided that this was too slow, and instead
java.io.File.set{Readable|Writable|Executable} should be used (see HADOOP-6304
for one such issue. It has a patch with the logic that wound up in FileUtil
though perhaps committed from a different patch). However the java.io.File
methods don't allow one to directly set all the permissions so the code first
has to clear permissions, then build them up again. This resulted in two
problems. First, there is a race condition when the file briefly has no
permissions even for the owner (see MAPREDUCE-2238 for more detail). Second,
Windows doesn't support clearing all permissions on the file (this JIRA).
The first problem was worked around in HADOOP-7110 (HADOOP-7432 backported it
to this branch) by using JNI native code instead. However, if the native code
is not available, then it falls back to the java.io.File methods. So, the
second problem still remains, that FileUtil.setPermission (and thus the
RawLocalFileSystem setPermission) does not work on Windows because Windows does
not have the native code implementation and also fails the java.io.File
fallback.
The issues FKorning ran into in a comment above appear to be wider than this
particular JIRA, though I may have misunderstood what led to his shorn yak.
Windows is listed as a supported platform for Hadoop, and some of our
developers use Windows as a development environment, so it's important for us
that hadoop at least functions on Windows, even if it's not as performant as
our production clusters on Linux. I noted that this is currently the highest
voted open JIRA for hadoop.
In order for it to function on Windows, I added a final fallback in
FileUtil.setPermission to revert to the older behavior of using a shell exec
fork for setPermission when the other methods fail. Perhaps it will be helpful
for others:
{code}
public static void setPermission(File f, FsPermission permission
) throws IOException {
FsAction user = permission.getUserAction();
FsAction group = permission.getGroupAction();
FsAction other = permission.getOtherAction();
// use the native/fork if the group/other permissions are different
// or if the native is available
if (group != other || NativeIO.isAvailable()) {
execSetPermission(f, permission);
return;
}
try
{
boolean rv = true;
// read perms
rv = f.setReadable(group.implies(FsAction.READ), false);
checkReturnValue(rv, f, permission);
if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) {
f.setReadable(user.implies(FsAction.READ), true);
checkReturnValue(rv, f, permission);
}
// write perms
rv = f.setWritable(group.implies(FsAction.WRITE), false);
checkReturnValue(rv, f, permission);
if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) {
f.setWritable(user.implies(FsAction.WRITE), true);
checkReturnValue(rv, f, permission);
}
// exec perms
rv = f.setExecutable(group.implies(FsAction.EXECUTE), false);
checkReturnValue(rv, f, permission);
if (group.implies(FsAction.EXECUTE) !=
user.implies(FsAction.EXECUTE)) {
f.setExecutable(user.implies(FsAction.EXECUTE), true);
checkReturnValue(rv, f, permission);
}
}
catch (IOException ioe)
{
LOG.warn("Java file permissions failed to set " + f + " to " +
permission + " falling back to fork");
execSetPermission(f, permission);
}
}
{code}
> taskTracker could not start because "Failed to set permissions" to "ttprivate
> to 0700"
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-7682
> URL: https://issues.apache.org/jira/browse/HADOOP-7682
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0
> Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1,
> jdk1.6.0_05
> Reporter: Magic Xie
>
> ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because
> java.io.IOException:Failed to set permissions of
> path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700
> at
> org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
> at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635)
> at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430)
> Since hadoop0.20.203 when the TaskTracker initialize, it checks the
> permission(TaskTracker Line 624) of
> (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR,
>
> org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup)
> call setPermission(Line 481) to deal with it, setPermission works fine on
> *nx, however,it dose not alway works on windows.
> setPermission call setReadable of Java.io.File in the line 498, but according
> to the Table1 below provided by oracle,setReadable(false) will always return
> false on windows, the same as setExecutable(false).
> http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/
> is it cause the task tracker "Failed to set permissions" to "ttprivate to
> 0700"?
> Hadoop 0.20.202 works fine in the same environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
