madhumita chakraborty created YARN-3962:
-------------------------------------------

             Summary: If we change node manager identity to run as virtual 
account, then resource localization service fails to start with incorrect 
permission
                 Key: YARN-3962
                 URL: https://issues.apache.org/jira/browse/YARN-3962
             Project: Hadoop YARN
          Issue Type: Bug
          Components: nodemanager
    Affects Versions: 2.7.1
            Reporter: madhumita chakraborty


For azure hdinsight we need to change node manager to run as virtual account 
instead of use account. Else after azure reimage, it wont be able to access the 
map output data of the running job in that node. But when we changed the 
nodemanager to run as virtual account we got this error, 
 2015-06-02 06:11:45,281 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Writing credentials to the nmPrivate file 
c:/apps1/temp/hdfs/nm-local-dir/nmPrivate/container_1433128260970_0007_01_000001.tokens.
 Credentials list: 
 2015-06-02 06:11:45,313 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Permissions incorrectly set for dir c:/apps1/temp/hdfs/nm-local-dir/usercache, 
should be rwxr-xr-x, actual value = rwxrwxr-x
 2015-06-02 06:11:45,313 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Attempting to initialize c:/apps1/temp/hdfs/nm-local-dir
 2015-06-02 06:11:45,375 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Permissions incorrectly set for dir c:/apps1/temp/hdfs/nm-local-dir/usercache, 
should be rwxr-xr-x, actual value = rwxrwxr-x
 2015-06-02 06:11:45,375 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Failed to setup local dir c:/apps1/temp/hdfs/nm-local-dir, which was marked as 
good.
 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions 
incorrectly set for dir c:/apps1/temp/hdfs/nm-local-dir/usercache, should be 
rwxr-xr-x, actual value = rwxrwxr-x
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1400)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1367)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$900(ResourceLocalizationService.java:137)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1085)
 2015-06-02 06:11:45,375 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Localizer failed
 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local 
dir c:/apps1/temp/hdfs/nm-local-dir, which was marked as good.
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1372)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$900(ResourceLocalizationService.java:137)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1085)
 Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions 
incorrectly set for dir c:/apps1/temp/hdfs/nm-local-dir/usercache, should be 
rwxr-xr-x, actual value = rwxrwxr-x
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1400)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1367)
Fix - When node manager runs as virtual account, the resourcelocalization 
service fails to come. It checks for the permission of usercache and file cache 
to be 755 and nmPrivate to be 700. But in windows, for virtual account, the 
owner and group is same. So this pemrission check fails. So added a check that 
is user is equal to group, then umask validation dont hold



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to