[
https://issues.apache.org/jira/browse/YARN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661004#comment-14661004
]
Chris Nauroth commented on YARN-3962:
-------------------------------------
This looks good to me. I agree with Xuan that it would be good to find a way
to add unit tests. Thank you, Madhumita!
> If we change node manager identity to run as virtual account, then resource
> localization service fails to start with incorrect permission
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-3962
> URL: https://issues.apache.org/jira/browse/YARN-3962
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.6.0
> Reporter: madhumita chakraborty
> Attachments: YARN-3962-002.patch, Yarn-3962.001.patch
>
>
> For azure hdinsight we need to change node manager to run as virtual account
> instead of use account. Else after azure reimage, it wont be able to access
> the map output data of the running job in that node. But when we changed the
> nodemanager to run as virtual account we got this error,
> 2015-06-02 06:11:45,281 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Writing credentials to the nmPrivate file
> c:/apps1/temp/hdfs/nm-local-dir/nmPrivate/container_1433128260970_0007_01_000001.tokens.
> Credentials list:
> 2015-06-02 06:11:45,313 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Permissions incorrectly set for dir
> c:/apps1/temp/hdfs/nm-local-dir/usercache, should be rwxr-xr-x, actual value
> = rwxrwxr-x
> 2015-06-02 06:11:45,313 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Attempting to initialize c:/apps1/temp/hdfs/nm-local-dir
> 2015-06-02 06:11:45,375 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Permissions incorrectly set for dir
> c:/apps1/temp/hdfs/nm-local-dir/usercache, should be rwxr-xr-x, actual value
> = rwxrwxr-x
> 2015-06-02 06:11:45,375 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Failed to setup local dir c:/apps1/temp/hdfs/nm-local-dir, which was marked
> as good.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions
> incorrectly set for dir c:/apps1/temp/hdfs/nm-local-dir/usercache, should be
> rwxr-xr-x, actual value = rwxrwxr-x
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1400)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1367)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$900(ResourceLocalizationService.java:137)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1085)
> 2015-06-02 06:11:45,375 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup
> local dir c:/apps1/temp/hdfs/nm-local-dir, which was marked as good.
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1372)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$900(ResourceLocalizationService.java:137)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1085)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> Permissions incorrectly set for dir
> c:/apps1/temp/hdfs/nm-local-dir/usercache, should be rwxr-xr-x, actual value
> = rwxrwxr-x
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1400)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1367)
> Fix - When node manager runs as virtual account, the resourcelocalization
> service fails to come. It checks for the permission of usercache and file
> cache to be 755 and nmPrivate to be 700. But in windows, for virtual account,
> the owner and group is same. So this pemrission check fails. So added a check
> that is user is equal to group, then umask validation dont hold
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
