[ https://issues.apache.org/jira/browse/YARN-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065823#comment-16065823 ]
Bibin A Chundatt commented on YARN-6708: ---------------------------------------- Thank you [~jlowe] for review. {quote} Is there a reason to use 755 permissions on the intermediate directories in the user cache? Note that we only allow 710 permissions on the final directly, and it seems intermediate directories should only require that as well, or 750 at the most. I don't see any reason to allow any "other" permissions on user-specific directories in the local cache. {quote} {{755}} is the existing directory permissions for cache folders in {{FsDownload#cacheperms}}. If Node manager service and users are in different group should be able to check the availability of existing cache folders during download and recovery. {{LocalResourcesTrackerImpl#handle}} {code} case REQUEST: if (rsrc != null && (!isResourcePresent(rsrc))) { LOG.info("Resource " + rsrc.getLocalPath() + " is missing, localizing it again"); removeResource(req); rsrc = null; } {code} {quote} It would improve readability if we moved the directory handling stuff to a utility method like createDirAndParents or something similar and pass the desired permissions for the dirs as an argument. {quote} Will update in next patch. {code} vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l total 28 drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./ drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../ drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:15 10/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:22 11/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:27 12/ drwxr-xr-x 3 mapred users 4096 Jun 10 12:31 13/ {code} {quote} If the parent of destDirPath is the cache root then we won't set the permissions of destDirPath but otherwise we will? {quote} Already existing {{FSDownload}} code handles this case. {{FSDownload}} *cacheperms* sets directory permissions as *755*. {{FSDownload}} should have been in {{nodemanager}} since its tightly coupled to the directoy permission wrt to localization . or am I missing something? {quote} AtomicLong use is overkill in the test since there's no thread contention on that object. {quote} Yes .. we dont require will change. In test tried to cover complete flow with multiple base directory, single base directory etc.. On second thought we really don't require. LocalCacheDirectoryManager part we could skip. Creating paths {{12}}, {{1/14}} {{0/0/85}} should be enough for current code change. {{FSDownload}} handles the final cache directory permissions. Even if 0/0/85 is created before download, in FSDownload for {{85}} the same could get reset rt?? The directory permission is 755 and in jenkins the umask is 022 to validate directory rights for code change used reflection. Container localizer USERCACHE permission could be package private but the above point of {{FSDownload}} will set the rights to 0755 or we should be checking only {{0/0}}?? > Nodemanager container crash after ext3 folder limit > --------------------------------------------------- > > Key: YARN-6708 > URL: https://issues.apache.org/jira/browse/YARN-6708 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Bibin A Chundatt > Assignee: Bibin A Chundatt > Priority: Critical > Attachments: YARN-6708.001.patch, YARN-6708.002.patch, > YARN-6708.003.patch, YARN-6708.004.patch > > > Configure umask as *027* for nodemanager service user > and {{yarn.nodemanager.local-cache.max-files-per-directory}} as {{40}}. After > 4 *private* dir localization next directory will be *0/14* > Local Directory cache manager > {code} > vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l > total 28 > drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./ > drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../ > drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/ > drwxr-xr-x 3 mapred users 4096 Jun 10 12:15 10/ > drwxr-xr-x 3 mapred users 4096 Jun 10 12:22 11/ > drwxr-xr-x 3 mapred users 4096 Jun 10 12:27 12/ > drwxr-xr-x 3 mapred users 4096 Jun 10 12:31 13/ > {code} > *drwxr-x---* 3 mapred users 4096 Jun 10 14:36 0/ is only *750* > Nodemanager user will not be able check for localization path exists or not. > {{LocalResourcesTrackerImpl}} > {code} > case REQUEST: > if (rsrc != null && (!isResourcePresent(rsrc))) { > LOG.info("Resource " + rsrc.getLocalPath() > + " is missing, localizing it again"); > removeResource(req); > rsrc = null; > } > if (null == rsrc) { > rsrc = new LocalizedResource(req, dispatcher); > localrsrc.put(req, rsrc); > } > break; > {code} > *isResourcePresent* will always return false and same resource will be > localized to {{0}} to next unique number -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org