[
https://issues.apache.org/jira/browse/YARN-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065823#comment-16065823
]
Bibin A Chundatt commented on YARN-6708:
----------------------------------------
Thank you [~jlowe] for review.
{quote}
Is there a reason to use 755 permissions on the intermediate directories in the
user cache? Note that we only allow 710 permissions on the final directly, and
it seems intermediate directories should only require that as well, or 750 at
the most. I don't see any reason to allow any "other" permissions on
user-specific directories in the local cache.
{quote}
{{755}} is the existing directory permissions for cache folders in
{{FsDownload#cacheperms}}. If Node manager service and users are in different
group should be able to check the availability of existing cache folders during
download and recovery. {{LocalResourcesTrackerImpl#handle}}
{code}
case REQUEST:
if (rsrc != null && (!isResourcePresent(rsrc))) {
LOG.info("Resource " + rsrc.getLocalPath()
+ " is missing, localizing it again");
removeResource(req);
rsrc = null;
}
{code}
{quote}
It would improve readability if we moved the directory handling stuff to a
utility method like createDirAndParents or something similar and pass the
desired permissions for the dirs as an argument.
{quote}
Will update in next patch.
{code}
vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l
total 28
drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./
drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../
drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/
drwxr-xr-x 3 mapred users 4096 Jun 10 12:15 10/
drwxr-xr-x 3 mapred users 4096 Jun 10 12:22 11/
drwxr-xr-x 3 mapred users 4096 Jun 10 12:27 12/
drwxr-xr-x 3 mapred users 4096 Jun 10 12:31 13/
{code}
{quote}
If the parent of destDirPath is the cache root then we won't set the
permissions of destDirPath but otherwise we will?
{quote}
Already existing {{FSDownload}} code handles this case. {{FSDownload}}
*cacheperms* sets directory permissions as *755*.
{{FSDownload}} should have been in {{nodemanager}} since its tightly coupled to
the directoy permission wrt to localization . or am I missing something?
{quote}
AtomicLong use is overkill in the test since there's no thread contention on
that object.
{quote}
Yes .. we dont require will change.
In test tried to cover complete flow with multiple base directory, single base
directory etc.. On second thought we really don't require.
LocalCacheDirectoryManager part we could skip. Creating paths {{12}}, {{1/14}}
{{0/0/85}} should be enough for current code change.
{{FSDownload}} handles the final cache directory permissions. Even if 0/0/85
is created before download, in FSDownload for {{85}} the same could get reset
rt??
The directory permission is 755 and in jenkins the umask is 022 to validate
directory rights for code change used reflection. Container localizer USERCACHE
permission could be package private but the above point of {{FSDownload}} will
set the rights to 0755 or we should be checking only {{0/0}}??
> Nodemanager container crash after ext3 folder limit
> ---------------------------------------------------
>
> Key: YARN-6708
> URL: https://issues.apache.org/jira/browse/YARN-6708
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Bibin A Chundatt
> Assignee: Bibin A Chundatt
> Priority: Critical
> Attachments: YARN-6708.001.patch, YARN-6708.002.patch,
> YARN-6708.003.patch, YARN-6708.004.patch
>
>
> Configure umask as *027* for nodemanager service user
> and {{yarn.nodemanager.local-cache.max-files-per-directory}} as {{40}}. After
> 4 *private* dir localization next directory will be *0/14*
> Local Directory cache manager
> {code}
> vm2:/opt/hadoop/release/data/nmlocal/usercache/mapred/filecache # l
> total 28
> drwx--x--- 7 mapred hadoop 4096 Jun 10 14:35 ./
> drwxr-s--- 4 mapred hadoop 4096 Jun 10 12:07 ../
> drwxr-x--- 3 mapred users 4096 Jun 10 14:36 0/
> drwxr-xr-x 3 mapred users 4096 Jun 10 12:15 10/
> drwxr-xr-x 3 mapred users 4096 Jun 10 12:22 11/
> drwxr-xr-x 3 mapred users 4096 Jun 10 12:27 12/
> drwxr-xr-x 3 mapred users 4096 Jun 10 12:31 13/
> {code}
> *drwxr-x---* 3 mapred users 4096 Jun 10 14:36 0/ is only *750*
> Nodemanager user will not be able check for localization path exists or not.
> {{LocalResourcesTrackerImpl}}
> {code}
> case REQUEST:
> if (rsrc != null && (!isResourcePresent(rsrc))) {
> LOG.info("Resource " + rsrc.getLocalPath()
> + " is missing, localizing it again");
> removeResource(req);
> rsrc = null;
> }
> if (null == rsrc) {
> rsrc = new LocalizedResource(req, dispatcher);
> localrsrc.put(req, rsrc);
> }
> break;
> {code}
> *isResourcePresent* will always return false and same resource will be
> localized to {{0}} to next unique number
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
