[
https://issues.apache.org/jira/browse/TEZ-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated TEZ-4185:
-------------------------------
Description:
Before updating file permissions TEZ check if the permission change is needed
with the following conditional:
{code:java}
if
(!SPILL_FILE_PERMS.equals(SPILL_FILE_PERMS.applyUMask(FsPermission.getUMask(conf))))
{
rfs.setPermission(filename, SPILL_FILE_PERMS);
} {code}
The problem is that the config object used by this conditional can be different
than the config used by the file system (rfs). The reason for this is the
underlaying file system instance cache in hadoop.
Either we should remove this conditional altogether (it's a local file system,
it won't generate NN calls) or modify it to use rfs.getConf().
was:
Before updating file permissions TEZ check if the permission change is needed
with the following conditional:
{code:java}
if
(!SPILL_FILE_PERMS.equals(SPILL_FILE_PERMS.applyUMask(FsPermission.getUMask(conf))))
{
rfs.setPermission(filename, SPILL_FILE_PERMS);
} {code}
If the config object is changing in the background then setPermission() call
will be skipped.
The rfs file system is always a local file system so there is no need to do
this check beforehand (setPermission doesn't generate an additional NameNode
call).
{code:java}
rfs = ((LocalFileSystem)FileSystem.getLocal(this.conf)).getRaw(); {code}
We can remove the if statement and do the permission change unconditionally.
> Tez may skip file permission update on intermediate output
> ----------------------------------------------------------
>
> Key: TEZ-4185
> URL: https://issues.apache.org/jira/browse/TEZ-4185
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.2, 0.10.1, 0.9.3
> Reporter: Attila Magyar
> Assignee: Attila Magyar
> Priority: Major
> Attachments: TEZ-4185.1.patch
>
>
> Before updating file permissions TEZ check if the permission change is needed
> with the following conditional:
> {code:java}
> if
> (!SPILL_FILE_PERMS.equals(SPILL_FILE_PERMS.applyUMask(FsPermission.getUMask(conf))))
> {
> rfs.setPermission(filename, SPILL_FILE_PERMS);
> } {code}
> The problem is that the config object used by this conditional can be
> different than the config used by the file system (rfs). The reason for this
> is the underlaying file system instance cache in hadoop.
> Either we should remove this conditional altogether (it's a local file
> system, it won't generate NN calls) or modify it to use rfs.getConf().
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
