[
https://issues.apache.org/jira/browse/HADOOP-16235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825575#comment-16825575
]
Steve Loughran commented on HADOOP-16235:
-----------------------------------------
[~iwasakims] the issue here is not whether the real state is exposed or not,
but that the Yarn NM distributed cache will look for posix permissions on
resource to localise, and if the permissions say "world readable" all the way
up the directory tree and !FileStatus.isEncrypted() then the NM will try to
download it to the shared cache.
if your user has submitted work where the credentials to access the store are
in delegation tokens (HADOOP-1606) then the download will fail, as the NM will
be running with the credentials and privileges of the VM, not the user.
This is about lying to the node manager to get it to not cache files.
now, there is a cost here: things like bit spark.tar.gz files won't be cached.
Which is why for ABFS we should think about "maybe there's a way to fix the
Distributed Cache" here to control what's cached across users and what isn't
-as the posix-level checks are obsolete. At the very least, if it can't
download to the cache, it should just leave it to the user localization to sort
it out
> ABFS VersionedFileStatus to declare that it isEncrypted()
> ---------------------------------------------------------
>
> Key: HADOOP-16235
> URL: https://issues.apache.org/jira/browse/HADOOP-16235
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Masatake Iwasaki
> Priority: Minor
> Attachments: HADOOP-16235.001.patch
>
>
> Files in ABFS are always encrypted; have VersionedFileStatus.isEncrypted()
> declare this, presumably just by changing the flag passed to the superclass's
> constructor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
