[
https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482076#comment-16482076
]
Eric Yang commented on YARN-8079:
---------------------------------
[~leftnoteasy] When the files are placed in resources directory, patch 10
implementation prevents mistake to overwrite system level generated files, such
as .token file, and launch_container.sh. However, this design can created
inconvenience for some users because existing Hadoop workload may already be
using the top level localized directory instead of resource directory. We may
not need to worry about launch_container.sh getting overwritten because
container-executor generates the file after static files are localized. Apps
will try to avoid .token files because they can not contact HDFS from
containers, if they overwrites the token files. In summary, it is likely safe
to remove the requirement of "resources" directory from my point of view.
> Support static and archive unmodified local resources in service AM
> -------------------------------------------------------------------
>
> Key: YARN-8079
> URL: https://issues.apache.org/jira/browse/YARN-8079
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Suma Shivaprasad
> Priority: Critical
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8079.001.patch, YARN-8079.002.patch,
> YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch,
> YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch,
> YARN-8079.009.patch, YARN-8079.010.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly
> read srcFile, instead it always construct {{remoteFile}} by using
> componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code}
> To me it is a common use case which services have some files existed in HDFS
> and need to be localized when components get launched. (For example, if we
> want to serve a Tensorflow model, we need to localize Tensorflow model
> (typically not huge, less than GB) to local disk. Otherwise launched docker
> container has to access HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
