[
https://issues.apache.org/jira/browse/FLINK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254117#comment-17254117
]
Till Rohrmann commented on FLINK-20681:
---------------------------------------
What happens if the user specifies a non-hdfs remote location (e.g.
s3://foobar/barfoo)? What would be the expected behaviour?
Even though it is not ideal, you could still put the user jars and the
third-party jars into Flink's {{lib}} directory (given that there are no
dependency conflicts).
I think we are already supporting to register remote files with the Yarn
application partially since {{YarnApplicationFileUploader}} contains some code
for it. Would then the only change necessary be changing
{{YarnClusterDescriptor.shipFiles}} from {{List<File>}} to {{List<Path>}}?
> Support specifying the hdfs path when ship archives or files
> -------------------------------------------------------------
>
> Key: FLINK-20681
> URL: https://issues.apache.org/jira/browse/FLINK-20681
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.12.0
> Reporter: Ruguo Yu
> Priority: Major
> Labels: pull-requests-available
> Fix For: 1.13.0
>
> Attachments: image-2020-12-23-20-58-41-234.png
>
>
> Currently, our team try to submit flink job that depends extra resource with
> yarn-application target, and use two options: "yarn.ship-archives" and
> "yarn.ship-files".
> But above options only support specifying local resource and shiping them to
> hdfs, besides if it can support remote resource on distributed filesystem
> (such as hdfs), then get the following benefits：
> * client will exclude the local resource uploading to accelerate the job
> submission process
> * yarn will cache them on the nodes so that they doesn't need to be
> downloaded for application
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
