[
https://issues.apache.org/jira/browse/FLINK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886342#comment-15886342
]
Haohui Mai edited comment on FLINK-5668 at 2/27/17 7:37 PM:
------------------------------------------------------------
[~rmetzger] -- just want to clarify FLINK-5631 here.
YARN downloads the resources from the specified paths and localizes the
resources on worker nodes. Note that the {{Path}} class in the Hadoop APIs
supports specifying filesystem other than the one specified in {{default.FS}}.
For example, {{new Path(URI.create("s3a://foo")}} specifies the a resource on
S3, regardless what {{default.FS}} is specified. FLINK-5631 enables YARN to
localize resources that are not stored on {{default.FS}}.
As a result I think that the current configuration should be sufficient as
Flink probably recognizes paths that specifies different file systems.
was (Author: wheat9):
[~rmetzger] -- just want to clarify FLINK-5631 here.
YARN downloads the resources from the specified paths and localizes the
resources on worker nodes. Note that the {{Path}} class in the Hadoop APIs
supports specifying filesystem other than the one specified in {{default.FS}}.
For example, {{new Path(URI.create("s3a://foo")}} specifies the a resource on
S3, regardless what {{default.FS}} is specified. FLINK-5631 enables YARN to
localize resources that are not stored on {{default.FS}}.
> Reduce dependency on HDFS at job startup time
> ---------------------------------------------
>
> Key: FLINK-5668
> URL: https://issues.apache.org/jira/browse/FLINK-5668
> Project: Flink
> Issue Type: Improvement
> Components: YARN
> Reporter: Bill Liu
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When create a Flink cluster on Yarn, JobManager depends on HDFS to share
> taskmanager-conf.yaml with TaskManager.
> It's better to share the taskmanager-conf.yaml on JobManager Web server
> instead of HDFS, which could reduce the HDFS dependency at job startup.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
