[jira] [Comment Edited] (FLINK-14360) Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
[
https://issues.apache.org/jira/browse/FLINK-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949883#comment-16949883
]
Shen Yinjie edited comment on FLINK-14360 at 10/12/19 3:09 AM:
---
Thanks [~azagrebin] for comments!
1)Current "paths" only contains some paths as ${fs.defaultFS}/user/ user.name
/.flink /appID /xxx according to client's fs.defaultFS setting.
2)For example, when we deploy flink on a Hadoop cluster with YARN
federation(two YARN subclusters have separate HDFSs, one HDFS enable RBF and
another enable HA). Flink should access RBF's ns1, ns2 and router-fed ns,
sometimes another hdfs will be accessed.Since YARN router will forward
client's flink job to a random YARN subcluster by some polices.
3)So, we add a new config in flink-conf.yaml for users to set neccessary HDFS
schemes to obtain tokens.
Not sure am I clear :)
was (Author: shenyinjie):
Thanks [~azagrebin] for comments!
1)Current "paths" only contains some paths as
${fs.defaultFS}/user/{user.name}/.flink/appID/xxx
according to client's fs.defaultFS setting.
2)For example, when we deploy flink on a Hadoop cluster with YARN
federation(two YARN subclusters have separate HDFSs, one HDFS enable RBF and
another enable HA). Flink should access RBF's ns1, ns2 and router-fed ns,
sometimes another hdfs will be accessed.Since YARN router will forward
client's flink job to a random YARN subcluster by some polices.
3)So, we add a new config in flink-conf.yaml for users to set neccessary HDFS
schemes to obtain tokens.
Not sure am I clear :)
> Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
> --
>
> Key: FLINK-14360
> URL: https://issues.apache.org/jira/browse/FLINK-14360
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Shen Yinjie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There's a scenario when deploy flink on yarn with multi hdfs cluster or hdfs
> federation, Flink need to get delegation tokens of all the namespaces before
> start appmaster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14360) Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
[
https://issues.apache.org/jira/browse/FLINK-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949883#comment-16949883
]
Shen Yinjie edited comment on FLINK-14360 at 10/12/19 3:08 AM:
---
Thanks [~azagrebin] for comments!
1)Current "paths" only contains some paths as
${fs.defaultFS}/user/{user.name}/.flink/appID/xxx
according to client's fs.defaultFS setting.
2)For example, when we deploy flink on a Hadoop cluster with YARN
federation(two YARN subclusters have separate HDFSs, one HDFS enable RBF and
another enable HA). Flink should access RBF's ns1, ns2 and router-fed ns,
sometimes another hdfs will be accessed.Since YARN router will forward
client's flink job to a random YARN subcluster by some polices.
3)So, we add a new config in flink-conf.yaml for users to set neccessary HDFS
schemes to obtain tokens.
Not sure am I clear :)
was (Author: shenyinjie):
Thanks [~azagrebin] for comments!
1)Current "paths" only contains some paths as
${fs.defaultFS}/user/{user.name}/.flink/appID/xxx according to client's
fs.defaultFS setting.
2)For example, when we deploy flink on a Hadoop cluster with YARN
federation(two YARN subclusters have separate HDFSs, one HDFS enable RBF and
another enable HA). Flink should access RBF's ns1, ns2 and router-fed ns,
sometimes another hdfs will be accessed.Since YARN router will forward
client's flink job to a random YARN subcluster by some polices.
3)So, we add a new config in flink-conf.yaml for users to set neccessary HDFS
schemes to obtain tokens.
Not sure am I clear :)
> Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
> --
>
> Key: FLINK-14360
> URL: https://issues.apache.org/jira/browse/FLINK-14360
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Shen Yinjie
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There's a scenario when deploy flink on yarn with multi hdfs cluster or hdfs
> federation, Flink need to get delegation tokens of all the namespaces before
> start appmaster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14360) Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
[ https://issues.apache.org/jira/browse/FLINK-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949216#comment-16949216 ] Yangze Guo edited comment on FLINK-14360 at 10/11/19 7:40 AM: -- cc [~fly_in_gis] Could you take a look at this ticket? was (Author: karmagyz): cc [~wangyang] Could you take a look at this ticket? > Flink on yarn should support obtain delegation tokens of multi hdfs namespaces > -- > > Key: FLINK-14360 > URL: https://issues.apache.org/jira/browse/FLINK-14360 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Shen Yinjie >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There's a scenario when deploy flink on yarn with multi hdfs cluster or hdfs > federation, Flink need to get delegation tokens of all the namespaces before > start appmaster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-14360) Flink on yarn should support obtain delegation tokens of multi hdfs namespaces
[ https://issues.apache.org/jira/browse/FLINK-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949216#comment-16949216 ] Yangze Guo edited comment on FLINK-14360 at 10/11/19 7:37 AM: -- cc [~wangyang] Could you take a look at this ticket? was (Author: karmagyz): cc [~taoyang] [~yangwang166] Could you take a look at this ticket? > Flink on yarn should support obtain delegation tokens of multi hdfs namespaces > -- > > Key: FLINK-14360 > URL: https://issues.apache.org/jira/browse/FLINK-14360 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Shen Yinjie >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There's a scenario when deploy flink on yarn with multi hdfs cluster or hdfs > federation, Flink need to get delegation tokens of all the namespaces before > start appmaster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
