[
https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076100#comment-17076100
]
Canbin Zheng commented on FLINK-12160:
--------------------------------------
Hi [~fly_in_gis]! Thanks for filing this issue, recently I would like to add
kind of such support too; but for the functionality of this parameter, I tend
to think different:
* It's confusing that the {{-kt}} in K8s setup only supports config files
whose size is smaller than 1MB while the {{-yt}} in YARN setup supports both
config files and user jars without 1MB limitation. I think it would be better
to keep consistency between the YARN and the K8s setup. Moreover, it would be
better to reuse the DFS instead of the ConfigMap to ship files.
* There are still many scenarios that people need to ship user jars, just like
what has been supported in the YARN setup. Take our streaming computing
platform for example, we provide several thin images baked with very basic
Flink distribution for the sake of downloading efficiency, then dynamically
ship the user dependencies(such as user jars, connector jars, etc.) when
deploying a new Flink cluster.
cc [~tison] [~trohrmann] WDYT?
> Support to ship user config files on kubernetes
> -----------------------------------------------
>
> Key: FLINK-12160
> URL: https://issues.apache.org/jira/browse/FLINK-12160
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
>
> When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
> ship user config files and jars. The files are registered as Yarn local
> resource which is saved on hdfs.
>
> However, we do not have a default existing dfs. The `ConfigMap` could be used
> to ship small files. The files will be stored in etcd and then mounted to
> jobmanager and taskmanager pod so that they could use it as locally.
> {code:java}
> -kt,--kubernetesship <arg> Ship files in the specified
> directory
> (t for transfer). Only small
> files(<1MB) could be supported.
> {code}
> [https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
