[
https://issues.apache.org/jira/browse/FLINK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076877#comment-17076877
]
Yang Wang commented on FLINK-12160:
-----------------------------------
[~felixzheng] [~trohrmann]
Thanks for your attention about this ticket. This is an old ticket and are not
applicable now. Last week, i have updated the description of FLINK-14460 and
try to conclude what should be done by Flink config options or pod template.
This ticket could be covered by user specified volume mount via pod template.
Also i have talked to our platform developer and users, this is acceptable for
them. I will close this ticket as "Won't fix".
Since there is no a default DFS in Kubernetes cluster, i am just thinking
whether shipping the user jars, dependencies, connector jars is the best
choice. For example, if the users are using S3 or Aliyun OSS for the persistent
storage, then the Flink client will need to upload these jars and download them
before the jobmanager launched(maybe in the init container).
Actually, we have another choice which is to use Flink distributed cache. These
jars are uploaded to jobmanager via rest client, just like the session mode.
Maybe we could introduce this behavior to the K8s perjob, which i will call it
client mode for perjob. The opposite of that is cluster mode for perjob, also
known as "application mode". Our blink internal implementation is similar to
this way.
> Support to ship user config files on kubernetes
> -----------------------------------------------
>
> Key: FLINK-12160
> URL: https://issues.apache.org/jira/browse/FLINK-12160
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
>
> When deploying flink cluster on Yarn, `yarnShip` cli option could be used to
> ship user config files and jars. The files are registered as Yarn local
> resource which is saved on hdfs.
>
> However, we do not have a default existing dfs. The `ConfigMap` could be used
> to ship small files. The files will be stored in etcd and then mounted to
> jobmanager and taskmanager pod so that they could use it as locally.
> {code:java}
> -kt,--kubernetesship <arg> Ship files in the specified
> directory
> (t for transfer). Only small
> files(<1MB) could be supported.
> {code}
> [https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
