[
https://issues.apache.org/jira/browse/FLINK-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955141#comment-16955141
]
Yang Wang commented on FLINK-9953:
----------------------------------
> Perjob cluster mode
I suggest to build the user image with required dependencies in per job mode.
And actually, standalone job cluster is also like. Many companies has use this
way in production. In order to solve dynamic dependency management, we could
add the init container before jm and tm pod starting. The init container could
download the jars and other files from http server, hdfs and other shared
storage. This make flink application more like k8s style. In this way, the
`MiniDispatcher` and `ClassPathJobGraphRetriever` is enough for the per job
mode.
The two parts submission is more like to start a session cluster to simulate
per job. So we will need to new dispatcher to accept job from rest and allow
only one job. Maybe we could support this in the future, but it need more
discussion.
> Submission cli
Currently the `flink run` coud only support detach mode for per job cluster on
Yarn. In attach mode, we use a session to simulate a per job cluster for
multi-parts. Do we need to keep the same behavior as flink on Yarn? We do not
need user jar in k8s per job mode, so using the `flink run` to start per job
cluster will be strange.
{code:java}
// detach, DeployJobCluster() Use the jar in the image, not in the cli.
./bin/flink run -d -m kubernetes-cluster ./examples/batch/WordCount.jar
// attach, DeploySessionCluster()
./bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar{code}
> Implementation plan
Let's focus on current design and move the production optimization to phase2. I
have created another umbrella jira to track. Also we need more feedback from
other users to improve the active kubernetes integration after phase1.
I will attach the PRs in the next few days.
[~felixzheng] [~trohrmann] How do you think?
> Active Kubernetes integration
> -----------------------------
>
> Key: FLINK-9953
> URL: https://issues.apache.org/jira/browse/FLINK-9953
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
> Reporter: Till Rohrmann
> Assignee: Yang Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is the umbrella issue tracking Flink's active Kubernetes integration.
> Active means in this context that the {{ResourceManager}} can talk to
> Kubernetes to launch new pods similar to Flink's Yarn and Mesos integration.
> Phase1 implementation will have complete functions to make flink running on
> kubernetes. Phrase1 is mainly focused on production optimization, including
> k8s native high-availability, storage, network, log collector and etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
