[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[ https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246024#comment-17246024 ] Boris Lublinsky commented on FLINK-20324: - Can we move it to 1.12.x? > Support customizing of containers for native kubernetes setup > -- > > Key: FLINK-20324 > URL: https://issues.apache.org/jira/browse/FLINK-20324 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Affects Versions: 1.11.2 > Environment: Kubernetes >Reporter: Boris Lublinsky >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2, 1.13.0 > > Original Estimate: 12h > Remaining Estimate: 12h > > A common requirement for Flink applications is usage of custom resources > (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage > of NFS-based checkpointing, require mounting NFS volumes, access to databases > might require environment variables and secrets, the ability to safely remove > cluster manager deployment might require owner's reference, etc. > All of these resources can be added to created pods using Pod decorators > already defined by Flink. So implementation of such support is as simple as > implementing of new decorators and wiring them correctly to the appropriate > factories. > > An implementation of such support is provided in this pull request > [https://github.com/apache/flink/pull/14005] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[ https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241676#comment-17241676 ] Boris Lublinsky commented on FLINK-20324: - Now added [https://github.com/apache/flink/pull/14284] implementing configuration mounts. Volume mount is moved to FLINK-15649 and owner's reference support to FLINK-20359 > Support customizing of containers for native kubernetes setup > -- > > Key: FLINK-20324 > URL: https://issues.apache.org/jira/browse/FLINK-20324 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Affects Versions: 1.11.2 > Environment: Kubernetes >Reporter: Boris Lublinsky >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Original Estimate: 12h > Remaining Estimate: 12h > > A common requirement for Flink applications is usage of custom resources > (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage > of NFS-based checkpointing, require mounting NFS volumes, access to databases > might require environment variables and secrets, the ability to safely remove > cluster manager deployment might require owner's reference, etc. > All of these resources can be added to created pods using Pod decorators > already defined by Flink. So implementation of such support is as simple as > implementing of new decorators and wiring them correctly to the appropriate > factories. > > An implementation of such support is provided in this pull request > [https://github.com/apache/flink/pull/14005] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[
https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239021#comment-17239021
]
Yang Wang commented on FLINK-20324:
---
[~lublinsky] Thanks for your efforts on the native K8s integration.
# You could find the environment setting in {{InitJobManagerDecorator}} and
{{InitTaskManagerDecorator}}.
# For the secret config options, it makes sense to distinguish them for
JobManager and TaskManager. But I am sure whether it is really necessary. Feel
free to create a new ticket if you come across this problem in you production
environment. And we could make the current config option as default if
specific(jobmanager/taskmanager) config option is not set.
# I am afraid we do not have much progress on the pod template. So I agree
with you that we could have the volume mounting separately. It is a more common
feature that we need when using native K8s integration in production. Let's
keep the volume mounting discussion under FLINK-15649. If we reach an
agreement, I think you could take over that ticket and start the PR working.
> Support customizing of containers for native kubernetes setup
> --
>
> Key: FLINK-20324
> URL: https://issues.apache.org/jira/browse/FLINK-20324
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Affects Versions: 1.11.2
> Environment: Kubernetes
>Reporter: Boris Lublinsky
>Priority: Major
> Fix For: 1.12.0, 1.11.2
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> A common requirement for Flink applications is usage of custom resources
> (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage
> of NFS-based checkpointing, require mounting NFS volumes, access to databases
> might require environment variables and secrets, the ability to safely remove
> cluster manager deployment might require owner's reference, etc.
> All of these resources can be added to created pods using Pod decorators
> already defined by Flink. So implementation of such support is as simple as
> implementing of new decorators and wiring them correctly to the appropriate
> factories.
>
> An implementation of such support is provided in this pull request
> [https://github.com/apache/flink/pull/14005]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[
https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238889#comment-17238889
]
Boris Lublinsky commented on FLINK-20324:
-
Thank you Yang Wang,
Digging a bit deeper in your answers:
Env variables.
I see documentation, but I do not see its implementation in the code. I would
assume those are implemented as decorators, but the only thing that I see there
is an EnvSecretsDecorator, which seems to take these env variables from secrets:
public static final ConfigOption>>
KUBERNETES_ENV_SECRET_KEY_REF =
key("kubernetes.env.secretKeyRef")
.mapType()
.asList()
.noDefaultValue()
.withDescription(
Description.builder()
.text("The user-specified secrets to set env variables in Flink container. The
value should be in " +
"the form of %s.",
TextElement.code("env:FOO_ENV,secret:foo_secret,key:foo_key;env:BAR_ENV,secret:bar_secret,key:bar_key"))
.build());
Which is not exactly that I was proposing. Also it always sets the same
environment on both Job and Task manager. Do we want this or we want to
distinguish between the two?
I could not find a single example, so I am probably missing something here.
Mounting secrets looks good, but not allowing to distinguish between Job and
Task manager. Is this correct?
I also looked at FLINK-15649 and it seems to be postponed. Is there any
progress for it?
Finally a new ticket
[https://issues.apache.org/jira/projects/FLINK/issues/FLINK-20359?filter=reportedbyme]
is created
> Support customizing of containers for native kubernetes setup
> --
>
> Key: FLINK-20324
> URL: https://issues.apache.org/jira/browse/FLINK-20324
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Affects Versions: 1.11.2
> Environment: Kubernetes
>Reporter: Boris Lublinsky
>Priority: Major
> Fix For: 1.12.0, 1.11.2
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> A common requirement for Flink applications is usage of custom resources
> (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage
> of NFS-based checkpointing, require mounting NFS volumes, access to databases
> might require environment variables and secrets, the ability to safely remove
> cluster manager deployment might require owner's reference, etc.
> All of these resources can be added to created pods using Pod decorators
> already defined by Flink. So implementation of such support is as simple as
> implementing of new decorators and wiring them correctly to the appropriate
> factories.
>
> An implementation of such support is provided in this pull request
> [https://github.com/apache/flink/pull/14005]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[ https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238542#comment-17238542 ] Yang Wang commented on FLINK-20324: --- [~lublinsky] Since release-1.12 is feature feature frozen, I am afraid we could not get this done in 1.12. For the owner reference, I suggest to create a new ticket for it. And then this ticket will be focused on volume mounting, similar to FLINK-15649. > Support customizing of containers for native kubernetes setup > -- > > Key: FLINK-20324 > URL: https://issues.apache.org/jira/browse/FLINK-20324 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Affects Versions: 1.11.2 > Environment: Kubernetes >Reporter: Boris Lublinsky >Priority: Major > Fix For: 1.12.0, 1.11.2 > > Original Estimate: 12h > Remaining Estimate: 12h > > A common requirement for Flink applications is usage of custom resources > (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage > of NFS-based checkpointing, require mounting NFS volumes, access to databases > might require environment variables and secrets, the ability to safely remove > cluster manager deployment might require owner's reference, etc. > All of these resources can be added to created pods using Pod decorators > already defined by Flink. So implementation of such support is as simple as > implementing of new decorators and wiring them correctly to the appropriate > factories. > > An implementation of such support is provided in this pull request > [https://github.com/apache/flink/pull/14005] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[ https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238489#comment-17238489 ] Boris Lublinsky commented on FLINK-20324: - Thank you Yang Wang Env variables is new in 1.12, I did not notice it - my bad Secrets mounting is also new in 1.12 and is exactly the same as I have done Volume/Config mount is quite similar to what I have implemented and I can easily extend it to do the same Owner reference is still missing. We are targeting 1.12 for this, so lets discuss what we can add easily and I can rework PR quickly > Support customizing of containers for native kubernetes setup > -- > > Key: FLINK-20324 > URL: https://issues.apache.org/jira/browse/FLINK-20324 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Affects Versions: 1.11.2 > Environment: Kubernetes >Reporter: Boris Lublinsky >Priority: Major > Fix For: 1.12.0, 1.11.2 > > Original Estimate: 12h > Remaining Estimate: 12h > > A common requirement for Flink applications is usage of custom resources > (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage > of NFS-based checkpointing, require mounting NFS volumes, access to databases > might require environment variables and secrets, the ability to safely remove > cluster manager deployment might require owner's reference, etc. > All of these resources can be added to created pods using Pod decorators > already defined by Flink. So implementation of such support is as simple as > implementing of new decorators and wiring them correctly to the appropriate > factories. > > An implementation of such support is provided in this pull request > [https://github.com/apache/flink/pull/14005] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20324) Support customizing of containers for native kubernetes setup
[ https://issues.apache.org/jira/browse/FLINK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238477#comment-17238477 ] Yang Wang commented on FLINK-20324: --- [~lublinsky] Thanks for creating this ticket. Even though we already have a plan to support the advanced feature via pod template, I agree with you that we may need to make some common feature having separate Flink config options. It is more convenient for our users. And these features are very important when we want to put native K8s integration in production. Let't try to sort out these features and reach a consensus before getting back to the PR. * Environment for JobManager/TaskManager ** We already could support this. You could find the config options here[1]. They are same as Yarn/Mesos deployment. * Secrets mount ** We already have this ability in 1.12[2]. Right? * PVC, ConfigMap mount ** I like the spark style for volume mount[3]. Then we could support PVC, ConfigMap, hostpath, etc. It is more flexible. WDYT? [1]. [https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#forwarding-environment-variables] [2]. [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#using-secrets] [3]. [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > Support customizing of containers for native kubernetes setup > -- > > Key: FLINK-20324 > URL: https://issues.apache.org/jira/browse/FLINK-20324 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Affects Versions: 1.11.2 > Environment: Kubernetes >Reporter: Boris Lublinsky >Priority: Major > Fix For: 1.12.0, 1.11.2 > > Original Estimate: 12h > Remaining Estimate: 12h > > A common requirement for Flink applications is usage of custom resources > (Environment variables, PVCs, Secrets, configMaps, etc). For example, usage > of NFS-based checkpointing, require mounting NFS volumes, access to databases > might require environment variables and secrets, the ability to safely remove > cluster manager deployment might require owner's reference, etc. > All of these resources can be added to created pods using Pod decorators > already defined by Flink. So implementation of such support is as simple as > implementing of new decorators and wiring them correctly to the appropriate > factories. > > An implementation of such support is provided in this pull request > [https://github.com/apache/flink/pull/14005] -- This message was sent by Atlassian Jira (v8.3.4#803005)
