[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525827#comment-17525827
]
Thomas Weise commented on FLINK-27309:
--
+1, my comment was regarding Flink deployments, not the operator itself.
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525798#comment-17525798
]
Gyula Fora commented on FLINK-27309:
Config changes should not affect the running deployments until the user makes
an upgrade (thus generating a new config for deployment). This is also
consistent with the current behaviour as I mentioned earlier.
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525796#comment-17525796
]
Gyula Fora commented on FLINK-27309:
Currently if the user changes the operator config configmap and the operator
pod is restarted / reloads the same thing happens. So I would say allowing the
operator to pick the config changes up dynamically (if configured) will not be
a significant change to the current behavior it only makes it more explicit.
(and actually allows the certain use-cases where this is useful)
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525790#comment-17525790
]
Thomas Weise commented on FLINK-27309:
--
I think most users would not expect changes to the default Flink config to
affect running deployments and not welcome the service interruption. I think a
running Flink deployment should only be affected by a CR change. There is also
the consideration of lingering default changes leading to a deployment failure
on the next restart, which are harder to correlate.
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525012#comment-17525012
]
Biao Geng commented on FLINK-27309:
---
Thanks [~gyfora]! Besides [~aitozi]'s valuable point, I also think about if the
changed default configs should take effect for a running job deployment.
>From k8s side, it may make sense that such change should be considered as
>`specChanged`, and all the running job will be suspended and restarted.
I am not sure it is proper to restart running jobs(i.e. changed default configs
affect running jobs) from Flink users' side.
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524993#comment-17524993
]
Gyula Fora commented on FLINK-27309:
I will give this a shot :)
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Assignee: Gyula Fora
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524250#comment-17524250
]
Aitozi commented on FLINK-27309:
I think it's a very useful feature in production. But do we need to outline
what config can dynamically take effect. Currently, most option are controlling
the reconciler interval or check behavior can take effect at the next reconcile
turn. But still with option like {{operator.reconciler.max.parallelism}} which
may can't take effect directly, Right ?
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-27309) Allow to load default flink configs in the k8s operator dynamically
[
https://issues.apache.org/jira/browse/FLINK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524240#comment-17524240
]
Gyula Fora commented on FLINK-27309:
+1
I am currently working on FLINK-27023 which will make this simpler. I will open
a PR tomorrow and hopefully we can base this work on top of that.
I think the dynamic config should simply rely on watching the configmap
containing the default configs for any changes and propagating that to the
relevant operator parts.
> Allow to load default flink configs in the k8s operator dynamically
> ---
>
> Key: FLINK-27309
> URL: https://issues.apache.org/jira/browse/FLINK-27309
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> Current default configs used by the k8s operator will be saved in the
> /opt/flink/conf dir in the k8s operator pod and will be loaded only once when
> the operator is created.
> Since the flink k8s operator could be a long running service and users may
> want to modify the default configs(e.g the metric reporter sampling interval)
> for newly created deployments, it may better to load the default configs
> dynamically(i.e. parsing the latest /opt/flink/conf/flink-conf.yaml) in the
> {{ReconcilerFactory}} and {{ObserverFactory}}, instead of redeploying the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
