[
https://issues.apache.org/jira/browse/FLINK-30530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653159#comment-17653159
]
Arseniy Tashoyan edited comment on FLINK-30530 at 12/30/22 4:43 PM:
--------------------------------------------------------------------
Hi [~gyfora]
The YAML descriptor contains settings, that can be common for all applications.
For example, the *podTemplate* can be identical for all. Then, *serviceAccount*
- same for all applications running in this Kubernetes cluster. {*}image{*},
*flinkVersion* and even *jarURI* - can be same for a number of applications.
Flink settings are more specific to a particular application. Flink memory
settings is a matter of fine tuning per application. Same for checkpointing
settings. Same for parallelism and many other settings.
For me it would be fine to have separate user-provided ConfigMaps for Flink and
Log4J settings: *flinkConfigMap* and {*}log4jConfigMap{*}. For example, we
could have two possible values for {*}log4jConfigMap{*}: console logging and
file logging. It would be better, than to copy-paste the same logging
configuration across many YAML descriptors.
Finally, it is more convenient to group related settings in respective
configuration files (in Kubernetes - ConfigMaps). That's why people have
separate flink-conf.yaml, log4j.properties, etc.
was (Author: tashoyan):
Hi [~gyfora]
The YAML descriptor contains settings, that can be common for all applications.
For example, the *podTemplate* can be identical for all. Then, *serviceAccount*
- same for all applications running in this Kubernetes cluster. {*}image{*},
*flinkVersion* and even *jarURI* - can be same for a number of applications.
Flink settings are more specific to a particular application. Flink memory
settings is a matter of fine tuning per application. Same for checkpointing
settings. Same for parallelism and many other settings.
For me it would be fine to have separate user-provided ConfigMaps for Flink and
Log4J settings: *flinkConfigMap* and {*}log4jConfigMap{*}. For example, we
could have two possible values for {*}log4jConfigMap{*}: console logging and
file logging. It would be better, than to copy-paste the same logging
configuration across many YAML descriptors.
Finally, it is more convenient to group related settings in respective
configuration files. That's why people have separate flink-conf.yaml,
log4j.properties, etc.
> Flink configuration from user-provided ConfigMap
> ------------------------------------------------
>
> Key: FLINK-30530
> URL: https://issues.apache.org/jira/browse/FLINK-30530
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Environment: Flink 1.15.2
> Flink Kubernetes operator 1.2.0
> Reporter: Arseniy Tashoyan
> Priority: Major
>
> Currently the Flink configuration can be specified in the YAML descriptor of
> FlinkDeployment via the _flinkConfiguration_ setting:
> {code:yaml}
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> ...
> {code}
> Same for the logging configuration:
> {code:yaml}
> logConfiguration:
> "log4j-console.properties": |
> rootLogger.level = DEBUG
> ...{code}
> This makes the YAML descriptor overloaded and huge. In addition, Flink and
> logging configuration may differ for different applications, while the
> Kubernetes settings maybe same for all applications. Therefore it makes sense
> to extract Flink and logging configurations from the YAML descriptor.
> This can be done via a user-provided ConfigMap:
> {code:yaml}
> flinkConfigMap: basic-example-flink-config
> {code}
> In this example we have a Flink application {_}basic-example{_}. The
> _basic-example-flink-config_ ConfigMap contains all config files used by
> Flink: flink-conf.yaml, log4j-console.properties, possibly other files. The
> content of this ConfigMap gets mounted as a volume to {_}/opt/flink/conf{_}.
> Therefore we can have different Flink settings for different applications and
> the same YAML descriptor for all of them (only the value for _flinkConfigMap_
> differs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
