[
https://issues.apache.org/jira/browse/FLINK-41251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41251:
-----------------------------------
Labels: pull-request-available (was: )
> Memory configuration is not validated when the spec sets JobManager or
> TaskManager memory
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-41251
> URL: https://issues.apache.org/jira/browse/FLINK-41251
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.13.0, kubernetes-operator-1.14.0,
> kubernetes-operator-1.15.0, kubernetes-operator-1.16.0,
> kubernetes-operator-1.16.1
> Reporter: Utkarsh Vashishtha
> Priority: Major
> Labels: pull-request-available
>
> Since FLINK-30411, {{DefaultValidator#validateTmSpec}} and
> {{DefaultValidator#validateJmSpec}} check the memory configuration with
> {{TaskExecutorProcessUtils#processSpecFromConfig}} and
> {{JobManagerProcessUtils}} only when the spec does not set memory through
> {{resource.memory}} or {{resources.requests.memory}}. When it does, the
> memory options in {{flinkConfiguration}} are never checked.
> An inconsistent memory configuration therefore passes validation, and the
> operator finds out only when it deploys the new cluster, after it has already
> shut down the running one. The deployment is then left without a JobManager
> until the spec is fixed.
> *Repro:* a running FlinkDeployment that sets
> {{spec.taskManager.resources.requests.memory}} (or the deprecated
> {{resource.memory}}), updated with {{taskmanager.memory.network.max: 32m}},
> which is below the default {{taskmanager.memory.network.min}} of 64m. The
> validator accepts the change and the operator shuts down the running cluster,
> then fails to deploy the new one with:
> {noformat}
> IllegalConfigurationException: TaskManager memory configuration failed:
> Inconsistently configured Key: 'taskmanager.memory.network.fraction' ,
> default: 0.1 ... min value must be less or equal to max value
> {noformat}
> When the validation does run, i.e. when the spec sets no memory, a failure is
> reported as "TaskManager resource memory must be defined using ..." whatever
> the actual cause, so a conflict like this one is reported as missing memory.
> *Fix:* when the spec sets memory, apply it to the configuration as
> {{FlinkConfigBuilder}} does ({{taskmanager.memory.process.size}} and
> {{jobmanager.memory.process.size}}) and validate the result, and report the
> exception's message instead of the generic one. We hit this in our production
> environment, PR to commit to upstream will follow.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
