Utkarsh Vashishtha created FLINK-41251:
------------------------------------------
Summary: Memory configuration is not validated when the spec sets
JobManager or TaskManager memory
Key: FLINK-41251
URL: https://issues.apache.org/jira/browse/FLINK-41251
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.16.1, kubernetes-operator-1.16.0,
kubernetes-operator-1.15.0, kubernetes-operator-1.14.0,
kubernetes-operator-1.13.0
Reporter: Utkarsh Vashishtha
Since FLINK-30411, {{DefaultValidator#validateTmSpec}} and
{{DefaultValidator#validateJmSpec}} check the memory configuration with
{{TaskExecutorProcessUtils#processSpecFromConfig}} and
{{JobManagerProcessUtils}} only when the spec does not set memory through
{{resource.memory}} or {{resources.requests.memory}}. When it does, the memory
options in {{flinkConfiguration}} are never checked.
An inconsistent memory configuration therefore passes validation, and the
operator finds out only when it deploys the new cluster, after it has already
shut down the running one. The deployment is then left without a JobManager
until the spec is fixed.
*Repro:* a running FlinkDeployment that sets
{{spec.taskManager.resources.requests.memory}} (or the deprecated
{{resource.memory}}), updated with {{taskmanager.memory.network.max: 32m}},
which is below the default {{taskmanager.memory.network.min}} of 64m. The
validator accepts the change and the operator shuts down the running cluster,
then fails to deploy the new one with:
{noformat}
IllegalConfigurationException: TaskManager memory configuration failed:
Inconsistently configured Key: 'taskmanager.memory.network.fraction' , default:
0.1 ... min value must be less or equal to max value
{noformat}
When the validation does run, i.e. when the spec sets no memory, a failure is
reported as "TaskManager resource memory must be defined using ..." whatever
the actual cause, so a conflict like this one is reported as missing memory.
*Fix:* when the spec sets memory, apply it to the configuration as
{{FlinkConfigBuilder}} does ({{taskmanager.memory.process.size}} and
{{jobmanager.memory.process.size}}) and validate the result, and report the
exception's message instead of the generic one. We hit this in our production
environment, PR to commit to upstream will follow.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
