Shubhankar Sunil Unhale created FLINK-40534:
-----------------------------------------------
Summary: Memory tuning config overrides are computed before the
ScalingExecutorPlugin chain and persisted unchanged when a plugin modifies the
scaling decision
Key: FLINK-40534
URL: https://issues.apache.org/jira/browse/FLINK-40534
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Shubhankar Sunil Unhale
When memory tuning is enabled via {{job.autoscaler.memory.tuning.enabled}} and
a {{ScalingExecutorPlugin}} modifies the scaling decision, which FLIP-575
explicitly supports ("Modified -> use modified decision"), the persisted memory
overrides still reflect the original pre-plugin decision.
In {{ScalingExecutor#scaleResource}}, on main at commit 38a9f197:
# {{computeScalingSummary}} builds the summaries
({{ScalingExecutor.java:172-179}})
# {{MemoryTuning.tuneTaskManagerMemory}} computes {{configOverrides}} from them
({{:192-198}})
# the max-resource/quota check also runs on them ({{:200-207}})
# the plugin chain may modify the summaries ({{:210-214}})
# the modified parallelism is persisted ({{:231-234}}), but
{{storeConfigChanges(context, configOverrides)}} persists the step-2 result
unchanged ({{:236}})
As a result, the applied TM memory sizing, including total process size,
network min/max, managed fraction and metaspace, is computed for a decision
that never deploys. The tuning output depends on the per-vertex decision:
network memory comes from {{computeNewParallelisms}} and heap from
{{MemoryScaling.applyMemoryScaling}}, so changing even one vertex leaves the
whole job-wide TM config mismatched ({{MemoryTuning.java:158-159}},
{{:175-177}}).
A plugin can't correct this itself. The SPI method {{apply(Context,
Map<JobVertexID, ScalingSummary>)}} exposes no {{ConfigChanges}} surface, and
the config reachable from the context is a defensive copy
({{ScalingExecutorPlugin.java:88-89}}, {{ConfigChanges.java:60-66}}). Since
FLINK-39959 a plugin can _see_ the tuned config, but has no write-back path.
A related consequence of the same ordering: since the max-resource/quota check
in step 3 also runs before the chain, a plugin that increases parallelism
bypasses the cluster-resource and quota guards entirely.
Scope: the veto path is safe since it returns before anything is persisted, and
memory tuning is off by default, so hitting this needs tuning enabled plus a
decision-modifying plugin ({{ScalingExecutor.java:215-217}},
{{AutoScalerOptions.java:286-289}}). {{ScalingExecutorPluginTest}} doesn't
cover the tuning interaction, and the plugins documentation page doesn't
mention it.
A possible fix without touching the SPI would be to recompute the tuning and
the max-resource check from the post-chain summaries before persisting, or to
skip persisting {{configOverrides}} when the chain modified the decision. That
said, FLINK-40455 plans to decouple tuning to run after the scaling decision
and see its outcome, which would resolve this ordering naturally, so this may
just be useful input for that work.
Happy to put up a PR with the recompute approach and a regression test if that
seems like the right direction.
Related: FLINK-39555, FLINK-40455, [PR
#1085|https://github.com/apache/flink-kubernetes-operator/pull/1085],
[FLIP-575|https://cwiki.apache.org/confluence/display/FLINK/FLIP-575%3A+Support+Customized+Autoscale+Algorithm],
[autotuning redesign
discussion|https://lists.apache.org/thread/vk6j8793rx2y78vhh1yvfjf4qntpfkm9]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
