[
https://issues.apache.org/jira/browse/FLINK-40534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubhankar Sunil Unhale updated FLINK-40534:
--------------------------------------------
Affects Version/s: 1.16.1
1.16.0
> Memory tuning config overrides are computed before the ScalingExecutorPlugin
> chain and persisted unchanged when a plugin modifies the scaling decision
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40534
> URL: https://issues.apache.org/jira/browse/FLINK-40534
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.16.0, 1.16.1
> Reporter: Shubhankar Sunil Unhale
> Priority: Major
>
> When memory tuning is enabled via {{job.autoscaler.memory.tuning.enabled}}
> and a {{ScalingExecutorPlugin}} modifies the scaling decision, which FLIP-575
> explicitly supports ("Modified -> use modified decision"), the persisted
> memory overrides still reflect the original pre-plugin decision.
> In {{ScalingExecutor#scaleResource}}, on main at commit 38a9f197:
> # {{computeScalingSummary}} builds the summaries
> ({{ScalingExecutor.java:172-179}})
> # {{MemoryTuning.tuneTaskManagerMemory}} computes {{configOverrides}} from
> them ({{:192-198}})
> # the max-resource/quota check also runs on them ({{:200-207}})
> # the plugin chain may modify the summaries ({{:210-214}})
> # the modified parallelism is persisted ({{:231-234}}), but
> {{storeConfigChanges(context, configOverrides)}} persists the step-2 result
> unchanged ({{:236}})
> As a result, the applied TM memory sizing, including total process size,
> network min/max, managed fraction and metaspace, is computed for a decision
> that never deploys. The tuning output depends on the per-vertex decision:
> network memory comes from {{computeNewParallelisms}} and heap from
> {{MemoryScaling.applyMemoryScaling}}, so changing even one vertex leaves the
> whole job-wide TM config mismatched ({{MemoryTuning.java:158-159}},
> {{:175-177}}).
> A plugin can't correct this itself. The SPI method {{apply(Context,
> Map<JobVertexID, ScalingSummary>)}} exposes no {{ConfigChanges}} surface, and
> the config reachable from the context is a defensive copy
> ({{ScalingExecutorPlugin.java:88-89}}, {{ConfigChanges.java:60-66}}). Since
> FLINK-39959 a plugin can _see_ the tuned config, but has no write-back path.
> A related consequence of the same ordering: since the max-resource/quota
> check in step 3 also runs before the chain, a plugin that increases
> parallelism bypasses the cluster-resource and quota guards entirely.
> Scope: the veto path is safe since it returns before anything is persisted,
> and memory tuning is off by default, so hitting this needs tuning enabled
> plus a decision-modifying plugin ({{ScalingExecutor.java:215-217}},
> {{AutoScalerOptions.java:286-289}}). {{ScalingExecutorPluginTest}} doesn't
> cover the tuning interaction, and the plugins documentation page doesn't
> mention it.
> A possible fix without touching the SPI would be to recompute the tuning and
> the max-resource check from the post-chain summaries before persisting, or to
> skip persisting {{configOverrides}} when the chain modified the decision.
> That said, FLINK-40455 plans to decouple tuning to run after the scaling
> decision and see its outcome, which would resolve this ordering naturally, so
> this may just be useful input for that work.
> Happy to put up a PR with the recompute approach and a regression test if
> that seems like the right direction.
> Related: FLINK-39555, FLINK-40455, [PR
> #1085|https://github.com/apache/flink-kubernetes-operator/pull/1085],
> [FLIP-575|https://cwiki.apache.org/confluence/display/FLINK/FLIP-575%3A+Support+Customized+Autoscale+Algorithm],
> [autotuning redesign
> discussion|https://lists.apache.org/thread/vk6j8793rx2y78vhh1yvfjf4qntpfkm9]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
