Dennis-Mircea Ciupitu created FLINK-39959:
---------------------------------------------
Summary: Improve the autoscaler context structure and propagation
Key: FLINK-39959
URL: https://issues.apache.org/jira/browse/FLINK-39959
Project: Flink
Issue Type: Improvement
Components: Autoscaler, Kubernetes Operator
Affects Versions: kubernetes-operator-1.15.0
Reporter: Dennis-Mircea Ciupitu
h1. Background
As part of the recent extensibility work, several plugin SPIs were introduced
into the autoscaler: the custom metric evaluator (FLIP-514), the parallelism
alignment mode (FLIP-586), and the scaling executor plugin (FLIP-575). Each SPI
defined its own nested {{Context}} type to carry the inputs a plugin needs at
its phase of the scaling process.
h1. Problem
Each plugin context independently wraps {{JobAutoScalerContext}} and re-exposes
or re-derives overlapping data such as the configuration, the evaluated
metrics, and the job topology. This has several downsides.
* Multiple parallel context types carry duplicated fields and getters.
* Configuration is exposed in more than one place. The autoscaler context
already carries the job configuration, yet the plugin contexts add their own
configuration views, which is confusing for plugin authors and easy to drift
out of sync.
* There is no single authoritative context flowing through the autoscaler
backbone. Each stage (collector, evaluator, executor) and each plugin builds
its own view, so propagation is ad hoc.
h1. Proposed improvement
Define a single main autoscaler context as the backbone, and have the plugin
contexts extend or enrich it instead of wrapping and duplicating it.
* One canonical context, built on {{JobAutoScalerContext}}, is created once per
autoscaling cycle and threaded through the backbone components.
* Each plugin phase enriches that context with only the additional,
phase-specific information it needs, instead of re-declaring shared fields.
* Per-instance and per-plugin configuration is layered on the single context
rather than duplicated next to the job configuration.
h2. Design notes
The operator already uses a compositional enriched-context pattern, where
{{FlinkResourceContext}} wraps the JOSDK {{Context}}. The autoscaler context
model should adopt a consistent single-context approach. The exact mechanism
(inheritance versus layered composition), and the way per-instance
configuration is merged and exposed, should be settled during design, while
keeping the concrete subtype such as {{KubernetesJobAutoScalerContext}}
accessible to plugins.
h1. Benefits
* Removes duplicated context fields and getters across the plugin SPIs.
* Gives plugin authors a single, consistent configuration surface.
* Makes context propagation across the autoscaler backbone explicit and
uniform, from the collector through the evaluator and executor to the plugins.
* Lowers maintenance cost. New context data is added once and becomes available
everywhere.
h1. Scope
This is a follow-up improvement agreed during the FLIP-575
({{ScalingExecutorPlugin}}) review, deferred to avoid blocking that PR. It
consolidates the existing context types and does not change plugin behavior or
introduce new SPIs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
