Hi all, I’d like to start a discussion on FLIP-XXX: Flink Kubernetes Operator Autotuning Redesign [1].
Flink Autotuning stands out as one of the most powerful components the operator can offer, and as the place where the real cost optimizations of a Flink platform are achieved. Autoscaling decides how many resources a job runs on, while autotuning decides how efficiently every provisioned resource is used. Fully developed, it continuously right-sizes memory, task slots, JobManager resources, and CPU for every managed job, without changing job behavior and without requiring users to be experts in Flink's memory model. Today, however, autotuning ships as a sub-feature of Flink Autoscaling. In practice this means it cannot run when the autoscaler is disabled, fresh recommendations are only computed when a scaling decision is actually made, and there is no seam for adding new tuning types. The tuning roadmap under FLINK-34538 [2] reflects this: JobManager memory (FLINK-34539) and task slots (FLINK-34540) tuning have been open and unassigned for about two years, and RocksDB-aware or CPU tuning have no tickets at all. The FLIP proposes promoting autotuning to a first-class optimizer, following the precedent of FLIP-334 [3], which decoupled the autoscaler from the operator. In short: * A new flink-autotuner module as a peer of flink-autoscaler, plus a small shared flink-optimizer-common base module. * A pluggable Tuner interface, so new tuning types (JobManager memory, task slots, RocksDB, CPU) can be added without touching the autoscaler & autotuner. * A target-aware ConfigChanges type, so the realizer can dispatch JobManager, TaskManager, Flink config, and Kubernetes changes without key-prefix heuristics. * An operator-side coordinator that runs both optimizers on a single metric snapshot per reconcile. The autotuner runs on every reconcile, and enablement of the two is fully orthogonal. * A dedicated job.autotuner.* config namespace, with the existing job.autoscaler.memory.tuning.* keys kept as deprecated aliases, plus a new floor-ratio option so the spec can act as a configurable lower bound for memory recommendations. The current TaskManager memory tuning algorithm itself is ported as-is. This FLIP is about the architecture, not the tuning math. Looking forward to your thoughts and suggestions! Best regards, Dennis [1] https://docs.google.com/document/d/1JuNa-_7NwwL-GjqWHm8nnUuFpdNtGuLJj4Zfnl8m_O0/edit?usp=sharing [2] https://issues.apache.org/jira/browse/FLINK-34538 [3] https://cwiki.apache.org/confluence/spaces/FLINK/pages/263424711/FLIP-334+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler
