Hi Yanis, Based on the information that you provided, I assume you run under this setup: Flink Kubernetes Operator in native mode, with deployments in application mode. Having this in mind, I'm framing the following response.
1. Resource-shape mismatch I see this as a provisioning problem more than a Flink one. Either the node shape has to match the JM + TM shape, or JM + TM must be designed to run under the ratio that the node provides. There is no Flink core or operator feature that fixes this automatically today. The operator's memory autotuning goes in that direction, since it moves TM memory requests toward real usage, meaning that in the end the job will run at the best CPU/memory ratio, but right now it only runs on scaling decisions. Decoupling it from scaling is tracked in FLINK-40455 [1], and once that matures (to include CPU tuning as well), it should help exactly this case. 2. Fragmentation There are 2 important things here: - The default GKE scheduler prefers emptier / balanced nodes, so it spreads your TMs, and nothing ever moves them back together. - Once the TM pods are spread, nothing in Kubernetes ever moves them back together. What I would try, based on what I know about GKE and how native mode behaves, is to switch the cluster autoscaler to the packing profile (optimize-utilization) so new pods land on busy nodes, and by marking the TM pods as safe to evict and JM pods with the opposite annotation. Both are plain pod annotations, so the operator's podTemplate can carry them. Eviction is workable for Flink because the JM notices the lost TM and requests a replacement, which is then scheduled onto the packed nodes. The real cost is that each job's task on the TM that was hosted on the consolidated nodes will need to be restarted from a checkpoint. With the adaptive scheduler in place, this will be made much smoother, since the job can keep running with fewer slots while the replacement comes up. So it ends up as a tradeoff between wasted capacity and how much disruption each job can accept. In the long term, I think the operator could play an important role here, as it already takes jobs through full redeploys for spec upgrades, and those moments recreate all the TM pods anyway, so they are natural windows to let the cluster repack without paying any extra disruption. Besides this, as I said, there will be some nice features added within the operator that will help those use cases. Best, Dennis [1] https://issues.apache.org/jira/browse/FLINK-40455 On Thu, Aug 27, 2026 at 2:14 AM Yanis Djeridi via dev <[email protected]> wrote: > Hi all, > > We run hundreds of Flink pipelines across several shared GKE clusters and > experience significant cluster overhead. By cluster overhead, we mean > allocatable CPU and memory available on active nodes that remain > unrequested but cannot be reclaimed because the nodes still host pods. > > > This comes mainly from two sources: > > > - > > TaskManager CPU/memory request ratios do not always align with our > machine shapes, leaving capacity stranded when either CPU or memory is > exhausted first. > - > > Deployments and autoscaling gradually scatter TaskManagers across nodes. > Because TaskManagers are bare pods, GKE cannot evict them for > consolidation, and Kubernetes does not proactively repack them. > > > As a result, partially occupied nodes can remain running indefinitely > despite substantial aggregate free capacity. > > Have others operating large shared Flink clusters encountered this? How do > you manage resource-shape mismatch and fragmentation? > > > Thanks, > > Yanis >
