Hi Lorenzo And thank you for the proposal. I'm sharing similar thoughts to the previous comments. Overall, the proposal is great, but there are a few things I think will help improve.
Cluster rebalance is an important operation, and Fluss already supports it, so I think it would be great to address: 1. Scale-out without a rebalance trigger leaves clusters unbalanced, so the operator must call rebalance() post scale-out and poll progress 2. Scale-in with no drain path is unusable, so the operator must do TEMPORARY_OFFLINE tag + rebalance + verify zero-bucket must precede replica reduction 3. Server ID reuse on PVC recycle corrupts ZK state, so the operator design should have ID assignment; so I would like to see the PVC lifecycle ownership I think it might also be useful to have a terminationGracePeriodSeconds/preStop hook design For PVCs, it's also not clear to me about the model, and I think it should be clear that we never auto-delete PVCs With the above, we can set a good foundation for the operator, and with these addressed its also a +1 for me Best, Giannis On Thu, May 14, 2026 at 9:51 AM Yunhong Zheng <[email protected]> wrote: > Hi Lorenzo, and thanks Anton for the detailed review. > > +1 to the overall direction. I'd like to add one more dimension to the > rolling-upgrade story that I think the FIP currently under-specifies: the > operator must gate progression on replica recovery on the way in, not only > on leader migration on the way out. > > Today the FIP talks about integrating with controlled shutdown, which > protects the outgoing pod (leader is migrated before termination). But once > the new pod comes up, a Fluss process can become listener-ready within > seconds, while: > - log replicas may still be catching up to the leader, > - PK/KV replicas may still be downloading the snapshot from remote > storage and replaying the changelog up to the high-watermark, > - none of the buckets hosted on that server have rejoined the ISR yet. > > If Pod.Ready flips to true at process-start time, a StatefulSet > RollingUpdate will happily move on to the next ordinal, and the operator > can find itself restarting pod N+1 while pod N is still recovering. For PK > tables with non-trivial RocksDB state this window can be minutes. > > This connects to Anton's point that "TCP probes are too weak for operator > readiness". I'd push it one step further: > > 1. Define Pod.Ready as "fully recovered", not just "process up". > Concretely: all replicas hosted on the server are LogCaughtUp && > (KvRestored if PK) && InSyncWithLeader. The cleanest implementation is a > Pod readiness gate (e.g. fluss.apache.org/replicas-recovered=True) > flipped by the operator after polling the cluster, rather than overloading > the HTTP/TCP probe. > > 2. Make the rolling-upgrade gate a two-condition predicate: > (a) the previously-restarted pod is fully recovered, AND > (b) cluster-level invariants under-min-isr buckets == 0 and kv-recovering > replicas == 0 held for a configurable stabilization window. > > 3. Freeze the observability contract here, even if drain stays in a > companion FIP. Path A as written cannot actually answer "is replica > recovery complete?" — replicasOnTabletServer() being internal (Anton's > point) is one example, but the bigger gap is that there is no > cluster-visible signal for KV restore progress or follower lag aggregated > per server. Concretely I'd like the FIP to commit to exposing: > - per-bucket / per-server in-sync predicate > - kv.snapshot.restore.in-progress and restore.bytes-remaining > - log.follower.lag-bytes / lag-time-ms > - cluster-level under-min-isr-bucket-count and offline-bucket-count > > 4. Extend status.tabletServer.pods[] accordingly: recovered, > underReplicatedBuckets, kvRestoreProgress, logCatchupLagBytes; and a > cluster-level summary (underMinIsrBucketCount, offlineBucketCount). This > also gives operators a SRE-grade observable surface during incidents. > > 5. Distinguish log-only vs PK workloads in defaults. A PK TabletServer can > take orders of magnitude longer to recover than a log-only one. A single > perPodTimeout is misleading; consider splitting controlledShutdownTimeout > from recoveryTimeout. On timeout the upgrade should stall with a Stalled > condition, not auto-rollback — auto-rollback for stateful systems is > dangerous and the operator should require explicit human input (e.g. an > annotation) to resume or proceed. > > 6. Coordinator/TabletServer ordering during upgrade. With > coordinator.replicas=1, there is a Coordinator-unavailable window during > its own upgrade. ControlledShutdown, ISR changes and KV-standby > coordination all depend on the Coordinator being up. The FIP should state > that TabletServer rolling only begins after the Coordinator is upgraded and > has stabilized for the same window, and that each TS step re-checks > Coordinator health. > > Otherwise +1 from me. Looking forward to the poc code. > > Yours, > Yunhong Zheng (Swuferhong) > > On 2026/05/12 16:17:20 Anton Borisov wrote: > > Hi Lorenzo, > > > > Thanks for writing this up. I like the direction. An operator is the > > right next step after the Helm chart, and the proposed shape is broadly > > reasonable. > > > > I checked the FIP against the current code and I think a few points > > are worth discussing/considering: > > > > 1. Drain / scale-in > > > > The FIP says Path B needs a new drain primitive, while Path A only blocks > > scale-in if the target TabletServer still has replicas. > > > > I think we already have most of the low-level pieces: > > > > - ServerTag.PERMANENT_OFFLINE / TEMPORARY_OFFLINE > > - AddServerTagRequest / RemoveServerTagRequest > > - RebalanceRequest > > - ListRebalanceProgressRequest > > - RebalanceStatus > > > > From the rebalance code, offline-tagged servers are not just excluded > > from new placements. ReplicaDistributionGoal moves replicas out of them, > > including followers, and the target replica count is effectively zero. > > > > So scale-in could be: > > > > AddServerTag(PERMANENT_OFFLINE) > > -> Rebalance > > -> wait for COMPLETED > > -> scale down StatefulSet > > > > TEMPORARY_OFFLINE may also be usable before rolling restart. > > > > Path A also needs server work as written. The operator needs to know > > whether a TabletServer still has replicas, but that count is internal in > > CoordinatorContext.replicasOnTabletServer() and is not exposed through > > Admin.getServerNodes(). > > > > So I do not think the choice is “Path A without server work” vs “Path B > > with server work”. Both need a small server/API decision. > > > > The important caveats are: > > > > - rebalance can fail if the remaining TabletServers cannot absorb the > > replicas; > > - I do not see min-ISR being considered by the rebalance executor, so > > decommission close to min-ISR may create write-availability windows; > > - tag + rebalance is currently multi-step, so the operator needs crash > > recovery if it dies between the two calls; > > - the operator would depend on @PublicEvolving Admin/rebalance APIs. > > > > Given that, I think a small wrapper may be cleaner: > > > > DecommissionServer(serverId) -> rebalanceId > > > > The server would own tag + rebalance as one operation. The operator would > > start it and poll progress. > > > > 2. Helm naming / adoption > > > > The FIP says operator naming matches the Helm chart to support in-place > > adoption. The current chart uses fixed names such as tablet-server, > > tablet-server-hs, coordinator-server and coordinator-server-hs. > > > > That means two Helm installs in one namespace collide. If the operator > > keeps these names, v1alpha1 is effectively one FlussCluster per > > namespace. > > > > I think the FIP should either state that constraint, or move both chart > > and operator resources to cluster-prefixed names. Otherwise adoption and > > multi-cluster UX are pulling in different directions. > > > > Smaller points > > > > - I would not use both status.phase and conditions. Conditions are enough > > and avoid forcing mutually-overlapping states into one enum. > > - TCP probes are too weak for operator readiness. Readiness should mean > > the server is actually usable from Fluss’ point of view: registered > > with the Coordinator, ZK connected, etc. > > - PVC retention should use StatefulSet > > persistentVolumeClaimRetentionPolicy. I would not add custom PVC > > finalizer logic unless we really need it. > > - Lake tiering is not mentioned. It is fine to keep it out of v1alpha1, > > but the FIP should say that explicitly. > > - Your "Configuration Updates" section asserts the registry is sourced > > from Fluss, but your Open Questions section > > correctly flags this as unresolved - I'd align the body with Open > > Questions. For v1alpha1 a small client-side allowlist mirrored > > from DynamicServerConfig.ALLOWED_CONFIG_KEYS is probably fine I reckon > > > > > > Overall, I like the proposal. The main thing I would clarify is the > > operator/server boundary: what the operator can safely orchestrate today, > > what needs a small server API, and what should be left out of v1alpha1. > > > > -- Anton > > > > чт, 7 мая 2026 г. в 09:57, Lorenzo Affetti via dev <[email protected] > >: > > > > > > Hi Michael, > > > > > > Thanks for the careful read. > > > > > > *Framework*: Java Operator SDK, not plain fabric8. JOSDK is built on > > > fabric8 anyway, so we keep the option to drop down whether needed. > > > > > > *Minimum Kubernetes version*: not currently fixed in the FIP. My > proposal > > > is 1.29 as the floor. > > > > > > Reasoning for framework: > > > > > > For a Fluss operator that has to coordinate rolling upgrades, scale-in > > > safety gates, dynamic vs restart-inducing config diffs, and a migration > > > state machine, JOSDK's dependent-resource and workflow primitives are > > > well-aligned. Fabric8 alone would push us toward reimplementing them > > > ourselves. > > > > > > The clearest signal is Strimzi. Strimzi predates JOSDK and was built > > > directly on fabric8—but their newer components (Access operator, Schema > > > Registry operator) use JOSDK. The most mature ASF distributed-system > > > operator effectively says: if we were starting today, we'd use JOSDK. > > > > > > For the version floor, the constraints are JOSDK 5.x (Java 17+) and the > > > Kubernetes APIs the operator uses — all GA well before 1.27: CRD v1 > (1.16), > > > admission webhook v1 (1.16), PDB v1 (1.21), Lease v1 (1.14). > > > The useful-but-optional StatefulSet > persistentVolumeClaimRetentionPolicy > > > was alpha in 1.23, beta in 1.27, GA in 1.32. > > > Kubernetes upstream currently patches 1.31–1.33. > > > > > > *1.29 keeps us within a defensible distance of upstream while leaving > room > > > for users on enterprise distributions a release or two behind. 1.27 is > more > > > permissive; 1.30+ stricter.*Open to community input on what Fluss users > > > actually run. > > > > > > I'll fold both into the FIP once we converge. > > > > > > Thank you! > > > > > > On Wed, May 6, 2026 at 3:51 PM Michael Koepf <[email protected]> > > > wrote: > > > > > > > Hi Lorenzo, > > > > > > > > Thanks for the FIP. I believe a dedicated Fluss Kubernetes Operator > will > > > > further simplify deployment and operations in large-scale production > > > > environments. > > > > > > > > I skimmed over the FIP. > > > > > > > > > We propose introducing a Fluss Kubernetes Operator, implemented in > Java > > > > [...] > > > > > > > > 1. The first question that comes to my mind; are there already > detailed > > > > plans regarding implementation? Do you plan to use the Java Operator > SDK > > > > framework (https://javaoperatorsdk.io/)? Or just the plain fabric8 > Java > > > > Kubernetes client (https://github.com/fabric8io/kubernetes-client)? > > > > > > > > 2. What will be the minimum supported Kubernetes version? > > > > > > > > Looking forward to this. > > > > > > > > Thanks. > > > > -- > > > > Best, > > > > Michael > > > > > > > > On 2026/05/05 09:05:45 Lorenzo Affetti via dev wrote: > > > > > Hello community! > > > > > > > > > > I would like to start a discussion about FIP-41: Fluss Kubernetes > > > > Operator. > > > > > Here is the motivation: > > > > > > > > > > Fluss 0.8 introduced a Helm chart > > > > > <https://github.com/apache/fluss/issues/779> that simplifies the > initial > > > > > deployment of a Fluss cluster on Kubernetes by packaging manifests, > > > > > configuration, and dependencies into a versioned release. While > this is a > > > > > good foundation, a Helm chart is fundamentally a one-shot > templating > > > > tool. > > > > > It has no awareness of Fluss's runtime state and cannot react to > > > > > operational events such as pod failures, rolling upgrades, or > scale-in > > > > > operations that risk data loss. > > > > > > > > > > Running Fluss in production on Kubernetes today requires users to > > > > manually > > > > > coordinate: > > > > > > > > > > - Safe rolling restarts of TabletServers, ensuring tablet > leadership > > > > is > > > > > migrated before each pod terminates (Fluss 0.8 introduced the > graceful > > > > > shutdown > > > > > < > > > > > https://fluss.apache.org/docs/maintenance/operations/graceful-shutdown/> > > > > > primitive > > > > > that makes this possible, but does not orchestrate it across > pods) > > > > > - Version upgrades that must sequence CoordinatorServer and > > > > TabletServer > > > > > updates correctly > > > > > - Scale-in operations where a TabletServer must be drained of > tablets > > > > > before its pod is terminated > > > > > - Recovery from partial failures (e.g., PVC reattachment, pod > identity > > > > > preservation across restarts) > > > > > - Leveraging Fluss 0.8's dynamic configuration updates > > > > > < > > > > > https://fluss.apache.org/docs/maintenance/operations/updating-configs/> > > > > for > > > > > keys that do not require restart, instead of triggering a > rolling > > > > restart > > > > > for every config change > > > > > > > > > > This gap between deployment and operations is best addressed by a > > > > *Kubernetes > > > > > Operator* — a controller that continuously reconciles the desired > state > > > > of > > > > > a FlussCluster resource against the actual state of the cluster, > and > > > > > executes Fluss-aware transitions safely. > > > > > > > > > > The Flink and Spark ecosystems have established this pattern > successfully > > > > > with the Apache Flink Kubernetes Operator > > > > > <https://github.com/apache/flink-kubernetes-operator> and the > Apache > > > > Spark > > > > > Kubernetes Operator < > https://github.com/apache/spark-kubernetes-operator > > > > >. > > > > > Fluss should follow suit. > > > > > > > > > > -- > > > > > Lorenzo Affetti > > > > > Team Leader of Stream Storage > > > > > [email protected] > > > > > www.ververica.com > > > > > ------------------------------ > > > > > > > > > > <https://www.ververica.com/> > > > > > Ververica GmbH | Herzogspitalstrasse 24 | 80331 München | Germany > > > > > > > > > > Follow us: > > > > > <https://www.linkedin.com/company/ververica/posts/?feedView=all> > > > > > <https://www.youtube.com/@ververica> > > > > > < > > > > > https://open.spotify.com/show/2XME9h8iBOyr6YupqM99ir?si=87b064644add42a1 > > > > >Available > > > > > on: <https://aws.amazon.com/marketplace/pp/prodview-luvmqd6leha4i > > > > > > > < > > > > > https://marketplace.microsoft.com/en-us/product/saas/ververica.vvc_managed?tab=Overview > > > > > > > > > > > > > > > Pflichtangaben/Mandatory Information > > > > > <https://www.ververica.com/mandatory-information> > > > > > > > > > > > > > > > > > > -- > > > Lorenzo Affetti > > > Senior Software Engineer @ Flink Team > > > Ververica <http://www.ververica.com> > > >
