Hi Lorenzo, and thanks Anton for the detailed review. +1 to the overall direction. I'd like to add one more dimension to the rolling-upgrade story that I think the FIP currently under-specifies: the operator must gate progression on replica recovery on the way in, not only on leader migration on the way out.
Today the FIP talks about integrating with controlled shutdown, which protects the outgoing pod (leader is migrated before termination). But once the new pod comes up, a Fluss process can become listener-ready within seconds, while: - log replicas may still be catching up to the leader, - PK/KV replicas may still be downloading the snapshot from remote storage and replaying the changelog up to the high-watermark, - none of the buckets hosted on that server have rejoined the ISR yet. If Pod.Ready flips to true at process-start time, a StatefulSet RollingUpdate will happily move on to the next ordinal, and the operator can find itself restarting pod N+1 while pod N is still recovering. For PK tables with non-trivial RocksDB state this window can be minutes. This connects to Anton's point that "TCP probes are too weak for operator readiness". I'd push it one step further: 1. Define Pod.Ready as "fully recovered", not just "process up". Concretely: all replicas hosted on the server are LogCaughtUp && (KvRestored if PK) && InSyncWithLeader. The cleanest implementation is a Pod readiness gate (e.g. fluss.apache.org/replicas-recovered=True) flipped by the operator after polling the cluster, rather than overloading the HTTP/TCP probe. 2. Make the rolling-upgrade gate a two-condition predicate: (a) the previously-restarted pod is fully recovered, AND (b) cluster-level invariants under-min-isr buckets == 0 and kv-recovering replicas == 0 held for a configurable stabilization window. 3. Freeze the observability contract here, even if drain stays in a companion FIP. Path A as written cannot actually answer "is replica recovery complete?" — replicasOnTabletServer() being internal (Anton's point) is one example, but the bigger gap is that there is no cluster-visible signal for KV restore progress or follower lag aggregated per server. Concretely I'd like the FIP to commit to exposing: - per-bucket / per-server in-sync predicate - kv.snapshot.restore.in-progress and restore.bytes-remaining - log.follower.lag-bytes / lag-time-ms - cluster-level under-min-isr-bucket-count and offline-bucket-count 4. Extend status.tabletServer.pods[] accordingly: recovered, underReplicatedBuckets, kvRestoreProgress, logCatchupLagBytes; and a cluster-level summary (underMinIsrBucketCount, offlineBucketCount). This also gives operators a SRE-grade observable surface during incidents. 5. Distinguish log-only vs PK workloads in defaults. A PK TabletServer can take orders of magnitude longer to recover than a log-only one. A single perPodTimeout is misleading; consider splitting controlledShutdownTimeout from recoveryTimeout. On timeout the upgrade should stall with a Stalled condition, not auto-rollback — auto-rollback for stateful systems is dangerous and the operator should require explicit human input (e.g. an annotation) to resume or proceed. 6. Coordinator/TabletServer ordering during upgrade. With coordinator.replicas=1, there is a Coordinator-unavailable window during its own upgrade. ControlledShutdown, ISR changes and KV-standby coordination all depend on the Coordinator being up. The FIP should state that TabletServer rolling only begins after the Coordinator is upgraded and has stabilized for the same window, and that each TS step re-checks Coordinator health. Otherwise +1 from me. Looking forward to the poc code. Yours, Yunhong Zheng (Swuferhong) On 2026/05/12 16:17:20 Anton Borisov wrote: > Hi Lorenzo, > > Thanks for writing this up. I like the direction. An operator is the > right next step after the Helm chart, and the proposed shape is broadly > reasonable. > > I checked the FIP against the current code and I think a few points > are worth discussing/considering: > > 1. Drain / scale-in > > The FIP says Path B needs a new drain primitive, while Path A only blocks > scale-in if the target TabletServer still has replicas. > > I think we already have most of the low-level pieces: > > - ServerTag.PERMANENT_OFFLINE / TEMPORARY_OFFLINE > - AddServerTagRequest / RemoveServerTagRequest > - RebalanceRequest > - ListRebalanceProgressRequest > - RebalanceStatus > > From the rebalance code, offline-tagged servers are not just excluded > from new placements. ReplicaDistributionGoal moves replicas out of them, > including followers, and the target replica count is effectively zero. > > So scale-in could be: > > AddServerTag(PERMANENT_OFFLINE) > -> Rebalance > -> wait for COMPLETED > -> scale down StatefulSet > > TEMPORARY_OFFLINE may also be usable before rolling restart. > > Path A also needs server work as written. The operator needs to know > whether a TabletServer still has replicas, but that count is internal in > CoordinatorContext.replicasOnTabletServer() and is not exposed through > Admin.getServerNodes(). > > So I do not think the choice is “Path A without server work” vs “Path B > with server work”. Both need a small server/API decision. > > The important caveats are: > > - rebalance can fail if the remaining TabletServers cannot absorb the > replicas; > - I do not see min-ISR being considered by the rebalance executor, so > decommission close to min-ISR may create write-availability windows; > - tag + rebalance is currently multi-step, so the operator needs crash > recovery if it dies between the two calls; > - the operator would depend on @PublicEvolving Admin/rebalance APIs. > > Given that, I think a small wrapper may be cleaner: > > DecommissionServer(serverId) -> rebalanceId > > The server would own tag + rebalance as one operation. The operator would > start it and poll progress. > > 2. Helm naming / adoption > > The FIP says operator naming matches the Helm chart to support in-place > adoption. The current chart uses fixed names such as tablet-server, > tablet-server-hs, coordinator-server and coordinator-server-hs. > > That means two Helm installs in one namespace collide. If the operator > keeps these names, v1alpha1 is effectively one FlussCluster per > namespace. > > I think the FIP should either state that constraint, or move both chart > and operator resources to cluster-prefixed names. Otherwise adoption and > multi-cluster UX are pulling in different directions. > > Smaller points > > - I would not use both status.phase and conditions. Conditions are enough > and avoid forcing mutually-overlapping states into one enum. > - TCP probes are too weak for operator readiness. Readiness should mean > the server is actually usable from Fluss’ point of view: registered > with the Coordinator, ZK connected, etc. > - PVC retention should use StatefulSet > persistentVolumeClaimRetentionPolicy. I would not add custom PVC > finalizer logic unless we really need it. > - Lake tiering is not mentioned. It is fine to keep it out of v1alpha1, > but the FIP should say that explicitly. > - Your "Configuration Updates" section asserts the registry is sourced > from Fluss, but your Open Questions section > correctly flags this as unresolved - I'd align the body with Open > Questions. For v1alpha1 a small client-side allowlist mirrored > from DynamicServerConfig.ALLOWED_CONFIG_KEYS is probably fine I reckon > > > Overall, I like the proposal. The main thing I would clarify is the > operator/server boundary: what the operator can safely orchestrate today, > what needs a small server API, and what should be left out of v1alpha1. > > -- Anton > > чт, 7 мая 2026 г. в 09:57, Lorenzo Affetti via dev <[email protected]>: > > > > Hi Michael, > > > > Thanks for the careful read. > > > > *Framework*: Java Operator SDK, not plain fabric8. JOSDK is built on > > fabric8 anyway, so we keep the option to drop down whether needed. > > > > *Minimum Kubernetes version*: not currently fixed in the FIP. My proposal > > is 1.29 as the floor. > > > > Reasoning for framework: > > > > For a Fluss operator that has to coordinate rolling upgrades, scale-in > > safety gates, dynamic vs restart-inducing config diffs, and a migration > > state machine, JOSDK's dependent-resource and workflow primitives are > > well-aligned. Fabric8 alone would push us toward reimplementing them > > ourselves. > > > > The clearest signal is Strimzi. Strimzi predates JOSDK and was built > > directly on fabric8—but their newer components (Access operator, Schema > > Registry operator) use JOSDK. The most mature ASF distributed-system > > operator effectively says: if we were starting today, we'd use JOSDK. > > > > For the version floor, the constraints are JOSDK 5.x (Java 17+) and the > > Kubernetes APIs the operator uses — all GA well before 1.27: CRD v1 (1.16), > > admission webhook v1 (1.16), PDB v1 (1.21), Lease v1 (1.14). > > The useful-but-optional StatefulSet persistentVolumeClaimRetentionPolicy > > was alpha in 1.23, beta in 1.27, GA in 1.32. > > Kubernetes upstream currently patches 1.31–1.33. > > > > *1.29 keeps us within a defensible distance of upstream while leaving room > > for users on enterprise distributions a release or two behind. 1.27 is more > > permissive; 1.30+ stricter.*Open to community input on what Fluss users > > actually run. > > > > I'll fold both into the FIP once we converge. > > > > Thank you! > > > > On Wed, May 6, 2026 at 3:51 PM Michael Koepf <[email protected]> > > wrote: > > > > > Hi Lorenzo, > > > > > > Thanks for the FIP. I believe a dedicated Fluss Kubernetes Operator will > > > further simplify deployment and operations in large-scale production > > > environments. > > > > > > I skimmed over the FIP. > > > > > > > We propose introducing a Fluss Kubernetes Operator, implemented in Java > > > [...] > > > > > > 1. The first question that comes to my mind; are there already detailed > > > plans regarding implementation? Do you plan to use the Java Operator SDK > > > framework (https://javaoperatorsdk.io/)? Or just the plain fabric8 Java > > > Kubernetes client (https://github.com/fabric8io/kubernetes-client)? > > > > > > 2. What will be the minimum supported Kubernetes version? > > > > > > Looking forward to this. > > > > > > Thanks. > > > -- > > > Best, > > > Michael > > > > > > On 2026/05/05 09:05:45 Lorenzo Affetti via dev wrote: > > > > Hello community! > > > > > > > > I would like to start a discussion about FIP-41: Fluss Kubernetes > > > Operator. > > > > Here is the motivation: > > > > > > > > Fluss 0.8 introduced a Helm chart > > > > <https://github.com/apache/fluss/issues/779> that simplifies the initial > > > > deployment of a Fluss cluster on Kubernetes by packaging manifests, > > > > configuration, and dependencies into a versioned release. While this is > > > > a > > > > good foundation, a Helm chart is fundamentally a one-shot templating > > > tool. > > > > It has no awareness of Fluss's runtime state and cannot react to > > > > operational events such as pod failures, rolling upgrades, or scale-in > > > > operations that risk data loss. > > > > > > > > Running Fluss in production on Kubernetes today requires users to > > > manually > > > > coordinate: > > > > > > > > - Safe rolling restarts of TabletServers, ensuring tablet leadership > > > is > > > > migrated before each pod terminates (Fluss 0.8 introduced the > > > > graceful > > > > shutdown > > > > < > > > https://fluss.apache.org/docs/maintenance/operations/graceful-shutdown/> > > > > primitive > > > > that makes this possible, but does not orchestrate it across pods) > > > > - Version upgrades that must sequence CoordinatorServer and > > > TabletServer > > > > updates correctly > > > > - Scale-in operations where a TabletServer must be drained of tablets > > > > before its pod is terminated > > > > - Recovery from partial failures (e.g., PVC reattachment, pod > > > > identity > > > > preservation across restarts) > > > > - Leveraging Fluss 0.8's dynamic configuration updates > > > > < > > > https://fluss.apache.org/docs/maintenance/operations/updating-configs/> > > > for > > > > keys that do not require restart, instead of triggering a rolling > > > restart > > > > for every config change > > > > > > > > This gap between deployment and operations is best addressed by a > > > *Kubernetes > > > > Operator* — a controller that continuously reconciles the desired state > > > of > > > > a FlussCluster resource against the actual state of the cluster, and > > > > executes Fluss-aware transitions safely. > > > > > > > > The Flink and Spark ecosystems have established this pattern > > > > successfully > > > > with the Apache Flink Kubernetes Operator > > > > <https://github.com/apache/flink-kubernetes-operator> and the Apache > > > Spark > > > > Kubernetes Operator <https://github.com/apache/spark-kubernetes-operator > > > >. > > > > Fluss should follow suit. > > > > > > > > -- > > > > Lorenzo Affetti > > > > Team Leader of Stream Storage > > > > [email protected] > > > > www.ververica.com > > > > ------------------------------ > > > > > > > > <https://www.ververica.com/> > > > > Ververica GmbH | Herzogspitalstrasse 24 | 80331 München | Germany > > > > > > > > Follow us: > > > > <https://www.linkedin.com/company/ververica/posts/?feedView=all> > > > > <https://www.youtube.com/@ververica> > > > > < > > > https://open.spotify.com/show/2XME9h8iBOyr6YupqM99ir?si=87b064644add42a1 > > > >Available > > > > on: <https://aws.amazon.com/marketplace/pp/prodview-luvmqd6leha4i> > > > > < > > > https://marketplace.microsoft.com/en-us/product/saas/ververica.vvc_managed?tab=Overview > > > > > > > > > > > > Pflichtangaben/Mandatory Information > > > > <https://www.ververica.com/mandatory-information> > > > > > > > > > > > > > -- > > Lorenzo Affetti > > Senior Software Engineer @ Flink Team > > Ververica <http://www.ververica.com> >
