Hi Lorenzo. Thank you for the detailed proposal. This will improve the operational experience of maintaining Fluss clusters.
1. Since configurationOverrides is rendered straight into a ConfigMap, any credential-bearing keys would end up in plaintext. Could we add a sensitive-overrides path that resolves from a Secret, even if the non-sensitive map stays as-is for v1alpha1? This is likely to be an important factor in application security reviews of users considering to adopt Fluss operator. 2. Can you clarify how the operator will terminate/roll pods e.g. delete or eviction API? I believe using direct delete with pod disruption budget of n-1 might cause unavailability in TOCTOU scenarios as direct deletion bypasses PDB. For reference, strimzi sets maxUnavailable to zero (effectively minAvailable N) [1] and direct deletion, because they recognise that readiness probe is a weak signal of ISR status: “ the readiness probe doesn't guarantee that all replicas are in-sync again”. Best regards Keith [1] https://github.com/strimzi/proposals/blob/main/029-adopt-the-drain-cleaner-utility.md#disruptions On Wed, 27 May 2026 at 15:13, Giannis Polyzos <[email protected]> wrote: > Hi Lorenzo > And thank you for the proposal. > > I'm sharing similar thoughts to the previous comments. Overall, the > proposal is great, but there are a few things I think will help improve. > > Cluster rebalance is an important operation, and Fluss already supports it, > so I think it would be great to address: > 1. Scale-out without a rebalance trigger leaves clusters unbalanced, so the > operator must call rebalance() post scale-out and poll progress > 2. Scale-in with no drain path is unusable, so the operator must do > TEMPORARY_OFFLINE tag + rebalance + verify zero-bucket must precede replica > reduction > 3. Server ID reuse on PVC recycle corrupts ZK state, so the operator design > should have ID assignment; so I would like to see the PVC lifecycle > ownership > > I think it might also be useful to have a > terminationGracePeriodSeconds/preStop hook design > > For PVCs, it's also not clear to me about the model, and I think it should > be clear that we never auto-delete PVCs > > With the above, we can set a good foundation for the operator, and with > these addressed its also a +1 for me > > Best, > Giannis > > On Thu, May 14, 2026 at 9:51 AM Yunhong Zheng <[email protected]> wrote: > > > Hi Lorenzo, and thanks Anton for the detailed review. > > > > +1 to the overall direction. I'd like to add one more dimension to the > > rolling-upgrade story that I think the FIP currently under-specifies: the > > operator must gate progression on replica recovery on the way in, not > only > > on leader migration on the way out. > > > > Today the FIP talks about integrating with controlled shutdown, which > > protects the outgoing pod (leader is migrated before termination). But > once > > the new pod comes up, a Fluss process can become listener-ready within > > seconds, while: > > - log replicas may still be catching up to the leader, > > - PK/KV replicas may still be downloading the snapshot from remote > > storage and replaying the changelog up to the high-watermark, > > - none of the buckets hosted on that server have rejoined the ISR yet. > > > > If Pod.Ready flips to true at process-start time, a StatefulSet > > RollingUpdate will happily move on to the next ordinal, and the operator > > can find itself restarting pod N+1 while pod N is still recovering. For > PK > > tables with non-trivial RocksDB state this window can be minutes. > > > > This connects to Anton's point that "TCP probes are too weak for operator > > readiness". I'd push it one step further: > > > > 1. Define Pod.Ready as "fully recovered", not just "process up". > > Concretely: all replicas hosted on the server are LogCaughtUp && > > (KvRestored if PK) && InSyncWithLeader. The cleanest implementation is a > > Pod readiness gate (e.g. fluss.apache.org/replicas-recovered=True) > > flipped by the operator after polling the cluster, rather than > overloading > > the HTTP/TCP probe. > > > > 2. Make the rolling-upgrade gate a two-condition predicate: > > (a) the previously-restarted pod is fully recovered, AND > > (b) cluster-level invariants under-min-isr buckets == 0 and kv-recovering > > replicas == 0 held for a configurable stabilization window. > > > > 3. Freeze the observability contract here, even if drain stays in a > > companion FIP. Path A as written cannot actually answer "is replica > > recovery complete?" — replicasOnTabletServer() being internal (Anton's > > point) is one example, but the bigger gap is that there is no > > cluster-visible signal for KV restore progress or follower lag aggregated > > per server. Concretely I'd like the FIP to commit to exposing: > > - per-bucket / per-server in-sync predicate > > - kv.snapshot.restore.in-progress and restore.bytes-remaining > > - log.follower.lag-bytes / lag-time-ms > > - cluster-level under-min-isr-bucket-count and offline-bucket-count > > > > 4. Extend status.tabletServer.pods[] accordingly: recovered, > > underReplicatedBuckets, kvRestoreProgress, logCatchupLagBytes; and a > > cluster-level summary (underMinIsrBucketCount, offlineBucketCount). This > > also gives operators a SRE-grade observable surface during incidents. > > > > 5. Distinguish log-only vs PK workloads in defaults. A PK TabletServer > can > > take orders of magnitude longer to recover than a log-only one. A single > > perPodTimeout is misleading; consider splitting controlledShutdownTimeout > > from recoveryTimeout. On timeout the upgrade should stall with a Stalled > > condition, not auto-rollback — auto-rollback for stateful systems is > > dangerous and the operator should require explicit human input (e.g. an > > annotation) to resume or proceed. > > > > 6. Coordinator/TabletServer ordering during upgrade. With > > coordinator.replicas=1, there is a Coordinator-unavailable window during > > its own upgrade. ControlledShutdown, ISR changes and KV-standby > > coordination all depend on the Coordinator being up. The FIP should state > > that TabletServer rolling only begins after the Coordinator is upgraded > and > > has stabilized for the same window, and that each TS step re-checks > > Coordinator health. > > > > Otherwise +1 from me. Looking forward to the poc code. > > > > Yours, > > Yunhong Zheng (Swuferhong) > > > > On 2026/05/12 16:17:20 Anton Borisov wrote: > > > Hi Lorenzo, > > > > > > Thanks for writing this up. I like the direction. An operator is the > > > right next step after the Helm chart, and the proposed shape is broadly > > > reasonable. > > > > > > I checked the FIP against the current code and I think a few points > > > are worth discussing/considering: > > > > > > 1. Drain / scale-in > > > > > > The FIP says Path B needs a new drain primitive, while Path A only > blocks > > > scale-in if the target TabletServer still has replicas. > > > > > > I think we already have most of the low-level pieces: > > > > > > - ServerTag.PERMANENT_OFFLINE / TEMPORARY_OFFLINE > > > - AddServerTagRequest / RemoveServerTagRequest > > > - RebalanceRequest > > > - ListRebalanceProgressRequest > > > - RebalanceStatus > > > > > > From the rebalance code, offline-tagged servers are not just excluded > > > from new placements. ReplicaDistributionGoal moves replicas out of > them, > > > including followers, and the target replica count is effectively zero. > > > > > > So scale-in could be: > > > > > > AddServerTag(PERMANENT_OFFLINE) > > > -> Rebalance > > > -> wait for COMPLETED > > > -> scale down StatefulSet > > > > > > TEMPORARY_OFFLINE may also be usable before rolling restart. > > > > > > Path A also needs server work as written. The operator needs to know > > > whether a TabletServer still has replicas, but that count is internal > in > > > CoordinatorContext.replicasOnTabletServer() and is not exposed through > > > Admin.getServerNodes(). > > > > > > So I do not think the choice is “Path A without server work” vs “Path B > > > with server work”. Both need a small server/API decision. > > > > > > The important caveats are: > > > > > > - rebalance can fail if the remaining TabletServers cannot absorb the > > > replicas; > > > - I do not see min-ISR being considered by the rebalance executor, so > > > decommission close to min-ISR may create write-availability windows; > > > - tag + rebalance is currently multi-step, so the operator needs crash > > > recovery if it dies between the two calls; > > > - the operator would depend on @PublicEvolving Admin/rebalance APIs. > > > > > > Given that, I think a small wrapper may be cleaner: > > > > > > DecommissionServer(serverId) -> rebalanceId > > > > > > The server would own tag + rebalance as one operation. The operator > would > > > start it and poll progress. > > > > > > 2. Helm naming / adoption > > > > > > The FIP says operator naming matches the Helm chart to support in-place > > > adoption. The current chart uses fixed names such as tablet-server, > > > tablet-server-hs, coordinator-server and coordinator-server-hs. > > > > > > That means two Helm installs in one namespace collide. If the operator > > > keeps these names, v1alpha1 is effectively one FlussCluster per > > > namespace. > > > > > > I think the FIP should either state that constraint, or move both chart > > > and operator resources to cluster-prefixed names. Otherwise adoption > and > > > multi-cluster UX are pulling in different directions. > > > > > > Smaller points > > > > > > - I would not use both status.phase and conditions. Conditions are > enough > > > and avoid forcing mutually-overlapping states into one enum. > > > - TCP probes are too weak for operator readiness. Readiness should mean > > > the server is actually usable from Fluss’ point of view: registered > > > with the Coordinator, ZK connected, etc. > > > - PVC retention should use StatefulSet > > > persistentVolumeClaimRetentionPolicy. I would not add custom PVC > > > finalizer logic unless we really need it. > > > - Lake tiering is not mentioned. It is fine to keep it out of v1alpha1, > > > but the FIP should say that explicitly. > > > - Your "Configuration Updates" section asserts the registry is sourced > > > from Fluss, but your Open Questions section > > > correctly flags this as unresolved - I'd align the body with Open > > > Questions. For v1alpha1 a small client-side allowlist mirrored > > > from DynamicServerConfig.ALLOWED_CONFIG_KEYS is probably fine I reckon > > > > > > > > > Overall, I like the proposal. The main thing I would clarify is the > > > operator/server boundary: what the operator can safely orchestrate > today, > > > what needs a small server API, and what should be left out of v1alpha1. > > > > > > -- Anton > > > > > > чт, 7 мая 2026 г. в 09:57, Lorenzo Affetti via dev < > [email protected] > > >: > > > > > > > > Hi Michael, > > > > > > > > Thanks for the careful read. > > > > > > > > *Framework*: Java Operator SDK, not plain fabric8. JOSDK is built on > > > > fabric8 anyway, so we keep the option to drop down whether needed. > > > > > > > > *Minimum Kubernetes version*: not currently fixed in the FIP. My > > proposal > > > > is 1.29 as the floor. > > > > > > > > Reasoning for framework: > > > > > > > > For a Fluss operator that has to coordinate rolling upgrades, > scale-in > > > > safety gates, dynamic vs restart-inducing config diffs, and a > migration > > > > state machine, JOSDK's dependent-resource and workflow primitives are > > > > well-aligned. Fabric8 alone would push us toward reimplementing them > > > > ourselves. > > > > > > > > The clearest signal is Strimzi. Strimzi predates JOSDK and was built > > > > directly on fabric8—but their newer components (Access operator, > Schema > > > > Registry operator) use JOSDK. The most mature ASF distributed-system > > > > operator effectively says: if we were starting today, we'd use JOSDK. > > > > > > > > For the version floor, the constraints are JOSDK 5.x (Java 17+) and > the > > > > Kubernetes APIs the operator uses — all GA well before 1.27: CRD v1 > > (1.16), > > > > admission webhook v1 (1.16), PDB v1 (1.21), Lease v1 (1.14). > > > > The useful-but-optional StatefulSet > > persistentVolumeClaimRetentionPolicy > > > > was alpha in 1.23, beta in 1.27, GA in 1.32. > > > > Kubernetes upstream currently patches 1.31–1.33. > > > > > > > > *1.29 keeps us within a defensible distance of upstream while leaving > > room > > > > for users on enterprise distributions a release or two behind. 1.27 > is > > more > > > > permissive; 1.30+ stricter.*Open to community input on what Fluss > users > > > > actually run. > > > > > > > > I'll fold both into the FIP once we converge. > > > > > > > > Thank you! > > > > > > > > On Wed, May 6, 2026 at 3:51 PM Michael Koepf < > [email protected]> > > > > wrote: > > > > > > > > > Hi Lorenzo, > > > > > > > > > > Thanks for the FIP. I believe a dedicated Fluss Kubernetes Operator > > will > > > > > further simplify deployment and operations in large-scale > production > > > > > environments. > > > > > > > > > > I skimmed over the FIP. > > > > > > > > > > > We propose introducing a Fluss Kubernetes Operator, implemented > in > > Java > > > > > [...] > > > > > > > > > > 1. The first question that comes to my mind; are there already > > detailed > > > > > plans regarding implementation? Do you plan to use the Java > Operator > > SDK > > > > > framework (https://javaoperatorsdk.io/)? Or just the plain fabric8 > > Java > > > > > Kubernetes client (https://github.com/fabric8io/kubernetes-client > )? > > > > > > > > > > 2. What will be the minimum supported Kubernetes version? > > > > > > > > > > Looking forward to this. > > > > > > > > > > Thanks. > > > > > -- > > > > > Best, > > > > > Michael > > > > > > > > > > On 2026/05/05 09:05:45 Lorenzo Affetti via dev wrote: > > > > > > Hello community! > > > > > > > > > > > > I would like to start a discussion about FIP-41: Fluss Kubernetes > > > > > Operator. > > > > > > Here is the motivation: > > > > > > > > > > > > Fluss 0.8 introduced a Helm chart > > > > > > <https://github.com/apache/fluss/issues/779> that simplifies the > > initial > > > > > > deployment of a Fluss cluster on Kubernetes by packaging > manifests, > > > > > > configuration, and dependencies into a versioned release. While > > this is a > > > > > > good foundation, a Helm chart is fundamentally a one-shot > > templating > > > > > tool. > > > > > > It has no awareness of Fluss's runtime state and cannot react to > > > > > > operational events such as pod failures, rolling upgrades, or > > scale-in > > > > > > operations that risk data loss. > > > > > > > > > > > > Running Fluss in production on Kubernetes today requires users to > > > > > manually > > > > > > coordinate: > > > > > > > > > > > > - Safe rolling restarts of TabletServers, ensuring tablet > > leadership > > > > > is > > > > > > migrated before each pod terminates (Fluss 0.8 introduced the > > graceful > > > > > > shutdown > > > > > > < > > > > > > > https://fluss.apache.org/docs/maintenance/operations/graceful-shutdown/> > > > > > > primitive > > > > > > that makes this possible, but does not orchestrate it across > > pods) > > > > > > - Version upgrades that must sequence CoordinatorServer and > > > > > TabletServer > > > > > > updates correctly > > > > > > - Scale-in operations where a TabletServer must be drained of > > tablets > > > > > > before its pod is terminated > > > > > > - Recovery from partial failures (e.g., PVC reattachment, pod > > identity > > > > > > preservation across restarts) > > > > > > - Leveraging Fluss 0.8's dynamic configuration updates > > > > > > < > > > > > > > https://fluss.apache.org/docs/maintenance/operations/updating-configs/> > > > > > for > > > > > > keys that do not require restart, instead of triggering a > > rolling > > > > > restart > > > > > > for every config change > > > > > > > > > > > > This gap between deployment and operations is best addressed by a > > > > > *Kubernetes > > > > > > Operator* — a controller that continuously reconciles the desired > > state > > > > > of > > > > > > a FlussCluster resource against the actual state of the cluster, > > and > > > > > > executes Fluss-aware transitions safely. > > > > > > > > > > > > The Flink and Spark ecosystems have established this pattern > > successfully > > > > > > with the Apache Flink Kubernetes Operator > > > > > > <https://github.com/apache/flink-kubernetes-operator> and the > > Apache > > > > > Spark > > > > > > Kubernetes Operator < > > https://github.com/apache/spark-kubernetes-operator > > > > > >. > > > > > > Fluss should follow suit. > > > > > > > > > > > > -- > > > > > > Lorenzo Affetti > > > > > > Team Leader of Stream Storage > > > > > > [email protected] > > > > > > www.ververica.com > > > > > > ------------------------------ > > > > > > > > > > > > <https://www.ververica.com/> > > > > > > Ververica GmbH | Herzogspitalstrasse 24 | 80331 München | Germany > > > > > > > > > > > > Follow us: > > > > > > <https://www.linkedin.com/company/ververica/posts/?feedView=all> > > > > > > <https://www.youtube.com/@ververica> > > > > > > < > > > > > > > https://open.spotify.com/show/2XME9h8iBOyr6YupqM99ir?si=87b064644add42a1 > > > > > >Available > > > > > > on: < > https://aws.amazon.com/marketplace/pp/prodview-luvmqd6leha4i > > > > > > > > > < > > > > > > > > https://marketplace.microsoft.com/en-us/product/saas/ververica.vvc_managed?tab=Overview > > > > > > > > > > > > > > > > > > Pflichtangaben/Mandatory Information > > > > > > <https://www.ververica.com/mandatory-information> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Lorenzo Affetti > > > > Senior Software Engineer @ Flink Team > > > > Ververica <http://www.ververica.com> > > > > > >
