Hi Lorenzo, Thank you for the update, read through the updated proposal. A few clarifications: 1. External clients. listeners.client.serviceType offers LoadBalancer/NodePort, but advertised.listeners is derived from the headless Service DNS, i.e. in-cluster names, and a client connects directly to each bucket leader's advertised address. Is out-of-cluster client access in scope for v1alpha1? If so, how does the operator derive a per-server external advertised address? LoadBalancer opens the port, but metadata still hands clients in-cluster DNS, similar to the per-broker case Strimzi handles. If not, should the client listener expose LoadBalancer/NodePort yet? 2. Per-step upgrade gate. It waits for getClusterHealth() == GREEN, which is all-or-nothing across every bucket via computeClusterHealth. How does the gate avoid stalling until recoveryTimeout when an unrelated bucket is briefly out of ISR and keeps the whole cluster non-GREEN, even though the just-restarted server is fine? Is GREEN meant to be scoped to the restarted server here, or is the per-server recovery predicate, currently filed under the observability open question as "not required for correctness", actually needed for upgrade liveness? 3. Storage resize. StatefulSet volumeClaimTemplates are immutable, so a change to tabletServer.storage.size cannot be applied to the StatefulSet template. Existing PVCs can be expanded in place if the StorageClass allows it, but new pods keep the old size until the StatefulSet is recreated. How does the operator handle a storage-size change, or is resize out of scope for v1alpha1?
One smaller question: Server tags are keyed by server id in ZK, and ids are ordinal-reused on scale down/up. Does the operator clear PERMANENT_OFFLINE on scale-in/out, so scaling back up does not reuse an ordinal id that is still tagged and gets silently fenced from placement by CoordinatorMetadataCache.getLiveServers? Let me know what you think -- Anton чт, 11 июн. 2026 г. в 16:09, Lorenzo Affetti via dev <[email protected]>: > > Subject: Re: [DISCUSS] FIP-41: Fluss Kubernetes Operator -- revised > > Hi all, > > Thanks Anton, Yunhong, Giannis, Keith and Michael -- the review materially > improved the FIP. I've pushed a revised version; the full diff is attached for > the details. Summary of the decisions: > > Rebalance / scale-in / scale-out (Anton, Giannis) > - Corrected two wrong claims: a decommission API DOES already exist, and the > coordinator does NOT auto-rebalance. > - Split on the read/drive line: > * v1alpha1 ENFORCES the scale-in safety gate -- it refuses to remove a > TabletServer that still hosts replicas -- backed by one small read > API we > commit to (Admin.describeTabletServers(), same shape as #3400). > Evacuation itself stays a manual admin action. > * The operator does NOT drive data movement in v1alpha1 (rebalance is > cluster-wide/unscoped, no listServerTags, not min-ISR-aware). > * v1beta1 drives it behind an opt-in, once listServerTags / > decommissionServer land. > - Scale-in and scale-out share one coherent (non-driving) posture. > > Recovery & upgrades (Yunhong) > - Readiness is anchored on the new getClusterHealth() GREEN signal -- GREEN > already means "KV-restored + caught-up + in-ISR + leaders-active". > - The operator gates each rolling-upgrade step on GREEN (held for a > stabilization window). > - Split controlledShutdownTimeout from recoveryTimeout; on timeout the > upgrade > stalls (no auto-rollback). > - Kept TabletServer-first ordering, per the official upgrade docs. > > Disruption & PVCs (Keith, Giannis, Anton) > - Operator rolls pods via direct delete + GREEN gate. > - PDB defaults to maxUnavailable: 0. > - PVCs use persistentVolumeClaimRetentionPolicy and are never auto-deleted. > - Server identity is ordinal-derived and guarded by disk.properties. > - Dropped status.phase in favour of conditions only. > > Implementation (Michael) > - Java Operator SDK (built on fabric8). > - Proposed minimum Kubernetes version: 1.29. > > A few things I deliberately scoped out and would value your read on: > > - Observability (Yunhong): GREEN plus its aggregate counts are the > operator's > control contract; richer per-server / KV-restore / lag signals are > diagnosis-grade (and largely exist as JMX metrics already), so they're an > optional v1beta1+ enhancement rather than frozen here. > > - Secrets (Keith): plaintext config is a pre-existing Fluss/chart > limitation, > not operator-specific. The operator materializes config in a Secret > (defense-in-depth, admittedly cosmetic); the real fix (config > interpolation / > secret references) is tracked at the Fluss/chart level. > > - Lake tiering: an explicit v1alpha1 non-goal. > > Details and exact wording are in the attached diff. Happy to discuss any of > the > above. > > Best, > Lorenzo > > On Fri, May 29, 2026 at 9:52 AM Keith Lee <[email protected]> wrote: >> >> Hi Lorenzo. >> >> Thank you for the detailed proposal. This will improve the operational >> experience of maintaining Fluss clusters. >> >> 1. Since configurationOverrides is rendered straight into a ConfigMap, any >> credential-bearing keys would end up in plaintext. Could we add a >> sensitive-overrides path that resolves from a Secret, even if the >> non-sensitive map stays as-is for v1alpha1? This is likely to be an >> important factor in application security reviews of users considering to >> adopt Fluss operator. >> >> 2. Can you clarify how the operator will terminate/roll pods e.g. delete or >> eviction API? I believe using direct delete with pod disruption budget of >> n-1 might cause unavailability in TOCTOU scenarios as direct deletion >> bypasses >> PDB. For reference, strimzi sets maxUnavailable to zero (effectively >> minAvailable N) [1] and direct deletion, because they recognise that >> readiness probe is a weak signal of ISR status: “ >> the readiness probe doesn't guarantee that all replicas are in-sync again”. >> >> Best regards >> Keith >> >> [1] >> https://github.com/strimzi/proposals/blob/main/029-adopt-the-drain-cleaner-utility.md#disruptions >> >> On Wed, 27 May 2026 at 15:13, Giannis Polyzos <[email protected]> wrote: >> >> > Hi Lorenzo >> > And thank you for the proposal. >> > >> > I'm sharing similar thoughts to the previous comments. Overall, the >> > proposal is great, but there are a few things I think will help improve. >> > >> > Cluster rebalance is an important operation, and Fluss already supports it, >> > so I think it would be great to address: >> > 1. Scale-out without a rebalance trigger leaves clusters unbalanced, so the >> > operator must call rebalance() post scale-out and poll progress >> > 2. Scale-in with no drain path is unusable, so the operator must do >> > TEMPORARY_OFFLINE tag + rebalance + verify zero-bucket must precede replica >> > reduction >> > 3. Server ID reuse on PVC recycle corrupts ZK state, so the operator design >> > should have ID assignment; so I would like to see the PVC lifecycle >> > ownership >> > >> > I think it might also be useful to have a >> > terminationGracePeriodSeconds/preStop hook design >> > >> > For PVCs, it's also not clear to me about the model, and I think it should >> > be clear that we never auto-delete PVCs >> > >> > With the above, we can set a good foundation for the operator, and with >> > these addressed its also a +1 for me >> > >> > Best, >> > Giannis >> > >> > On Thu, May 14, 2026 at 9:51 AM Yunhong Zheng <[email protected]> wrote: >> > >> > > Hi Lorenzo, and thanks Anton for the detailed review. >> > > >> > > +1 to the overall direction. I'd like to add one more dimension to the >> > > rolling-upgrade story that I think the FIP currently under-specifies: the >> > > operator must gate progression on replica recovery on the way in, not >> > only >> > > on leader migration on the way out. >> > > >> > > Today the FIP talks about integrating with controlled shutdown, which >> > > protects the outgoing pod (leader is migrated before termination). But >> > once >> > > the new pod comes up, a Fluss process can become listener-ready within >> > > seconds, while: >> > > - log replicas may still be catching up to the leader, >> > > - PK/KV replicas may still be downloading the snapshot from remote >> > > storage and replaying the changelog up to the high-watermark, >> > > - none of the buckets hosted on that server have rejoined the ISR yet. >> > > >> > > If Pod.Ready flips to true at process-start time, a StatefulSet >> > > RollingUpdate will happily move on to the next ordinal, and the operator >> > > can find itself restarting pod N+1 while pod N is still recovering. For >> > PK >> > > tables with non-trivial RocksDB state this window can be minutes. >> > > >> > > This connects to Anton's point that "TCP probes are too weak for operator >> > > readiness". I'd push it one step further: >> > > >> > > 1. Define Pod.Ready as "fully recovered", not just "process up". >> > > Concretely: all replicas hosted on the server are LogCaughtUp && >> > > (KvRestored if PK) && InSyncWithLeader. The cleanest implementation is a >> > > Pod readiness gate (e.g. fluss.apache.org/replicas-recovered=True) >> > > flipped by the operator after polling the cluster, rather than >> > overloading >> > > the HTTP/TCP probe. >> > > >> > > 2. Make the rolling-upgrade gate a two-condition predicate: >> > > (a) the previously-restarted pod is fully recovered, AND >> > > (b) cluster-level invariants under-min-isr buckets == 0 and kv-recovering >> > > replicas == 0 held for a configurable stabilization window. >> > > >> > > 3. Freeze the observability contract here, even if drain stays in a >> > > companion FIP. Path A as written cannot actually answer "is replica >> > > recovery complete?" — replicasOnTabletServer() being internal (Anton's >> > > point) is one example, but the bigger gap is that there is no >> > > cluster-visible signal for KV restore progress or follower lag aggregated >> > > per server. Concretely I'd like the FIP to commit to exposing: >> > > - per-bucket / per-server in-sync predicate >> > > - kv.snapshot.restore.in-progress and restore.bytes-remaining >> > > - log.follower.lag-bytes / lag-time-ms >> > > - cluster-level under-min-isr-bucket-count and offline-bucket-count >> > > >> > > 4. Extend status.tabletServer.pods[] accordingly: recovered, >> > > underReplicatedBuckets, kvRestoreProgress, logCatchupLagBytes; and a >> > > cluster-level summary (underMinIsrBucketCount, offlineBucketCount). This >> > > also gives operators a SRE-grade observable surface during incidents. >> > > >> > > 5. Distinguish log-only vs PK workloads in defaults. A PK TabletServer >> > can >> > > take orders of magnitude longer to recover than a log-only one. A single >> > > perPodTimeout is misleading; consider splitting controlledShutdownTimeout >> > > from recoveryTimeout. On timeout the upgrade should stall with a Stalled >> > > condition, not auto-rollback — auto-rollback for stateful systems is >> > > dangerous and the operator should require explicit human input (e.g. an >> > > annotation) to resume or proceed. >> > > >> > > 6. Coordinator/TabletServer ordering during upgrade. With >> > > coordinator.replicas=1, there is a Coordinator-unavailable window during >> > > its own upgrade. ControlledShutdown, ISR changes and KV-standby >> > > coordination all depend on the Coordinator being up. The FIP should state >> > > that TabletServer rolling only begins after the Coordinator is upgraded >> > and >> > > has stabilized for the same window, and that each TS step re-checks >> > > Coordinator health. >> > > >> > > Otherwise +1 from me. Looking forward to the poc code. >> > > >> > > Yours, >> > > Yunhong Zheng (Swuferhong) >> > > >> > > On 2026/05/12 16:17:20 Anton Borisov wrote: >> > > > Hi Lorenzo, >> > > > >> > > > Thanks for writing this up. I like the direction. An operator is the >> > > > right next step after the Helm chart, and the proposed shape is broadly >> > > > reasonable. >> > > > >> > > > I checked the FIP against the current code and I think a few points >> > > > are worth discussing/considering: >> > > > >> > > > 1. Drain / scale-in >> > > > >> > > > The FIP says Path B needs a new drain primitive, while Path A only >> > blocks >> > > > scale-in if the target TabletServer still has replicas. >> > > > >> > > > I think we already have most of the low-level pieces: >> > > > >> > > > - ServerTag.PERMANENT_OFFLINE / TEMPORARY_OFFLINE >> > > > - AddServerTagRequest / RemoveServerTagRequest >> > > > - RebalanceRequest >> > > > - ListRebalanceProgressRequest >> > > > - RebalanceStatus >> > > > >> > > > From the rebalance code, offline-tagged servers are not just excluded >> > > > from new placements. ReplicaDistributionGoal moves replicas out of >> > them, >> > > > including followers, and the target replica count is effectively zero. >> > > > >> > > > So scale-in could be: >> > > > >> > > > AddServerTag(PERMANENT_OFFLINE) >> > > > -> Rebalance >> > > > -> wait for COMPLETED >> > > > -> scale down StatefulSet >> > > > >> > > > TEMPORARY_OFFLINE may also be usable before rolling restart. >> > > > >> > > > Path A also needs server work as written. The operator needs to know >> > > > whether a TabletServer still has replicas, but that count is internal >> > in >> > > > CoordinatorContext.replicasOnTabletServer() and is not exposed through >> > > > Admin.getServerNodes(). >> > > > >> > > > So I do not think the choice is “Path A without server work” vs “Path B >> > > > with server work”. Both need a small server/API decision. >> > > > >> > > > The important caveats are: >> > > > >> > > > - rebalance can fail if the remaining TabletServers cannot absorb the >> > > > replicas; >> > > > - I do not see min-ISR being considered by the rebalance executor, so >> > > > decommission close to min-ISR may create write-availability windows; >> > > > - tag + rebalance is currently multi-step, so the operator needs crash >> > > > recovery if it dies between the two calls; >> > > > - the operator would depend on @PublicEvolving Admin/rebalance APIs. >> > > > >> > > > Given that, I think a small wrapper may be cleaner: >> > > > >> > > > DecommissionServer(serverId) -> rebalanceId >> > > > >> > > > The server would own tag + rebalance as one operation. The operator >> > would >> > > > start it and poll progress. >> > > > >> > > > 2. Helm naming / adoption >> > > > >> > > > The FIP says operator naming matches the Helm chart to support in-place >> > > > adoption. The current chart uses fixed names such as tablet-server, >> > > > tablet-server-hs, coordinator-server and coordinator-server-hs. >> > > > >> > > > That means two Helm installs in one namespace collide. If the operator >> > > > keeps these names, v1alpha1 is effectively one FlussCluster per >> > > > namespace. >> > > > >> > > > I think the FIP should either state that constraint, or move both chart >> > > > and operator resources to cluster-prefixed names. Otherwise adoption >> > and >> > > > multi-cluster UX are pulling in different directions. >> > > > >> > > > Smaller points >> > > > >> > > > - I would not use both status.phase and conditions. Conditions are >> > enough >> > > > and avoid forcing mutually-overlapping states into one enum. >> > > > - TCP probes are too weak for operator readiness. Readiness should mean >> > > > the server is actually usable from Fluss’ point of view: registered >> > > > with the Coordinator, ZK connected, etc. >> > > > - PVC retention should use StatefulSet >> > > > persistentVolumeClaimRetentionPolicy. I would not add custom PVC >> > > > finalizer logic unless we really need it. >> > > > - Lake tiering is not mentioned. It is fine to keep it out of v1alpha1, >> > > > but the FIP should say that explicitly. >> > > > - Your "Configuration Updates" section asserts the registry is sourced >> > > > from Fluss, but your Open Questions section >> > > > correctly flags this as unresolved - I'd align the body with Open >> > > > Questions. For v1alpha1 a small client-side allowlist mirrored >> > > > from DynamicServerConfig.ALLOWED_CONFIG_KEYS is probably fine I reckon >> > > > >> > > > >> > > > Overall, I like the proposal. The main thing I would clarify is the >> > > > operator/server boundary: what the operator can safely orchestrate >> > today, >> > > > what needs a small server API, and what should be left out of v1alpha1. >> > > > >> > > > -- Anton >> > > > >> > > > чт, 7 мая 2026 г. в 09:57, Lorenzo Affetti via dev < >> > [email protected] >> > > >: >> > > > > >> > > > > Hi Michael, >> > > > > >> > > > > Thanks for the careful read. >> > > > > >> > > > > *Framework*: Java Operator SDK, not plain fabric8. JOSDK is built on >> > > > > fabric8 anyway, so we keep the option to drop down whether needed. >> > > > > >> > > > > *Minimum Kubernetes version*: not currently fixed in the FIP. My >> > > proposal >> > > > > is 1.29 as the floor. >> > > > > >> > > > > Reasoning for framework: >> > > > > >> > > > > For a Fluss operator that has to coordinate rolling upgrades, >> > scale-in >> > > > > safety gates, dynamic vs restart-inducing config diffs, and a >> > migration >> > > > > state machine, JOSDK's dependent-resource and workflow primitives are >> > > > > well-aligned. Fabric8 alone would push us toward reimplementing them >> > > > > ourselves. >> > > > > >> > > > > The clearest signal is Strimzi. Strimzi predates JOSDK and was built >> > > > > directly on fabric8—but their newer components (Access operator, >> > Schema >> > > > > Registry operator) use JOSDK. The most mature ASF distributed-system >> > > > > operator effectively says: if we were starting today, we'd use JOSDK. >> > > > > >> > > > > For the version floor, the constraints are JOSDK 5.x (Java 17+) and >> > the >> > > > > Kubernetes APIs the operator uses — all GA well before 1.27: CRD v1 >> > > (1.16), >> > > > > admission webhook v1 (1.16), PDB v1 (1.21), Lease v1 (1.14). >> > > > > The useful-but-optional StatefulSet >> > > persistentVolumeClaimRetentionPolicy >> > > > > was alpha in 1.23, beta in 1.27, GA in 1.32. >> > > > > Kubernetes upstream currently patches 1.31–1.33. >> > > > > >> > > > > *1.29 keeps us within a defensible distance of upstream while leaving >> > > room >> > > > > for users on enterprise distributions a release or two behind. 1.27 >> > is >> > > more >> > > > > permissive; 1.30+ stricter.*Open to community input on what Fluss >> > users >> > > > > actually run. >> > > > > >> > > > > I'll fold both into the FIP once we converge. >> > > > > >> > > > > Thank you! >> > > > > >> > > > > On Wed, May 6, 2026 at 3:51 PM Michael Koepf < >> > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Hi Lorenzo, >> > > > > > >> > > > > > Thanks for the FIP. I believe a dedicated Fluss Kubernetes Operator >> > > will >> > > > > > further simplify deployment and operations in large-scale >> > production >> > > > > > environments. >> > > > > > >> > > > > > I skimmed over the FIP. >> > > > > > >> > > > > > > We propose introducing a Fluss Kubernetes Operator, implemented >> > in >> > > Java >> > > > > > [...] >> > > > > > >> > > > > > 1. The first question that comes to my mind; are there already >> > > detailed >> > > > > > plans regarding implementation? Do you plan to use the Java >> > Operator >> > > SDK >> > > > > > framework (https://javaoperatorsdk.io/)? Or just the plain fabric8 >> > > Java >> > > > > > Kubernetes client (https://github.com/fabric8io/kubernetes-client >> > )? >> > > > > > >> > > > > > 2. What will be the minimum supported Kubernetes version? >> > > > > > >> > > > > > Looking forward to this. >> > > > > > >> > > > > > Thanks. >> > > > > > -- >> > > > > > Best, >> > > > > > Michael >> > > > > > >> > > > > > On 2026/05/05 09:05:45 Lorenzo Affetti via dev wrote: >> > > > > > > Hello community! >> > > > > > > >> > > > > > > I would like to start a discussion about FIP-41: Fluss Kubernetes >> > > > > > Operator. >> > > > > > > Here is the motivation: >> > > > > > > >> > > > > > > Fluss 0.8 introduced a Helm chart >> > > > > > > <https://github.com/apache/fluss/issues/779> that simplifies the >> > > initial >> > > > > > > deployment of a Fluss cluster on Kubernetes by packaging >> > manifests, >> > > > > > > configuration, and dependencies into a versioned release. While >> > > this is a >> > > > > > > good foundation, a Helm chart is fundamentally a one-shot >> > > templating >> > > > > > tool. >> > > > > > > It has no awareness of Fluss's runtime state and cannot react to >> > > > > > > operational events such as pod failures, rolling upgrades, or >> > > scale-in >> > > > > > > operations that risk data loss. >> > > > > > > >> > > > > > > Running Fluss in production on Kubernetes today requires users to >> > > > > > manually >> > > > > > > coordinate: >> > > > > > > >> > > > > > > - Safe rolling restarts of TabletServers, ensuring tablet >> > > leadership >> > > > > > is >> > > > > > > migrated before each pod terminates (Fluss 0.8 introduced the >> > > graceful >> > > > > > > shutdown >> > > > > > > < >> > > > > > >> > > https://fluss.apache.org/docs/maintenance/operations/graceful-shutdown/> >> > > > > > > primitive >> > > > > > > that makes this possible, but does not orchestrate it across >> > > pods) >> > > > > > > - Version upgrades that must sequence CoordinatorServer and >> > > > > > TabletServer >> > > > > > > updates correctly >> > > > > > > - Scale-in operations where a TabletServer must be drained of >> > > tablets >> > > > > > > before its pod is terminated >> > > > > > > - Recovery from partial failures (e.g., PVC reattachment, pod >> > > identity >> > > > > > > preservation across restarts) >> > > > > > > - Leveraging Fluss 0.8's dynamic configuration updates >> > > > > > > < >> > > > > > >> > > https://fluss.apache.org/docs/maintenance/operations/updating-configs/> >> > > > > > for >> > > > > > > keys that do not require restart, instead of triggering a >> > > rolling >> > > > > > restart >> > > > > > > for every config change >> > > > > > > >> > > > > > > This gap between deployment and operations is best addressed by a >> > > > > > *Kubernetes >> > > > > > > Operator* — a controller that continuously reconciles the desired >> > > state >> > > > > > of >> > > > > > > a FlussCluster resource against the actual state of the cluster, >> > > and >> > > > > > > executes Fluss-aware transitions safely. >> > > > > > > >> > > > > > > The Flink and Spark ecosystems have established this pattern >> > > successfully >> > > > > > > with the Apache Flink Kubernetes Operator >> > > > > > > <https://github.com/apache/flink-kubernetes-operator> and the >> > > Apache >> > > > > > Spark >> > > > > > > Kubernetes Operator < >> > > https://github.com/apache/spark-kubernetes-operator >> > > > > > >. >> > > > > > > Fluss should follow suit. >> > > > > > > >> > > > > > > -- >> > > > > > > Lorenzo Affetti >> > > > > > > Team Leader of Stream Storage >> > > > > > > [email protected] >> > > > > > > www.ververica.com >> > > > > > > ------------------------------ >> > > > > > > >> > > > > > > <https://www.ververica.com/> >> > > > > > > Ververica GmbH | Herzogspitalstrasse 24 | 80331 München | Germany >> > > > > > > >> > > > > > > Follow us: >> > > > > > > <https://www.linkedin.com/company/ververica/posts/?feedView=all> >> > > > > > > <https://www.youtube.com/@ververica> >> > > > > > > < >> > > > > > >> > > https://open.spotify.com/show/2XME9h8iBOyr6YupqM99ir?si=87b064644add42a1 >> > > > > > >Available >> > > > > > > on: < >> > https://aws.amazon.com/marketplace/pp/prodview-luvmqd6leha4i >> > > > >> > > > > > > < >> > > > > > >> > > >> > https://marketplace.microsoft.com/en-us/product/saas/ververica.vvc_managed?tab=Overview >> > > > > > > >> > > > > > > >> > > > > > > Pflichtangaben/Mandatory Information >> > > > > > > <https://www.ververica.com/mandatory-information> >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Lorenzo Affetti >> > > > > Senior Software Engineer @ Flink Team >> > > > > Ververica <http://www.ververica.com> >> > > > >> > > >> > > > > > -- > Lorenzo Affetti > Senior Software Engineer @ Flink Team > Ververica
