Hi Yang, Thanks for the detailed proposal. I created a Google Doc [1] to collect my feedback. I left the key items here for reference, please go to the Google Doc for a detailed explanation.
1. DDL: avoid dual entry points; use Schema for the API, per-index namespaced properties for SQL connectors *2. Push vs Pull replication: please reconsider push mode 3. Index Table should have a log, in WAL mode, not Arrow format 4. Visibility semantics: why is it strictly synchronous, and can it be opt-in? 5. Dynamic CREATE INDEX / DROP INDEX *6. TTL cleanup needs to cover non-auto-partitioned tables too 7. Sparse Index validation rule is incorrect as written 8. Add an end-to-end write-path diagram 9. Two-Hop lookup: document the stale-pointer pattern explicitly Among these, items (2) and (6) represent the most critical design feedback, as they may necessitate significant refactoring of the overall architecture. I look forward to your response. Best, Jark [1]: https://docs.google.com/document/d/1clt3UzBTcPCXh0DsjDiVgQBJBX_NNpCSo81dIpzz4_c/edit?usp=sharing On Tue, 14 Apr 2026 at 11:22, Yang Wang <[email protected]> wrote: > > Hi everyone, > > I'd like to start a discussion on FIP-38: Global Secondary Index. > <https://cwiki.apache.org/confluence/display/FLUSS/FIP-38%3A+Global+Secondary+Index> > > Motivation > > Today, Fluss Primary Key Tables only support lookups by primary key. In > real-time data warehouse scenarios, Flink Lookup Join frequently needs to > join dimension tables on non-primary-key columns (e.g., user_id, device_id). > Without native secondary index support, users are forced to maintain > redundant tables, introduce external index systems, or resort to full table > scans — none of which balance cost, consistency, and real-time performance > well. > > Proposal Summary > > FIP-38 introduces a native Global Secondary Index with an "Index as Table" > design: each secondary index is represented as an independent Index Table > with its own buckets, KV store, and WAL, fully reusing Fluss's existing > distributed infrastructure (Coordinator, Replica protocol, Snapshot). > > Key design points: > > - Data Model: Index Table PK = index columns + data table PK columns, > hash-distributed by index columns > - Pull-based Replication: Index Bucket Leaders pull data from Data > Bucket Leaders via a new FetchIndex RPC, with server-level request > batching > - Index Commit Horizon: Synchronous write visibility — min(all Index > Bucket HWs) — ensures data is only visible after all Index Buckets have > confirmed the write, implemented through a dual-watermark DelayedWrite > mechanism > - Two-Hop Lookup: Client → Index Table (prefix scan → PKs) → Data Table > (point get → rows), works for both partitioned and non-partitioned tables > - Partitioned Table Support: Index Table is non-partitioned (global), > solving cross-partition lookup; TTL Compaction Filter handles expired > partition cleanup > - Sparse Index: Nullable index columns support Partial Update > compatibility > > The full proposal is available here: > https://cwiki.apache.org/confluence/display/FLUSS/FIP-38%3A+Global+Secondary+Index > <https://cwiki.apache.org/confluence/display/FLUSS/FIP-38%3A+Global+Secondary+Index> > > > Looking forward to your feedback and suggestions! > > > Best regards, Yang
