Hi Cheng, Thank you for driving this excellent work! Your FIP document shows great thought and initiative. I've gone through it and have some questions and suggestions that I hope can further enhance this valuable contribution.
1、Regarding the Title, I believe we could consider changing it to "Support full scan in batch mode for PrimaryKey Table". The term "Streaming" might cause confusion with Flink's streaming/batch modes, and this revised title would provide better clarity. 2、In the Motivation section, I think there are two particularly important benefits worth highlighting: (1) OLAP engines will be able to perform full snapshot reads on Fluss primary-key tables. (2) This approach can replace the current KvSnapshotBatchScanner, allowing the Fluss client to eliminate its RocksDB dependency entirely. 3、Concerning the Proposed Changes, could you clarify when exactly the client creates a KV snapshot on the server side, and when we send the bucket_scan_req? Let me share my thinking on this: When Flink attempts to read from a PrimaryKey table, the FlinkSourceEnumerator in the JobMaster generates HybridSnapshotLogSplit and dispatches them to SplitReaders running on the TaskManager. The JobMaster doesn't actually read data—it merely defines and manages the splits. Therefore, we need to ensure the JM has sufficient information to determine the boundary of the KV snapshot and the startOffset of the LogSplit. I suggest we explicitly create a snapshot (or as you've termed it, a new_scan_request) on the server side. This way, the FlinkSourceEnumerator can use it to define a HybridSnapshotLogSplit, and the SplitReaders can perform pollBatch operations on this snapshot (which would be bound to the specified scanner_id). 4、 Could you elaborate on why the new KvBatchScanner isn't reusable? What's the reasoning behind this limitation? (I believe RocksDB iterators do support the seekToFirst operation.) If a TaskManager fails over before a checkpoint, rescanning an existing snapshot seems like a natural requirement. 5、I think it would be beneficial to include some detailed design aspects regarding Flink's integration with the new BatchScanner. Overall, this is a solid foundation for an important enhancement. Looking forward to discussing these points further! Best regards, Yang Wang Cheng <[email protected]> 于2025年10月22日周三 17:09写道： > Hi all, > > > As of v0.8, Fluss only supports KV snapshot batch scan and limit KV batch > scan. The former approach is constrained by snapshot availability and > remote storage performance, while the later one is only applicable to > queries with LIMIT clause and risks high memory pressure. > > > To address those limitations, Giannis Polyzos and I are writing to propose > FIP-17: a general-purpose streaming KV scan for Fluss [1]. > > > Any feedback and suggestions on this proposal are welcome! > > > [1]: > https://cwiki.apache.org/confluence/display/FLUSS/FIP-17+Streaming+KV+Scan+RPC > > Regards, > Cheng > > > >
