DanielWang2035 opened a new pull request, #17238: URL: https://github.com/apache/iotdb/pull/17238
## [Draft] Subscription: Implement IoTConsensus-based subscription ### Description Adds a consensus-based subscription path that reads directly from IoTConsensus WAL, bypassing the Pipe framework. Activated for live-mode, Tablet-format topics (`mode=live AND format≠TsFileHandler`). No changes to existing Pipe-based subscription behavior. ### How It Works 1. `IoTConsensusServerImpl.write()` offers `IndexedConsensusRequest` to registered in-memory queues (non-blocking, drop-on-full) 2. Background `prefetchLoop` thread drains pending entries, fills WAL gaps, deserializes InsertNode, converts to Tablet via `ConsensusLogToTabletConverter`, and enqueues `SubscriptionEvent` 3. Consumer polls events, commits; `ConsensusSubscriptionCommitManager` tracks per-(group, topic, region) progress with contiguous-advance semantics for at-least-once delivery 4. WAL retention extended: safe-delete index = min(replication progress, subscription progress) ### New Files (5 core + 1 interface + 1 POJO) - `ConsensusPrefetchingQueue` — dual-path (memory + WAL) prefetch queue, one per (group, topic, region) - `ConsensusLogToTabletConverter` — InsertNode → Tablet conversion with Tree/Table pattern filtering - `ConsensusSubscriptionCommitManager` — commit state persistence & recovery - `ConsensusSubscriptionSetupHandler` — setup/teardown, auto-binds new regions via `IoTConsensus.onNewPeerCreated` - `ConsensusSubscriptionBroker` — per-group broker managing multi-region queues - `ISubscriptionBroker` — common interface for pipe/consensus brokers - `SubscriptionConsensusProgress` — (searchIndex, commitIndex) POJO ### Modified Files - `SubscriptionBrokerAgent` — dual broker routing (pipe + consensus) - `IoTConsensusServerImpl` — subscription queue registration, WAL safe-delete adjustment - `CreateSubscriptionProcedure` / `DropSubscriptionProcedure` — skip Pipe for consensus topics - `SubscriptionConsumerAgent`, `ConsumerGroupMeta`, `SubscriptionBroker`, `SubscriptionEvent` ### Known Limitations - No time-based filtering (WAL lacks time index) - Seek not yet implemented - Tests are manual (TODO: migrate to IT framework) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
