hongkunxu opened a new issue, #20219: URL: https://github.com/apache/pinot/issues/20219
# [Feature] Add Kafka replicate mode for append-only realtime tables ## Background Currently, Pinot's Kafka LLC ingestion assigns a consuming segment to multiple Servers. Each replica independently reads the same Kafka partition, decodes and transforms messages, and builds its own mutable segment. This provides multiple queryable copies of consuming data, but couples the replication factor with ingestion cost. Increasing replication also increases duplicate Kafka reads, ingestion CPU, and mutable-segment memory usage—even when additional replicas are primarily needed for querying completed segments. The existing `DOWNLOAD` completion mode can reduce duplicate final segment construction, but does not eliminate the preceding duplicate consumption and realtime indexing. ## Proposed Behavior Introduce an opt-in `replicateMode` configuration that separates Kafka consumption from completed-segment replication: 1. During normal operation, one leader Server consumes and indexes each Kafka partition. 2. The consuming segment remains queryable on its leader. 3. After the segment is built and successfully committed, follower Servers download and load the same immutable segment. 4. Completed segments retain the configured replication factor, including the leader's copy, and loaded replicas can serve queries. Different partitions can have different leaders, preserving ingestion parallelism across the table. The feature should reuse Pinot's segment completion, segment loading, and IdealState/ExternalView-based routing mechanisms. ## Benefits - **Reduce duplicate Kafka reads:** avoid fetching the same input independently on every replica. - **Reduce ingestion CPU:** execute decoding, filtering, transformations, and realtime indexing once per partition during normal operation. - **Reduce mutable-segment memory:** avoid maintaining equivalent mutable segments on multiple Servers. - **Separate query replication from ingestion work:** retain replicated storage and query capacity for completed segments without requiring every replica to consume Kafka. For example, with three replicas, source-side reads could decrease from approximately three copies of the input to one during normal operation. This does not imply an equivalent reduction in total CPU or network usage: immutable-segment distribution still consumes resources, and actual savings require measurement. ## Initial Scope and Trade-offs - Support Kafka LLC ingestion for append-only realtime tables. - Configure through `ingestionConfig.streamIngestionConfig.replicateMode`, defaulting to `false` to preserve existing behavior. - Exclude full upsert, partial upsert, and dedup tables from initial support. Their additional mutable state requires separate replication and recovery designs. - Followers do not hold a live copy of the consuming segment. Leader failure therefore requires Kafka replay and can temporarily reduce query freshness and completeness. - Queries against consuming data concentrate on the leader; completed-segment queries can use loaded replicas. - Evaluate ingestion savings alongside segment-distribution overhead, query performance, and recovery costs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
