Jackie-Jiang opened a new pull request, #18931: URL: https://github.com/apache/pinot/pull/18931
## Summary Removes the long-deprecated Kafka high-level-consumer (HLC) code path and the now-dead stream config keys that only existed to support it. Pinot dropped HLC support around v0.12; the remaining classes, methods, and config keys were unused scaffolding. ### Removed - **HLC SPI**: `StreamLevelConsumer`, `PartitionLevelConsumer`, and `PartitionLevelStreamConfig` (all `@Deprecated` public SPI in `pinot-spi`). - **`StreamConsumerFactory`**: deleted the deprecated `createPartitionLevelConsumer(String, int)` and `createStreamLevelConsumer(...)` methods. `createPartitionGroupConsumer(String, PartitionGroupConsumptionStatus)` — previously a concrete default that delegated to the deleted `createPartitionLevelConsumer` — is now `abstract`. - **HLC instance metadata**: `InstanceZKMetadata` (public `pinot-common` class) and its `ZKMetadataProvider#getInstanceZKMetadata` / `#setInstanceZKMetadata` accessors, plus the now-unused `/CONFIGS/INSTANCE` PropertyStore prefix. These stored per-instance Kafka HLC group/partition maps that nothing reads post-HLC. - **Kafka**: the `KafkaStreamConfigProperties.HighLevelConsumer` constant block and the dangling `KafkaStarterUtils.KAFKA_STREAM_LEVEL_CONSUMER_CLASS_NAME` constant (pointed at a class that no longer exists). - **Dead segment code**: `MutableSegmentImpl#getMinTime` / `#getMaxTime` and the `_timeColumnName` field they required (HLC-only), and the now-orphaned `IngestionUtils#extractTimeValue` helper. - **Dead stream config keys** in example/quickstart/test configs: `stream.kafka.hlc.zk.connect.string`, `stream.kafka.hlc.bootstrap.server`, `stream.kafka.consumer.type`, and `stream.kafka.zk.broker.url`. None of these are read by `StreamConfig` anymore — the low-level consumer resolves brokers from `stream.kafka.broker.list`. `StreamConfig` stores the raw config map and does not reject unknown keys, so existing table configs that still carry these keys continue to parse unchanged. ### Backward-incompatible SPI change `StreamConsumerFactory` is the public extension point third-party stream connectors subclass, so this is a **source- and binary-incompatible SPI break for out-of-tree stream plugins**: - An external factory that only implemented the (now-deleted) `createPartitionLevelConsumer` and relied on the old default `createPartitionGroupConsumer` bridge will no longer compile, and a pre-built plugin jar run against the new SPI will throw `AbstractMethodError` at consumer creation. - External code importing `PartitionLevelConsumer`, `StreamLevelConsumer`, or `PartitionLevelStreamConfig` breaks at link time. **Migration for external stream connectors**: implement `createPartitionGroupConsumer(String, PartitionGroupConsumptionStatus)` directly (it returns a new, caller-owned `PartitionGroupConsumer` that the caller closes). All in-tree factories (Kafka 3.0/4.0, Kinesis, Pulsar) already do this, so the built-in connectors are unaffected. All built-in `StreamConsumerFactory` implementations were verified to override `createPartitionGroupConsumer`; there are no remaining references to any removed class, method, or config key anywhere in the tree. > **Note:** please apply the `backward-incompat` label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
