npawar opened a new pull request #8309: URL: https://github.com/apache/pinot/pull/8309
For https://github.com/apache/pinot/issues/8219 The root cause of issue was that the kafka sets the OffsetResetStrategy in SubscriptionState to "latest" by default. From `KafkaConsumer` ``` OffsetResetStrategy offsetResetStrategy = OffsetResetStrategy.valueOf(config.getString(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG).toUpperCase(Locale.ROOT)); this.subscriptions = new SubscriptionState(logContext, offsetResetStrategy); ``` And from `ConsumerConfig`: ``` .define(AUTO_OFFSET_RESET_CONFIG, Type.STRING, "latest", in("latest", "earliest", "none"), Importance.MEDIUM, AUTO_OFFSET_RESET_DOC) ``` As a result, when `_consumer.seek(offset)` happens to an offset that is out of range, it gets reset to `latest`. Adding consumer config to reset this to `earliest`. Have verified via tests in KafkaPartitionConsumerTest and also manually, that this doesn't in any way affect table consumption from smallest/largest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
