samarthjain commented on issue #11658: URL: https://github.com/apache/druid/issues/11658#issuecomment-983320650
So I know now how we ended up with an offset that was significantly higher than the earliest offset and the reason is similar to what you mentioned, @FrankChen021. The first few versions of the supervisor were talking to Kafka Cluster A. Then, the stream was recreated on Kafka Cluster B and the spec updated accordingly. As a result offsets that Druid had stored for this stream/datasource were no longer valid. We should have stopped the supervisor, wait for all handoffs to complete and then cleared all offsets in DB before resubmitting the new spec with the updated Kafka cluster endpoint. In general, updating streams or clusters or num partitions is problematic with Druid's Kafka ingestion and invariably involves clearing offsets stored in DB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
