jihoonson commented on issue #9322: Kafka Indexing Service consumer properties not respected URL: https://github.com/apache/druid/issues/9322#issuecomment-583048861 Hi @abossert, I believe it is hard-coded by design. The Kafka indexing service is designed to guarantee the exactly-once ingestion which requires Druid to track valid offsets of partitions. If some offsets are expired and automatically reset in Kafka, Druid will find some gaps between the offsets from Kafka and the ones it is aware of, which will lead to an internal error. If you want to reset automatically with the Kafka indexing service, please consider setting `resetOffsetAutomatically` and `useEarliestOffset` properly in the tuningConfig. Also, please note that resetting automatically will lead to DATA LOSS (when `useEarliestOffset = false) or DATA DUPLICATION (when `useEarliestOffset = true).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
