Michael, It depends on the semantics you want to get. About retries in general, as long as a producer retries internally, you would not even notice. Only after retries are exhausted, an exception is thrown.
Kafka Streams allows you to implement a handler for this (cf https://kafka.apache.org/11/documentation/streams/developer-guide/config-streams.html#default-production-exception-handler) that allows you to react to the error as you wish. You can either use a provided handler or implement a custom one. You can either skip over a record or let Kafka Streams stop processing. It might make sense to write the record to a "retry topic", but it depends on the error. If the whole cluster is down, of course you cannot write to the retry topic either. If the output topic is currently under-repliated and does not allow for new write, the "retry topic" might be available thought. For exactly-once, producer retries are set to MAX_VALUE and thus the application would re-try practically forever. -Matthias On 6/16/18 1:52 PM, Michael Eugene wrote: > Hi I am trying to understand when to retry sending messages to topics and > when to start trying to send to "retry" topics. The scenario is basically > > 1. A KafkaStreams application is consuming from a topic and sending to a > topic. The "retries" is set at the default of 10. > > 2a. After 10 retries, does it make sense to then try to publish to another > "retry topic"? > 2a1. What mechanism is there to know its the 10th retry, and to then start > sending to a "retry topic" after the 10th? > > 2b. Or after 10 retries - for that message if its not successful its just > done. Since there is no real difference between sending to a "retry topic" > and sending to a non-retry topic, why not just set retries to a high level > (like 100). > > 3. On an implementation level (Ive read the kafka docs, i find it a bit high > level) can someone throw a nugget out there about how exactly-once semantics > would erase the need for an "retry topic"? > > If u have time to answer any part of the above question, thank you in advance. > > > > > > Get Outlook for Android<https://aka.ms/ghei36> > >
signature.asc
Description: OpenPGP digital signature
