Thanks Michał. On Wednesday, June 28, 2017 at 4:15:30 PM UTC+5:30, Michal Borowiecki wrote: > > If you need exactly once semantics against your target database, the > common pattern is to store your last processed offset in that database > transactionally together with your output records, instead of committing > back to kafka. On startup you'd read the last offset from your database and > seek to that offset. > > See section on external-offset-storage: > http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#external-offset-storage > > You'd need to use the plainSource as the committableSource is > fundamentally for at-least once delivery: > > > http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#offset-storage-in-kafka > > Hope that helps, > > Michał > > On 28/06/17 11:20, Arun wrote: > > Hi, > > I am using Akka Kafka Consumer.committablePartitionedSource to stream > messages from kafka and group them based on group key with groupedWithin . > Grouped records should be sink into database and then it should able to > commit offset. > > The code skeleton is as following: > > val consumerSettings = ConsumerSettings(system, new ByteArrayDeserializer, > new ByteArrayDeserializer) > .withBootstrapServers(kafkaServers) > .withGroupId("testclientId") > .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") > val kafkaSource:Source[CommittableMessage[Array[Byte], Array[Byte]], Control]= > > Consumer.committablePartitionedSource(consumerSettings, > Subscriptions.topics(topicName)) .flatMapMerge(maxPartitions, _._2) > > val flow=Flow[In] > > .groupBy[K](maximumGroupSize, groupKeyFn) > .map(e => groupKeyFn(e) -> map(e)) > .mergeSubstreams > .groupedWithin(maximumGroupSize, FiniteDuration(10, TimeUnit.SECONDS)) > > kafkaSource.via(flow).to(sinkToDBSaveOffSet).run > > The problem what we are facing, what if sinkToDB is successful, but > SaveOffSet failed or what if system crashes before SaveOffSet is called, that > means we will have duplicate records in the database. We don't want to do > query operation before insert to find duplicate record. > > Is there any way we can use any fault tolerance strategy to avoid such > duplicate message or saveOffset can become one transaction? > > Thanks & Regards, > > Arun > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > -- > <http://www.openbet.com/> Michal Borowiecki > Senior Software Engineer L4 > T: +44 208 742 1600 > > > +44 203 249 8448 > > > > E: [email protected] <javascript:> > W: www.openbet.com > OpenBet Ltd > > Chiswick Park Building 9 > > 566 Chiswick High Rd > > London > > W4 5XT > > UK > <https://www.openbet.com/email_promo> > This message is confidential and intended only for the addressee. If you > have received this message in error, please immediately notify the > [email protected] <javascript:> and delete it from your system as well as > any copies. The content of e-mails as well as traffic data may be monitored > by OpenBet for employment and security purposes. To protect the environment > please do not print this e-mail unless necessary. OpenBet Ltd. Registered > Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, > United Kingdom. A company registered in England and Wales. Registered no. > 3134634. VAT no. GB927523612 >
-- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
