If you need exactly once semantics against your target database, the
common pattern is to store your last processed offset in that database
transactionally together with your output records, instead of committing
back to kafka. On startup you'd read the last offset from your database
and seek to that offset.
See section on external-offset-storage:
http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#external-offset-storage
You'd need to use the plainSource as the committableSource is
fundamentally for at-least once delivery:
http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#offset-storage-in-kafka
Hope that helps,
Michał
On 28/06/17 11:20, Arun wrote:
Hi,
I am using Akka Kafka Consumer.committablePartitionedSource to stream
messages from kafka and group them based on group key with
groupedWithin . Grouped records should be sink into database and then
it should able to commit offset.
The code skeleton is as following:
val consumerSettings =ConsumerSettings(system,new ByteArrayDeserializer,new
ByteArrayDeserializer)
.withBootstrapServers(kafkaServers)
.withGroupId("testclientId")
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
val kafkaSource:Source[CommittableMessage[Array[Byte], Array[Byte]], Control]=
Consumer.committablePartitionedSource(consumerSettings,
Subscriptions.topics(topicName)) .flatMapMerge(maxPartitions, _._2)
val flow=Flow[In]
.groupBy[K](maximumGroupSize, groupKeyFn)
.map(e => groupKeyFn(e) -> map(e))
.mergeSubstreams
.groupedWithin(maximumGroupSize,FiniteDuration(10, TimeUnit.SECONDS))
kafkaSource.via(flow).to(sinkToDBSaveOffSet).run
The problem what we are facing, what if sinkToDB is successful, but SaveOffSet
failed or what if system crashes before SaveOffSet is called, that means we
will have duplicate records in the database. We don't want to do query
operation before insert to find duplicate record.
Is there any way we can use any fault tolerance strategy to avoid such
duplicate message or saveOffset can become one transaction?
Thanks & Regards,
Arun
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: [email protected]
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
[email protected] <mailto:[email protected]> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
