Re: is there a way to make sure two consumers receive the same message from the broker?

:) On Tue, Nov 8, 2016 at 1:37 AM, AmirHossein Roozbahany wrote: > Excuse me this part was non-sense: if the latest update to a document in > es always win in Cassandra's LWW, they will "eventually" "converge". > > From: AmirHossein

RE: is there a way to make sure two consumers receive the same message from the broker?

Excuse me this part was non-sense: if the latest update to a document in es always win in Cassandra's LWW, they will "eventually" "converge". From: AmirHossein Roozbahany Sent: ‎11/‎8/‎2016 8:16 AM To:

RE: is there a way to make sure two consumers receive the same message from the broker?

Generally Cassandra itself is not consistent enough, even with quorum read-writes, say one of the writes fail, the nodes who received the data won't roll back and it might lead to dirty reads which in turn makes roll back logic tricky (if not impossible). You can use linearizable writes but if

Re: is there a way to make sure two consumers receive the same message from the broker?

And there is this https://github.com/vroyer/elassandra which is still under active development and not sure how they plan to keep up with Apache Cassandra moving forward. On Mon, Nov 7, 2016 at 9:36 AM, kant kodali wrote: > Fixing typo's > > Hi Tauzell, > > Yeah our users

Re: is there a way to make sure two consumers receive the same message from the broker?

Fixing typo's Hi Tauzell, Yeah our users want to query, do aggregations on Elastic Search directly and we cannot have inconsistent data because say the writes didn't make it into Cassandra but made it to Elastic search then a simple aggregations like count will lead to a wrong answer but again

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Tauzell, Yeah our users want to query, do aggregations on Elastic Search directly and we cannot have inconsistent data because say the writes didn't make it into Cassandra but made it to Elastic search then a simple aggregations like count will lead to a wrong answer but again as @Hans

RE: is there a way to make sure two consumers receive the same message from the broker?

Here is a scenario where this could be useful: Add the kafka offset as a field on the record in both Cassandra and Elasticsearch Now when you get search results from Elastic search and look up details in Cassandra you can know if they come from the same kafka record. If you can use the

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi AmitHossein, I still don't see how that guarantees consistency at any given time. other words how do I know at time X the data in Cassandra and ES are the same. Thanks On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany wrote: > Hi > > Can you use elasticsearch

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Hans, We currently do #2 and thats is quite slow so yeah In thoery #1 is probably a better choice although its not quite what we want since it doesn't guarantee consistency at any given time as you have already pointed out. Thanks a lot for the response! kant On Mon, Nov 7, 2016 at 6:31 AM,

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Can you use elasticsearch _version field as cassandra's writetime?(_version is strictly increasing, cassandra uses writetime for applying LWW, so last write in elasticsearch will always win) It needs no transaction and makes databases convergent. From:

Re: is there a way to make sure two consumers receive the same message from the broker?

I don't believe that either of your two storage systems support distributed atomic transactions. You are just going to have to do one of the following: 1) update them separately (in parallel) and be aware that their committed offsets may be slightly different at certain points in time 2) update

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Hans, The two storages we use are Cassandra and Elastic search and they are on the same datacenter for now. The Programming Language we use is Java and OS would be Ubuntu or CentOS. We get messages in JSON format so we insert into Elastic Search directly and for Cassandra we transform JSON

Re: is there a way to make sure two consumers receive the same message from the broker?

That depends on what system you are using for storage. Are you talking about databases, file systems, or perhaps some other form of storage? What is the rate and size and format of the updates? Are the two storage systems in the same datacenter or separated by a WAN? What programming language

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi! Thanks. any pointers on how to do that? On Sun, Nov 6, 2016 at 2:32 PM, Tauzell, Dave wrote: > You should have one consumer pull the message and submit the data to each > storage using an XA transaction. > > > On Nov 5, 2016, at 19:49, kant kodali

Re: is there a way to make sure two consumers receive the same message from the broker?

You should have one consumer pull the message and submit the data to each storage using an XA transaction. > On Nov 5, 2016, at 19:49, kant kodali wrote: > > yes this problem can definetly be approached in many ways but given the > hard constraints by our clients we don't

Re: is there a way to make sure two consumers receive the same message from the broker?

yes this problem can definetly be approached in many ways but given the hard constraints by our clients we don't seem to have many options. so the problem is we have to keep two storages systems in sync all the time. so whatever the data that is storage 1 should also be in storage 2 at any given

Re: is there a way to make sure two consumers receive the same message from the broker?

Yes exactly. If consumer 1 gets message with offset 17 then it can write that offset into an external storage that consumer 2 can also check to ensure that it keeps in sync with consumer 1. Just curious though why you would need to do this? What is the use case because there may be a better

Re: is there a way to make sure two consumers receive the same message from the broker?

I am new to Kafka and reading this statement "write consumer 1 and consumer 2 to share a common external offset storage" I can interpret it many ways but my best guess is as follows. Are you saying write the current offset of each consumer to a common external storage? On Sat, Nov 5, 2016 at

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Hans, What do you mean by "write consumer 1 and consumer 2 to share a common external offset storage" ? can you please elaborate a bit more. Thanks! On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen wrote: > There is no built in mechanism to do this in Apache Kafka but if

Re: is there a way to make sure two consumers receive the same message from the broker?

There is no built in mechanism to do this in Apache Kafka but if you can write consumer 1 and consumer 2 to share a common external offset storage then you may be able to build the functionality you seek. -hans > On Nov 5, 2016, at 3:55 PM, kant kodali wrote: > > Sorry

Re: is there a way to make sure two consumers receive the same message from the broker?

Sorry there is a typo. here is a restatement. Is there a way to make sure two consumers receive the same message from the kafka broker in a atomic way? such that if consumer 1 gets a message consumer 2 should also get that message and if consumer 1 fails for whatever reason consumer 2 should