Hello Aruna,
if the duplication you are referring to is the duplication of the
events/records that arrive and are consumed to/from Kafka, exactly-once
semantics and transactions are what you are looking for.
Kafka is prepared (since version 0.11 IIRC) to support exactly once, it
means that events
I want to process a single message at a time to avoid duplication.
On Mon, Jul 15, 2019 at 9:45 PM Pere Urbón Bayes
wrote:
> The good question, Aruna, is why would you like to do that?
>
> -- Pere
>
> Missatge de aruna ramachandran del dia dl., 15 de
> jul. 2019 a les 14:17:
>
> > Is there a w
The good question, Aruna, is why would you like to do that?
-- Pere
Missatge de aruna ramachandran del dia dl., 15 de
jul. 2019 a les 14:17:
> Is there a way to get the Consumer to only read one message at a time and
> commit the offset after processing a single message.
>
--
Pere Urbon-Baye
Explore and set max.poll.records.
On Mon, Jul 15, 2019 at 5:47 PM aruna ramachandran
wrote:
> Is there a way to get the Consumer to only read one message at a time and
> commit the offset after processing a single message.
>
--
*Suman*
*OlaCabs*
:54 AM Sven Ludwig wrote:
> One more question:
>
> Is there a way to ask Kafka which ProducerRecord.key is mapped to which
> TopicPartition (for debugging purposes etc.)?
>
>
>
> Gesendet: Montag, 14. Januar 2019 um 13:49 Uhr
> Von: "Sven Ludwig"
> An: users@kaf
One more question:
Is there a way to ask Kafka which ProducerRecord.key is mapped to which
TopicPartition (for debugging purposes etc.)?
Gesendet: Montag, 14. Januar 2019 um 13:49 Uhr
Von: "Sven Ludwig"
An: users@kafka.apache.org
Betreff: Aw: Re: Re: Doubts in Kafka
Hi,
>> .
Uhr
Von: "Peter Levart"
An: users@kafka.apache.org, "Sven Ludwig"
Betreff: Re: Aw: Re: Doubts in Kafka
On 1/10/19 2:26 PM, Sven Ludwig wrote:
> Okay, but
>
> what if one also needs to preserve the order of messages coming from a
> particular device?
>
> With
> records. You usually retain them for enough time so you don't loose them
> before processing them + some safety time...
>
> Regards, Peter
>
> >
> > Sven
> >
> >
> > Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr
> > Von: "Pete
processed
records. You usually retain them for enough time so you don't loose them
before processing them + some safety time...
Regards, Peter
Sven
Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr
Von: "Peter Levart"
An: users@kafka.apache.org, "aruna ramachandran"
na ramachandran"
Betreff: Re: Doubts in Kafka
Hi Aruna,
On 1/10/19 8:19 AM, aruna ramachandran wrote:
> I am using keyed partitions with 1000 partitions, so I need to create 1000
> consumers because consumers groups and re balancing concepts is not worked
> in the case of manually as
Hi Aruna,
On 1/10/19 8:19 AM, aruna ramachandran wrote:
I am using keyed partitions with 1000 partitions, so I need to create 1000
consumers because consumers groups and re balancing concepts is not worked
in the case of manually assigned consumers.Is there any replacement for the
above problem.
Thanks for the solution please help me to try out the keyed partitioning
with 1000 partitions.And also give suggestions to try Kafka with node js.
On Tue, 8 Jan 2019, 9:53 pm Todd Palino OK, in that case you’ll want to do something like use the sensor ID as the
> key of the message. This will ass
On 08.01.2019 17:11, aruna ramachandran wrote:
> I need to process single sensor messages in serial (order of messages
> should not be changed)at the same time I have to process 1 sensors
> messages in parallel please help me to configure the topics and partitions.
>
If you want to process e
Yes, As Todd said you have to use some id as the key to partition.
The rebalancing will be an over head and if you increase the partitions
later you will lose the order.
you can go through
https://anirudhbhatnagar.com/2016/08/22/achieving-order-guarnetee-in-kafka-with-partitioning/
for more unders
OK, in that case you’ll want to do something like use the sensor ID as the
key of the message. This will assure that every message for that sensor ID
ends up in the same partition (which will assure strict ordering of
messages for that sensor ID).
Then you can create a number of partitions to get
I need to process single sensor messages in serial (order of messages
should not be changed)at the same time I have to process 1 sensors
messages in parallel please help me to configure the topics and partitions.
On Tue, Jan 8, 2019 at 9:19 PM Todd Palino wrote:
> I think you’ll need to expa
I think you’ll need to expand a little more here and explain what you mean
by processing them in parallel. Nearly by definition, parallelization and
strict ordering are mutually exclusive concepts.
-Todd
On Tue, Jan 8, 2019 at 10:40 AM aruna ramachandran
wrote:
> I need to process the 1 sen
17 matches
Mail list logo
