Re: Dynamic Partitioning

how many partitions there are. > > Ryanne > > On Tue, Jan 8, 2019, 8:57 AM marimuthu eee wrote: > > > Hi Ryanne Dolan, > > > > My requirement is to process notification messages in parallel.So i have > > an idea to pick kafka as a messaging system.But i need

Re: Dynamic Partitioning

cation messages in parallel.So i have > an idea to pick kafka as a messaging system.But i need dynamic partitioning* > to handle individual user notification messages in serial manner*.Is it > possible to create dynamic partition creation for a particular topic. > > > > On Tue, Ja

Re: Dynamic Partitioning

Hi Ryanne Dolan, My requirement is to process notification messages in parallel.So i have an idea to pick kafka as a messaging system.But i need dynamic partitioning* to handle individual user notification messages in serial manner*.Is it possible to create dynamic partition creation

Re: Dynamic Partitioning

Can you explain why you think you need dynamic partitioning? Maybe we can help you design without it. Ryanne On Tue, Jan 8, 2019, 4:04 AM marimuthu eee Hi, > > My requirement is to process notification messages in parallel.So i have an > idea to pick kafka as a messaging system.B

Re: Dynamic Partitioning

wrote: > Hi, > > My requirement is to process notification messages in parallel.So i have an > idea to pick kafka as a messaging system.But i need dynamic partitioning to > handle individual user notification messages.Is it possible to create > dynamic partition creation for

Dynamic Partitioning

Hi, My requirement is to process notification messages in parallel.So i have an idea to pick kafka as a messaging system.But i need dynamic partitioning to handle individual user notification messages.Is it possible to create dynamic partition creation for a particular topic.

Re: Kafka Streams dynamic partitioning

> I think this should be ' pick number of partitions that matches max number > of possible keys in stream to be partitioned '. > At least in my usecase , in which I am trying to partition stream by key > and make windowed aggregations, if there are less number of topic > partitions than possible

Re: Kafka Streams dynamic partitioning

Thanks, I got the point. That solves my problem. On Wed, Oct 5, 2016 at 10:58 PM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hi, > > even if you have more distinct keys than partitions (ie, different key > go to the same partition),

Re: Kafka Streams dynamic partitioning

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, even if you have more distinct keys than partitions (ie, different key go to the same partition), if you do "aggregate by key" Streams will automatically separate the keys and compute an aggregate per key. Thus, you do not need to worry about

Re: Kafka Streams dynamic partitioning

Hi, @Ali IMO, Yes. That is the job of kafka server to assign kafka instances partition(s) to process. Each instance can process more than one partition but one partition cannot be processed by more than one instance. @Michael, Thanks for reply. >Rather, pick the number of partitions in a way

Re: Kafka Streams dynamic partitioning

> It's often a good idea to over-partition your topics. For example, even if today 10 machines (and thus 10 partitions) would be sufficient, pick a higher number of partitions (say, 50) so you have some wiggle room to add more machines (11...50) later if need be. If you create e.g 30 partitions,

Re: Kafka Streams dynamic partitioning

> So, in this case I should know the max number of possible keys so that > I can create that number of partitions. Assuming I understand your original question correctly, then you would not need to do/know this. Rather, pick the number of partitions in a way that matches your needs to process

Re: Kafka Streams dynamic partitioning

Hi Guozhang, So, in this case I should know the max number of possible keys so that I can create that number of partitions. Thanks Adrienne On Wed, Oct 5, 2016 at 1:00 AM, Guozhang Wang wrote: > By default the partitioner will use murmur hash on the key and mode on >

Re: Kafka Streams dynamic partitioning

By default the partitioner will use murmur hash on the key and mode on current num.partitions to determine which partitions to go to, so records with the same key will be assigned to the same partition. Would that be OK for your case? Guozhang On Tue, Oct 4, 2016 at 3:00 PM, Adrienne Kole

Kafka Streams dynamic partitioning

Hi, >From Streams documentation, I can see that each Streams instance is processing data independently (from other instances), reads from topic partition(s) and writes to specified topic. So here, the partitions of topic should be determined beforehand and should remain static. In my usecase I

Dynamic partitioning

Hi all, My understanding is that with 0.8.1.x you can manually change the number of partitions on the broker, and this change is going to be picked up by the producers and consumers (high level). kafka-topics.sh --alter --zookeeper zk.net:2181/stream --topic test --partitions 3 Is that the

Re: Dynamic partitioning

That command will change how many partitions the topic has. What you are looking for I think is https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool which allows you to change what partitions are running on which replicas and which replicas