Well it definitely gives me something to move ahead with.

I am however puzzled how I could observe a really even distribution over
the partitions when specifying `PARTITIONER_CLASS_CONFIG`, whereas when I
remove it the same set of test messages are written to only one partition.

Thanks
Mikkel

--


On Fri, Nov 15, 2019 at 12:22 AM Matthias J. Sax <matth...@confluent.io>
wrote:

> In Kafka Streams the producer config `PARTITIONER_CLASS_CONFIG` does not
> take effect, because Kafka Streams computes and set partition numbers
> explicitly and thus the producer does never use the partitioner to
> compute a partition, but accepts whatever Kafka Streams specifies on
> each `ProducerRecord`.
>
> If you want to change the partitioning strategy, you need to implement a
> custom `StreamPartitioner` and pass it into the corresponding methods.
> For example, `to()` or `through()`.
>
> Hope this helps.
>
>
> -Matthias
>
> On 11/14/19 9:51 AM, Mikkel Gadegaard wrote:
> > I've set up a POC using KafkaStreams with microservices consuming and
> > producing from/to topics. In the beginning I hadn't thought about
> > partition strategy, and so I was using the DefaultPartitioner for
> producer
> > partition assignments. My messages have keys (I use these for
> > forking/joining), and the keys are time based UUIDs, this causes some
> > rather uneven distribution on my topics. I looked around google and
> > stumbled on KIP-369 (Alternative Partitioner to Support "Always
> > Round-Robin" Selection) and figured that would be what I needed, so since
> > 2.4 isn't out yet I borrowed the class from the PR on github, added it to
> > my project and added the property to my config, like so:
> >
> > streamProperties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,
> > RoundRobinPartitioner.class.getCanonicalName());
> >
> >
> > And the round robin strategy works on a newly added topic, spreading
> > messages evenly over 4 partitions. But, and I'm finally getting to my
> > question, it doesn't seem to have any effect on existing topics, in other
> > words, it seems to be continuing to use the DefaultPartitioner for topics
> > created before I added the RoundRobinPartioner class to my
> > project/properties.
> >
> > Is it me that just hasn't understood that it is impossible to change
> > strategy for an existing partition or do I have to do something specific
> > apart from re-deploying the Microservice containing the producer?
> >
> > Thanks
> > Mikkel
> >
> > --
> >
>
>

Reply via email to