Hi,

I'm checking Key_Shared feature and faced some unexpected behavior.
What happened roughly is, multiple messages with the same key go to
different consumers (consumer threads) even though Key_Shared is set.
Also, the distribution of messages to consumers is quite skewed.

Let me explain what I did, what happened, and what was my expectation
in more detail.

What I did:
1. Created a partitioned topic named
"persistent://my-tenant/my-namespace/my-topic8" (# of partitions is 8)
2. Ran a producer (the code is here.
https://github.com/feeblefakie/misc/blob/master/pulsar/src/main/java/MyProducer.java
)
  - it just produces 10000 messages with SinglePartition. each message
has a key (from "0" to "999")
3. Ran a consumer process (the code is here.
https://github.com/feeblefakie/misc/blob/master/pulsar/src/main/java/MyConsumer.java
)
  - It consumes messages with Key_Shared and 20 consumer threads set
by ClientBuilder.listenerThreads(20).

What happened:
Multiple messages with the same key go to different consumer threads.
For example, the following is a set of messages with key "100" but
some goes to b4fa8 others go to 96ee9.

#ConsumerName key
b4fa8 100
b4fa8 100
b4fa8 100
b4fa8 100
96ee9 100
96ee9 100
96ee9 100
96ee9 100
96ee9 100
96ee9 100

Also, distribution of messages are pretty skewed.

# numOfMessages consumerName
 289 15588
 346 21dcb
 138 284ca
 379 29138
 195 30276
 533 500ae
 375 5c4cb
 338 654bf
 194 699fe
 271 6c6c8
 220 73691
 152 7f5d1
 372 96ee9
 359 9b55d
 315 a22fd
 196 a752a
4243 b4fa8 <- this consumes most of the messages
 387 cc11a
 307 d0f5d
 391 ec8b8

What I expected:

First, multiple messages with the same key go to the same consumer
thread as described in below.
http://pulsar.apache.org/docs/en/concepts-messaging/#key_shared
Second, messages are almost equally balanced between consumer threads.

So, am I doing something wrong ? or my expectation is wrong ?
If my expectation is correct, It would be great if anyone can teach me
how to make it happen.

Best regards,
Hiroyuki

Reply via email to