sollecitom opened a new issue #12224: URL: https://github.com/apache/pulsar/issues/12224
When using processors with in-memory state, it's important to know the partitions assigned and de-assigned. This allows to load and unload the state, and to recover based from a durable eventually consistent snapshot by loading it based on these partitions. At the moment, you can try to derive this information from the Message IDs, but there are 2 problems with this approach. 1. You don't know when a partition is de-assigned for a Consumer. This forces you to use a time-based cache eviction policy. 2. When you recover from a crash, you don't know the whole set of partitions assigned, meaning you're forced to load from durable storage the state for each partition when you encounter it in a MessageID. This means a less efficient startup. Ideally, you should expose something like the [ConsumerRebalanceListener](https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) in Kafka. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
