GitHub user StormLord07 created a discussion: Guaranteeing reply delivery to the same instance in KeyShared subscription without per-instance topics
Hello, I’m trying to implement a request/response pattern between two programs using Pulsar, but I want to avoid: * Creating a large number of per-instance topics or subscriptions * Duplicating messages to all instances # Idea * **Program A** may have multiple instances. * All share the same KeyShared subscription on a topic. * Each instance has its own logical “key” (e.g., `key_1`, `key_2`, …) and consumes only messages routed to it by message key hash range. * When sending a request, it uses a producer and sets a message property `reply_id = key_1` to indicate where to send the response. * **Program B** also has multiple instances. * Consumes from the same request topic. * When replying, it publishes a message with `key = key_1`. --- # Problem If Program B publishes with `key = key_1`, can we guarantee that the message will be routed back to the *same instance* of Program A that originally sent the request? Assumptions: * Using KeyShared subscription mode * Key stays consistent (`key_1` maps to the same consumer) * Auto-split or sticky hash range policy is used --- # Reasoning * ConsumerExclusive: Apps starting up are **not guaranteed** to have freed the previous subscription (consumer name still considered active by the broker). This forces us to recreate new subscriptions each time, which would lead to topic/subscription bloat over time. replication tends to slow things down a lot. since sometimes the long disconnected exclusive consumers tend to stick and replicate. * If we try to use ConsumerShared, we risk the response being routed to an instance of the program that is in the process of shutting down, resulting in lost or undeliverable replies. * ConsumerFailover seems like a good idea, but I am a bit fuzzy about its working. And it seems its even worse, since the message will most certanly be lost to the old instance since it's a master at the moment. And the other problem is we can't exactly know which instance is being restarted, but that i may be wrong, I do not exactly know how they are being restarted. --- # TLDR The question is whether the KeyShared hash-range mapping is stable enough between producer and consumer keys to make this safe, or whether we still need per-instance topics/subscriptions for strict targeting. Or is there a better/suggested way? GitHub link: https://github.com/apache/pulsar/discussions/24616 ---- This is an automatically sent email for commits@pulsar.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@pulsar.apache.org
