lhotari commented on code in PR #23770: URL: https://github.com/apache/pulsar/pull/23770#discussion_r2905108272
########## pip/pip-398.md: ########## @@ -0,0 +1,253 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(false/*true*/) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +The PIP aims to provide management of subscription replication at the broker, namespace and topic levels using the +Pulsar Admin CLI and API. + +# High Level Design + +Introduces the `replicateSubscriptionState` configuration to enabling subscription replication on the broker, namespace +and topic levels, when enabled, all consumers under the broker/namespace/topic will automatically replicate their +subscription states to remote clusters. + +The priority for the subscription replication configuration is as follows: + +- consumer level > topic level > namespace level > broker level. +- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic, namespace, and broker levels are + ignored. +- If set at the topic level, the namespace-level configuration is ignored. +- If set at the namespace level, the broker-level configuration is ignored. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = true, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, Boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, Boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + } Review Comment: > Subscription replication is enabled if: > > * The consumer sets true > * The first non-null policy found in the following order is true: topic → namespace → broker. Once a policy is found (even if false), lower levels are ignored. Ok. I was thinking about a scenario where there would be a need to ignore and override any setting on the consumer. Even if the consumer would enable replication, it would be possible to turn replication off with policies. With a `Boolean` this wouldn't be possible. Perhaps we don't need to support overriding the client side setting in this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
