nodece commented on code in PR #23770: URL: https://github.com/apache/pulsar/pull/23770#discussion_r2298445009
########## pip/pip-398.md: ########## @@ -0,0 +1,253 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(false/*true*/) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +The PIP aims to provide management of subscription replication at the broker, namespace and topic levels using the +Pulsar Admin CLI and API. + +# High Level Design + +Introduces the `replicateSubscriptionState` configuration to enabling subscription replication on the broker, namespace +and topic levels, when enabled, all consumers under the broker/namespace/topic will automatically replicate their +subscription states to remote clusters. + +The priority for the subscription replication configuration is as follows: + +- consumer level > topic level > namespace level > broker level. +- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic, namespace, and broker levels are + ignored. +- If set at the topic level, the namespace-level configuration is ignored. +- If set at the namespace level, the broker-level configuration is ignored. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = true, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, Boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, Boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + } Review Comment: We have a `pulsar-admin topics set-replicated-subscription-status` command, the current implementation uses Boolean for similar that, so I followed the same design for consistency and to avoid introducing a new pattern in the CLI/API. The meaning is: - null → remove configuration(I can add a remove api instead of this) - true → enable - false → disable (**but true has priority if both are set in the different levels**) Using an enum would make the meaning explicit, but it introduces additional complexity. ########## pip/pip-398.md: ########## @@ -0,0 +1,253 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(false/*true*/) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +The PIP aims to provide management of subscription replication at the broker, namespace and topic levels using the +Pulsar Admin CLI and API. + +# High Level Design + +Introduces the `replicateSubscriptionState` configuration to enabling subscription replication on the broker, namespace +and topic levels, when enabled, all consumers under the broker/namespace/topic will automatically replicate their +subscription states to remote clusters. + +The priority for the subscription replication configuration is as follows: + +- consumer level > topic level > namespace level > broker level. +- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic, namespace, and broker levels are + ignored. +- If set at the topic level, the namespace-level configuration is ignored. +- If set at the namespace level, the broker-level configuration is ignored. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = true, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, Boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, Boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, Boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + } Review Comment: @lhotari -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
