Copilot commented on code in PR #23770: URL: https://github.com/apache/pulsar/pull/23770#discussion_r2917091133
########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation Review Comment: The document uses multiple top-level `#` headings after the title. For consistent Markdown structure (and better TOC rendering), section headings after `# PIP-398: ...` should typically be `## ...` (and deeper sections `### ...`). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals Review Comment: The document uses multiple top-level `#` headings after the title. For consistent Markdown structure (and better TOC rendering), section headings after `# PIP-398: ...` should typically be `## ...` (and deeper sections `### ...`). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. Review Comment: The bold formatting is broken due to an unmatched/interrupted `**` across lines (line 82 starts with `**`). Rephrase so the emphasis markers are balanced (e.g., keep the bold phrase on one line or close/open `**` correctly). ```suggestion (broker, namespace, topic) are `false` or `null`.** ``` ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: + +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = false, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + void removeReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Void> removeReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + void removeReplicateSubscriptionState(String topic) throws PulsarAdminException; + CompletableFuture<Void> removeReplicateSubscriptionStateAsync(String topic); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.TopicPoliciesImpl` class. + +### Consumer level + +No changes. When the consumer with `replicateSubscriptionState=true`, the subscription will be snapshot. If `false`, no +operation will be performed. + +## Public-facing Changes + +### Public API + +> **Note:** Setting `false` at namespace or topic level is accepted but not applied (enable-only semantics). Use the +`DELETE` endpoint or `remove-` command to clear the configuration instead. Review Comment: There’s a semantic inconsistency: earlier text implies `false` is a configured value (which would participate in precedence and potentially override lower levels), but later the note says `false` is 'accepted but not applied' (suggesting it behaves like 'unset'). Please clarify whether `false` is stored and affects precedence (even if it doesn’t actively disable existing subscriptions), or whether it should be treated as equivalent to `null`/not-configured and therefore should not override a lower-level `true`. ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: + +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = false, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class Review Comment: The proposed config/property name is inconsistent: the text says `replicate_subscriptions_state` (plural `subscriptions`) while the `Policies` snippet uses `replicate_subscription_state` (singular `subscription`). This will be confusing for implementers and API consumers; align the name consistently across broker/namespace/topic representations (and ensure the chosen name matches the intended config key/public API). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: + +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = false, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.Namespaces` interface: + ```java + public interface Namespaces { + void setReplicateSubscriptionState(String namespace, boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String namespace, boolean enabled); + Boolean getReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String namespace); + void removeReplicateSubscriptionState(String namespace) throws PulsarAdminException; + CompletableFuture<Void> removeReplicateSubscriptionStateAsync(String namespace); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.NamespacesImpl` class. + +### Topic level + +1. Add the field `Boolean replicateSubscriptionState` to the `org.apache.pulsar.common.policies.data.TopicPolicies` + class to enable subscription replication at the topic level: + ```java + public class TopicPolicies { + public Boolean replicateSubscriptionState; + } + ``` +2. Add the management methods to the `org.apache.pulsar.client.admin.TopicPolicies` interface: + ```java + public interface TopicPolicies { + void setReplicateSubscriptionState(String topic, boolean enabled) throws PulsarAdminException; + CompletableFuture<Void> setReplicateSubscriptionStateAsync(String topic, boolean enabled); + Boolean getReplicateSubscriptionState(String topic, boolean applied) throws PulsarAdminException; + CompletableFuture<Boolean> getReplicateSubscriptionStateAsync(String topic, boolean applied); + void removeReplicateSubscriptionState(String topic) throws PulsarAdminException; + CompletableFuture<Void> removeReplicateSubscriptionStateAsync(String topic); + } + ``` +3. Implement the management methods in the `org.apache.pulsar.client.admin.internal.TopicPoliciesImpl` class. + +### Consumer level + +No changes. When the consumer with `replicateSubscriptionState=true`, the subscription will be snapshot. If `false`, no +operation will be performed. + +## Public-facing Changes + +### Public API + +> **Note:** Setting `false` at namespace or topic level is accepted but not applied (enable-only semantics). Use the +`DELETE` endpoint or `remove-` command to clear the configuration instead. + +##### Namespace level + +- `POST /{tenant}/{namespace}/replicateSubscriptionState`: set the subscription replication configuration on the + namespace level. + - Content-Type: `application/json` + - Body: `true` or `false`. +- `GET /{tenant}/{namespace}/replicateSubscriptionState`: get subscription replication configuration on the namespace + level. + - Response: `true`, `false`, or `null` (not configured) +- `DELETE /{tenant}/{namespace}/replicateSubscriptionState`: remove the subscription replication configuration on the + namespace level. + +##### Topic level + +- `POST /{tenant}/{namespace}/{topic}/replicateSubscriptionState`: set the subscription replication configuration on + the topic level. + - Content-Type: `application/json` + - Body: `true` or `false`. +- `GET /{tenant}/{namespace}/{topic}/replicateSubscriptionState`: get subscription replication configuration on the + topic level. + - Parameters: + - `applied=true`: get the effective configuration following `Topic > Namespace > Broker` precedence. + - `applied=false`: get the topic level configuration only. + - Response: `true`, `false`, or `null` (not configured) +- `DELETE /{tenant}/{namespace}/{topic}/replicateSubscriptionState`: remove the subscription replication configuration + on the topic level. + +### CLI + +#### Namespace level + +- `pulsar-admin namespaces set-replicate-subscription-state <tenant>/<namespace> --enabled true/false` to + set the subscription replication on the namespace level. +- `pulsar-admin namespaces get-replicate-subscription-state <tenant>/<namespace>` to get the subscription + replication configuration on the namespace level. +- `pulsar-admin namespaces remove-replicate-subscription-state <tenant>/<namespace>` to remove the subscription + replication configuration on the namespace level. + +#### Topic level + +- `pulsar-admin topicPolicies set-replicate-subscription-state <tenant>/<namespace>/<topic> --enabled true/false` + to set the subscription replication on the topic level. +- `pulsar-admin topicPolicies get-replicate-subscription-state <tenant>/<namespace>/<topic>` to get the + subscription replication configuration on the topic level. +- `pulsar-admin topicPolicies remove-replicate-subscription-state <tenant>/<namespace>/<topic>` to remove the + subscription replication configuration on the topic level. + +# Security Considerations + +Both write and read operations require the necessary permissions, which already exist in Pulsar. + +## Namespace level + +- Write the subscription replication configuration: + - Required: `PolicyName.REPLICATED_SUBSCRIPTION` with `WRITE` permission. +- Read the subscription replication configuration: + - Required: `PolicyName.REPLICATED_SUBSCRIPTION` with `READ` permission. + +## Topic level + +- Write the subscription replication configuration + - Required: `TopicOperation.SET_REPLICATED_SUBSCRIPTION_STATUS` permission. +- Read the subscription replication configuration: + - Required: `TopicOperation.GET_REPLICATED_SUBSCRIPTION_STATUS` permission. + +# Backward & Forward Compatibility + +## Upgrade + +None. + +## Downgrade / Rollback + +If the broker is downgrade, users need to ensure that the subscription replication configuration is reset to maintain +the correct replication behavior by the pulsar-admin CLI or API: Review Comment: Grammar fix: 'If the broker is downgrade' should be 'If the broker is downgraded' (or 'If the broker is downgraded/rolled back'). ```suggestion If the broker is downgraded, users need to ensure that the subscription replication configuration is reset to maintain the correct replication behavior via the pulsar-admin CLI or API: ``` ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design Review Comment: The document uses multiple top-level `#` headings after the title. For consistent Markdown structure (and better TOC rendering), section headings after `# PIP-398: ...` should typically be `## ...` (and deeper sections `### ...`). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: + +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = false, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; Review Comment: The proposed config/property name is inconsistent: the text says `replicate_subscriptions_state` (plural `subscriptions`) while the `Policies` snippet uses `replicate_subscription_state` (singular `subscription`). This will be confusing for implementers and API consumers; align the name consistently across broker/namespace/topic representations (and ensure the chosen name matches the intended config key/public API). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge Review Comment: The document uses multiple top-level `#` headings after the title. For consistent Markdown structure (and better TOC rendering), section headings after `# PIP-398: ...` should typically be `## ...` (and deeper sections `### ...`). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class Review Comment: The proposed config/property name is inconsistent: the text says `replicate_subscriptions_state` (plural `subscriptions`) while the `Policies` snippet uses `replicate_subscription_state` (singular `subscription`). This will be confusing for implementers and API consumers; align the name consistently across broker/namespace/topic representations (and ensure the chosen name matches the intended config key/public API). ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. Review Comment: There’s a semantic inconsistency: earlier text implies `false` is a configured value (which would participate in precedence and potentially override lower levels), but later the note says `false` is 'accepted but not applied' (suggesting it behaves like 'unset'). Please clarify whether `false` is stored and affects precedence (even if it doesn’t actively disable existing subscriptions), or whether it should be treated as equivalent to `null`/not-configured and therefore should not override a lower-level `true`. ```suggestion The **highest level that is explicitly configured to `true` determines the effective policy**; lower levels are ignored once such a higher-level configuration exists. * **Broker, Namespace, and Topic levels** follow **enable-only semantics**: * `true` → enables subscription replication and is treated as a configured value for precedence. * `false` or `null` → are treated as **not configured**: they do **not disable** existing replicated subscriptions and do **not override** a lower-level `true`. * **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: * `true` → enables replication. * `false` → expresses that the consumer does not request replication but does **not automatically disable** replication if enabled by higher-level policies. ``` ########## pip/pip-398.md: ########## @@ -0,0 +1,291 @@ +# PIP-398: Subscription replication on the broker, namespace and topic levels + +# Background knowledge + +https://github.com/apache/pulsar/pull/4299 introduces the subscription replication feature on the consumer level: + +```java +Consumer<byte[]> consumer = pulsarClient.newConsumer().topic("topic").replicateSubscriptionState(true /* false */) + .subscriptionName("sub").subscribe(); +``` + +While this provides flexibility, it introduces overhead in managing replication for a large number of consumers. Users +need to manually enable the `replicateSubscriptionState` flag for each consumer, which can become cumbersome in +large-scale deployments. + +# Motivation + +The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover +scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are +consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration +to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration. + +# Goals + +## In Scope + +- Provide management of subscription replication at the broker, namespace and topic levels using the Pulsar Admin CLI + and API. +- Define the priority order for subscription replication configuration across different levels. + +## Out of Scope + +- Changes to the existing consumer-level `replicateSubscriptionState` configuration. +- Changes to the subscription replication mechanism itself (how states are replicated across clusters). + +# High Level Design + +This PIP introduces the `replicateSubscriptionState` configuration at the **broker, namespace, and topic levels** to +simplify the management of subscription replication. + +With this proposal, administrators can configure subscription replication at higher levels so that consumers +automatically inherit the effective configuration. + +# Enable-only Semantics & Effective Logic + +## Enable-only Semantics + +Subscription replication policies follow Pulsar's hierarchical configuration model: + +``` +Topic > Namespace > Broker +``` + +The **highest level with a configured value determines the effective policy**; lower levels are ignored once a +higher-level configuration exists. + +* **Broker, Namespace, and Topic levels** follow **enable-only semantics**: + * `true` → enables subscription replication. + * `false` or `null` → do **not disable** existing replicated subscriptions. + +* **Consumer configuration** participates in the evaluation but is **not part of the policy hierarchy**: + * `true` → enables replication. + * `false` → does **not automatically disable** replication. + * `null` → inherits the effective configuration from higher-level policies. + +Explicitly **disabling replication** for a subscription is only possible via the **Pulsar Admin API** targeting that +subscription. + +## Effective Replication Logic + +The broker determines the replication state for each subscription as follows: + +1. Determine the **effective policy** using Pulsar's precedence: `Topic > Namespace > Broker`. + +2. If the **effective policy is `true`**, the subscription is marked as replicated. + +3. If the effective policy is not `true`, the broker evaluates the **consumer configuration**: + * `true` → enables replication + * `false` or `null` → no change to the existing subscription state + +4. A subscription can be explicitly disabled via **Pulsar Admin**, but this succeeds **only if all higher-level policies + ** (broker, namespace, topic) are `false` or `null`. + + If a higher-level policy later evaluates to `true` (e.g., due to topic reload or namespace/broker policy change), + the broker will reapply the effective policy and mark the subscription as replicated again. + +### Design Consistency with Consumer-level Logic + +This design builds upon the existing **consumer-level logic**, which already follows **enable-only semantics**: + +```java +// https://github.com/apache/pulsar/blob/8798a4662300dfa3d330dbb0277073874cc2d86e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1153-L1156 +if (replicated != null && replicated && !subscription.isReplicated()) { + subscription.setReplicated(true); +} +``` + +Key points: + +* **Consumer-level default changed** from `false` to `null` ([PR #23757](https://github.com/apache/pulsar/pull/23757)). + * Combined with the multi-level policy evaluation, this ensures **compatibility across all client languages and versions**: + * Older clients defaulting to `false` + * Newer clients defaulting to `null` + * Both behave correctly with **broker, namespace, and topic-level policies**. + +# Detailed Design + +## Design & Implementation Details + +### Broker level + +Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class +to control subscription replication at the broker level: + +```java +public class ServiceConfiguration implements PulsarConfiguration { + @FieldContext( + category = CATEGORY_SERVER, + required = false, + dynamic = false, + doc = "The default value for replicating subscription state." + ) + private Boolean replicate_subscriptions_state; +} +``` + +### Namespace level + +1. Add the field `replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class + to control subscription replication at the namespace level: + ```java + public class Policies { + @SuppressWarnings("checkstyle:MemberName") + public Boolean replicate_subscription_state; Review Comment: The proposed config/property name is inconsistent: the text says `replicate_subscriptions_state` (plural `subscriptions`) while the `Policies` snippet uses `replicate_subscription_state` (singular `subscription`). This will be confusing for implementers and API consumers; align the name consistently across broker/namespace/topic representations (and ensure the chosen name matches the intended config key/public API). ```suggestion public Boolean replicate_subscriptions_state; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
