devinbost edited a comment on issue #6151: Intermittent 500's from Admin CLI and Zookeeper exceptions in broker logs URL: https://github.com/apache/pulsar/issues/6151#issuecomment-579071183 There might be a problem with how the clusters are getting replicated. There are two local clusters in this configuration and one global cluster. Only one of the local clusters is replicating to the global cluster. This was on the global zookeeper before removing pulsar-cluster2: ``` [zk: 172.20.0.11:2184(CONNECTED) 11] get /admin/policies/public/functions { "auth_policies": { "namespace_auth": {}, "destination_auth": {}, "subscription_auth_roles": {} }, "replication_clusters": ["pulsar-cluster2", "pulsar-cluster1"], "bundles": { "boundaries": ["0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff"], "numBundles": 4 }, "backlog_quota_map": {}, "clusterDispatchRate": {}, "subscriptionDispatchRate": {}, "clusterSubscribeRate": {}, "latency_stats_sample_rate": {}, "message_ttl_in_seconds": 0, "retention_policies": { "retentionTimeInMinutes": -1, "retentionSizeInMB": -1 }, "deleted": false, "encryption_required": false, "subscription_auth_mode": "None", "max_producers_per_topic": 0, "max_consumers_per_topic": 0, "max_consumers_per_subscription": 0, "compaction_threshold": 0, "offload_threshold": -1, "schema_auto_update_compatibility_strategy": "Full", "schema_validation_enforced": false } cZxid = 0x60000023e ctime = Fri Apr 12 00:35:39 UTC 2019 mZxid = 0x69500469ace mtime = Mon Jan 27 18:24:30 UTC 2020 pZxid = 0x60000023e cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 793 numChildren = 0 ``` Global only has /admin/* It doesn't have /namespace/ However, geo-replication hasn't been configured yet on any of the namespaces. ``` [zk: 172.20.0.11:2184(CONNECTED) 3] ls /admin/ policies local-policies configuration clusters ``` ``` [zk: 172.20.0.11:2184(CONNECTED) 3] ls /admin/clusters [pulsar-cluster2, global, pulsar-cluster1] ``` This is present on the global zookeeper (formatted to make reading easier) after removing pulsar-cluster2: ``` [zk: 172.20.0.11:2184(CONNECTED) 13] get /admin/policies/public/functions { "auth_policies": { "namespace_auth": {}, "destination_auth": {}, "subscription_auth_roles": {} }, "replication_clusters": ["pulsar-cluster1"], "bundles": { "boundaries": ["0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff"], "numBundles": 4 }, "backlog_quota_map": { "destination_storage": { "limit": 10737418240, "policy": "producer_request_hold" } }, "clusterDispatchRate": { "pulsar-cluster1": { "dispatchThrottlingRateInMsg": 0, "dispatchThrottlingRateInByte": 0, "ratePeriodInSecond": 1 } }, "subscriptionDispatchRate": { "pulsar-cluster1": { "dispatchThrottlingRateInMsg": 0, "dispatchThrottlingRateInByte": 0, "ratePeriodInSecond": 1 } }, "clusterSubscribeRate": { "pulsar-cluster1": { "subscribeThrottlingRatePerConsumer": 0, "ratePeriodInSecond": 30 } }, "latency_stats_sample_rate": {}, "message_ttl_in_seconds": 0, "retention_policies": { "retentionTimeInMinutes": -1, "retentionSizeInMB": -1 }, "deleted": false, "encryption_required": false, "subscription_auth_mode": "None", "max_producers_per_topic": 0, "max_consumers_per_topic": 0, "max_consumers_per_subscription": 0, "compaction_threshold": 0, "offload_threshold": -1, "schema_auto_update_compatibility_strategy": "Full", "schema_validation_enforced": false } cZxid = 0x60000023e ctime = Fri Apr 12 00:35:39 UTC 2019 mZxid = 0x696000002cc mtime = Tue Jan 28 04:10:45 UTC 2020 pZxid = 0x60000023e cversion = 0 dataVersion = 2 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 1137 numChildren = 0 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
