f-ld commented on issue #5436: Topic do not always expire as expected URL: https://github.com/apache/pulsar/issues/5436#issuecomment-544889113 Actually found something for that topic. It has been created in region1 via a subscription on a non-existing topic. That subscription has a consumer kept alive. During that creation, producers are created for replication in all regions and after some time (30-90 secs), because of inactivity, replicators are closed. Example of logs for another topic having same symptoms : ``` 14:40:02.099 [pulsar-inactivity-monitor-24-1] INFO org.apache.pulsar.broker.service.AbstractReplicator - [persistent://tenant/namespace/topic-036db877][region1 -> region5] Disconnect replicator at position 1860323:-1 with backlog 0 14:40:02.099 [pulsar-inactivity-monitor-24-1] INFO org.apache.pulsar.broker.service.AbstractReplicator - [persistent://tenant/namespace/topic-036db877][region1 -> region4] Disconnect replicator at position 1860323:-1 with backlog 0 14:40:02.099 [pulsar-inactivity-monitor-24-1] INFO org.apache.pulsar.broker.service.AbstractReplicator - [persistent://tenant/namespace/topic-036db877][region1 -> region2] Disconnect replicator at position 1860323:-1 with backlog 0 14:40:02.099 [pulsar-inactivity-monitor-24-1] INFO org.apache.pulsar.broker.service.AbstractReplicator - [persistent://tenant/namespace/topic-036db877][region1 -> region3] Disconnect replicator at position 1860323:-1 with backlog 0 14:40:02.114 [pulsar-io-22-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://tenant/namespace/topic-036db877] [pulsar.repl.region1] Closed Producer 14:40:02.170 [pulsar-io-22-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://tenant/namespace/topic-036db877] [pulsar.repl.region1] Closed Producer 14:40:02.280 [pulsar-io-22-13] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://tenant/namespace/topic-036db877] [pulsar.repl.region1] Closed Producer 14:40:02.318 [pulsar-io-22-6] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://tenant/namespace/topic-036db877] [pulsar.repl.region1] Closed Producer ``` I then have within the next few seconds the same logs in region3 / region4 / region 5. The only region not closing replicators is region2 because there is still a consumer in subscription at the origin of the topic creation. And the stats are from region1 the same as reported in origin description: ``` { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "averageMsgSize" : 0.0, "storageSize" : 0, "publishers" : [ ], "subscriptions" : { }, "replication" : { "region2" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0, "inboundConnection" : "/10.11.1.47:40798", "inboundConnectedSince" : "2019-10-21T14:39:31.158Z" }, "region3" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 }, "region4" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 }, "region5" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 } }, "deduplicationStatus" : "Disabled" } ``` and from region2: ``` { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "averageMsgSize" : 0.0, "storageSize" : 0, "publishers" : [ ], "subscriptions" : { }, "replication" : { "region1" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0, "inboundConnection" : "/10.11.1.47:40798", "inboundConnectedSince" : "2019-10-21T14:39:31.158Z" }, "region3" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 }, "region4" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 }, "region5" : { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateExpired" : 0.0, "replicationBacklog" : 0, "connected" : false, "replicationDelayInSeconds" : 0 } }, "deduplicationStatus" : "Disabled" } ``` So I suppose this is normal: - in region2, subscription does not expire because of alive consumer - because subscription does not expire, topic does not either and replicators to other regions are kept from that region to other regions - in other regions, because of inactivity (no subscription, no producer other that replicators) then topics do expire and replicators from that region to others are closed. And I checked: - that some message is published in topic in region1 (or region3 / region4 / region5), then topic init is triggered again, all replicators recreated and message broadcasted everywhere -> OK - that if I start another consumer in region3, then stats only do change in region3 showing that consumer. There is no restart of replicators -> OK - that if some message is published in topic in region1, then replicators -- stil alive -- push it to other regions -> OK So there is no bug here, just a misunderstanding from my side. Closing the issue
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
