yangou opened a new issue #10464:
URL: https://github.com/apache/pulsar/issues/10464
**Describe the bug**
There are chances that messages were created before the subscriptions were
created. So we disabled the topic auto-deletion in the namespace. If the
messages were created 4 hours ahead of time, my subscription with the initial
position works as expected. However if the messages were created more than 6
hours ahead of time, after my consumer establishes the subscription, the
backlog won't get cleared. Consumers are able to consume from the point of
subscription, but none of the earlier messages would be delivered to consumers.
**To Reproduce**
1. Create a namespace with the below policies:
```
---
bundles:
numBundles: 64 #
bundles serve the purpose of topic load balancing among brokers
backlog_quota_map:
destination_storage:
limit: 1099511627776 # 1TB,
maximal backlog size of unack'ed messages
policy: producer_request_hold # keep
the producer waiting/timeout when backlog reaches limit
deduplicationEnabled: false
autoTopicCreationOverride:
allowAutoTopicCreation: true
topicType: non-partitioned
autoSubscriptionCreationOverride:
allowAutoSubscriptionCreation: true
message_ttl_in_seconds: 0 # don't
expire unack'ed messages
subscription_expiration_time_minutes: 0 # don't
expire subscription once inactive(no connections, no consumptions, etc)
retention_policies:
retentionTimeInMinutes: 0 # don't
retent the messages around once they are consumed
retentionSizeInMB: 0 # save
as above
delayed_delivery_policies:
active: false #
disable delayed delivery in minutely namespace
inactive_topic_policies:
deleteWhileInactive: true #
enable auto-deletion on inactive topics
inactiveTopicDeleteMode: delete_when_subscriptions_caught_up # once
subscroptions have caught up allow deletion on the topics
maxInactiveDurationSeconds: 216000 # wait
util 60 hours after topic become inactive, this will make this namespace to
have 60*60*2=7200 topics at max
```
2. publish some messages to a topic such that topic gets automatically
created.
3. wait for 6 hours, create a `Shared` consumer using the `Earliest`
position.
4. You won't see any messages being delivered.
5. Get the topic stats:
```
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 74768,
"msgInCounter" : 495,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 0,
"backlogSize" : 0,
"publishers" : [ ],
"subscriptions" : {
"scheduler" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 472,
"msgBacklogNoDelayed" : 472,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1619913730283,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"consumerName" : "zacqt",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1000,
"blockedConsumerOnUnackedMsgs" : false,
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"metadata" : { },
"connectedSince" : "2021-05-02T00:02:10.281Z",
"address" : "/10.72.94.193:56002"
} ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { }
}
},
"replication" : { },
"deduplicationStatus" : "Disabled"
}
```
**Expected behavior**
The consumer starts with the Earliest position would clear up the backlog
created before the subscription was created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
