rdhabalia opened a new issue #12257:
URL: https://github.com/apache/pulsar/issues/12257
### Issue
When topics with high publish rate and high fan-out with large number of
subscriptions or large number of replicators, build backlog then it's really
difficult to drain the backlog for such topics while they serving high
publish-rate. It's reproducible with multiple topics (100) with each 10K
writes, 6 backlog replicators and multiple subscriptions. It becomes impossible
to drain backlogs even if most of the cursors are draining similar backlog
because a large number of cold-reads from bookie makes overall backlog draining
slower and even bookie-cache are not able to keep up with cached entries and
eventually had to do cold-reads.
In such scenario, we would like to force broker to cache backlog reads so,
other subscriotions/replicators draining similar backlog can drain faster
without going to a bookie and reduce number of cold-reads from bookie.
Note: Bookie has read-cache but it's not helping in this usecase because of
backlog draining of multiple topics across multiple brokers. Bookie eventually
returns `TooManyRequestsException` with high reads which also slows down the
backlog draining. and we are not able to drain the backlog and topics with high
publish rates are keep building the backlog.
Below is the stats of the topic which can be used to reproduce the issue
where due to slow reads across many topics builds backlog on all topics and
it's not possible to drain such backlog while incoming high msgRateIn.
```
msgRateIn : 6K, msgRateOut: 4.5K, local-backlog: 3M, msgRateOut per
replicator region: 4K, replication backlog:4M
"replication" : {
"us1" : {
"msgRateIn" : 4637.9,
"msgRateOut" : 4534.8,
"msgRateExpired" : 0.0,
"replicationBacklog" : 3976205,
:
"us2" : {
"msgRateIn" : 4347.5,
"msgRateOut" : 4439.6,
"msgRateExpired" : 0.0,
"replicationBacklog" : 3934344,
:
"us3" : {
"msgRateIn" : 4254.2,
"msgRateOut" : 4534.0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 3974243,
:
:
:
:
"us6" : {
"msgRateIn" : 4845.3,
"msgRateOut" : 4485.4,
"msgRateExpired" : 0.0,
"replicationBacklog" : 3974529,
:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
