devinbost commented on issue #6054: URL: https://github.com/apache/pulsar/issues/6054#issuecomment-843580510
Rajan pointed out that `updateLedgersIdsComplete` is called from a different thread, and it only happens when there's a ledger rollover, so I'm looking upstream from it to see if I can find where the slowdown is happening. Without the extra logs in `updateLedgersIdsComplete`, we still see a lot of this on the broker that isn't performing: ``` DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] Ledger already created when timeout task is triggered DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [myTenant/myNamespace/persistent/bot-filtered] createComplete rc=-23 ledger=-1 DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object ctx) is true DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Calling checkAndCompleteLedgerOpTask(rc, lh, ctx)) on ManagedLedgerImpl.createComplete(..) ``` So, `ManagedLedgerImpl` is still a likely suspect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
