but we did find messages about resetting of the cursor for the topic with
the problem:
"@timestamp",message,mediaId,taskId,appName
"Apr 25, 2022 @ 12:18:31.922","19:18:31.922
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 285143:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.926","19:18:31.926
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 285463:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.926","19:18:31.926
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 285143:0 before current read position 290249:238 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.937","19:18:31.937
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 285463:0 skipping from current read position 285143:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.937","19:18:31.937
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 285790:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.957","19:18:31.957
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 286120:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.957","19:18:31.957
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 285790:0 skipping from current read position 285463:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.978","19:18:31.978
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 286439:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.978","19:18:31.978
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 286120:0 skipping from current read position 285790:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.999","19:18:31.999
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 286439:0 skipping from current read position 286120:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:31.999","19:18:31.999
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 286754:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.020","19:18:32.020
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 286754:0 skipping from current read position 286439:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.020","19:18:32.020
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 287227:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.041","19:18:32.041
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 287227:0 skipping from current read position 286754:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.041","19:18:32.041
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 287627:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.062","19:18:32.062
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 288001:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.062","19:18:32.061
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 287627:0 skipping from current read position 287227:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.077","19:18:32.077
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 288373:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.077","19:18:32.077
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 288001:0 skipping from current read position 287627:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.098","19:18:32.098
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 288373:0 skipping from current read position 288001:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.098","19:18:32.098
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 288733:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.119","19:18:32.118
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 288733:0 skipping from current read position 288373:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.119","19:18:32.119
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 289101:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.136","19:18:32.136
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 289101:0 skipping from current read position 288733:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.136","19:18:32.136
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 289480:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.150","19:18:32.150
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 289480:0 skipping from current read position 289101:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.150","19:18:32.150
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 289849:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.171","19:18:32.171
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 289849:0 skipping from current read position 289480:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.171","19:18:32.171
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
Initiate reset position to 290249:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.188","19:18:32.188
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 285022:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.188","19:18:32.188
[BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-1]
reset position to 290249:0 skipping from current read position 289849:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.208","19:18:32.208
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 285345:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.208","19:18:32.207
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 285022:0 before current read position 290104:1946 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.228","19:18:32.228
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 285345:0 skipping from current read position 285022:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.228","19:18:32.228
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 285668:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.249","19:18:32.249
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 285992:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.249","19:18:32.249
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 285668:0 skipping from current read position 285345:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.270","19:18:32.270
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 285992:0 skipping from current read position 285668:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.270","19:18:32.270
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 286320:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.291","19:18:32.291
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 286320:0 skipping from current read position 285992:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.291","19:18:32.291
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 286635:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.311","19:18:32.311
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 286635:0 skipping from current read position 286320:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.311","19:18:32.311
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 287023:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.332","19:18:32.332
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 287023:0 skipping from current read position 286635:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.332","19:18:32.332
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 287500:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.353","19:18:32.353
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 287500:0 skipping from current read position 287023:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.353","19:18:32.353
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 287873:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.374","19:18:32.374
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 287873:0 skipping from current read position 287500:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.374","19:18:32.374
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 288244:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.395","19:18:32.395
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 288244:0 skipping from current read position 287873:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.395","19:18:32.395
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 288603:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.416","19:18:32.416
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 288603:0 skipping from current read position 288244:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.416","19:18:32.416
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 288977:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.433","19:18:32.432
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 288977:0 skipping from current read position 288603:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.433","19:18:32.433
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 289356:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.452","19:18:32.452
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 289724:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.452","19:18:32.452
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 289356:0 skipping from current read position 288977:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.473","19:18:32.473
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 289724:0 skipping from current read position 289356:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.473","19:18:32.473
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
Initiate reset position to 290104:0 on cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
"Apr 25, 2022 @ 12:18:32.494","19:18:32.494
[BookKeeperClientWorker-OrderedExecutor-1-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[platform/system/persistent/tsc-sm-request-recognition-en-odd-partition-0]
reset position to 290104:0 skipping from current read position 289724:0 on
cursor
platform_persistent%3A%2F%2Fplatform%2Fsystem%2Ftsc-sm-request-recognition-en-odd_queue","-","-","pulsar-broker"
On Mon, Apr 25, 2022 at 2:16 PM Tecno Brain <cerebrotecnolog...@gmail.com>
wrote:
> I observed the same problem again, the cursor of a topic got reset about
> an hour and 15 minutes.
> The queue went from zero to about 2,000 elements in a second without any
> application adding messages. All messages that had occurred during that
> time were replayed.
> (My application, as you can see, does not generate a lot of messages for
> this topic).
>
> I need to understand how the cursors are stored.
>
> Can anyone point me to any documentation or code?
>
> I could not find any exceptions in the log for that topic, but I did found
> around the same time the following exception in a different topic. Any help
> is appreciated.
>
> "Apr 25, 2022 @ 12:18:32.509","19:18:32.509
> [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
> org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble:
> [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering
> to Placement Policy. quarantinedBookies: []"
> "Apr 25, 2022 @ 12:18:31.958","19:18:31.958 [pulsar-io-4-3] WARN
> org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble:
> [pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
> pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering
> to Placement Policy. quarantinedBookies: []"
> "Apr 25, 2022 @ 12:18:31.921","java.lang.NullPointerException: null"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:247)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$71(BrokerService.java:1684)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413)
> ~[io.streamnative-pulsar-common-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185)
> ~[io.streamnative-pulsar-common-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1693)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32)
> [io.streamnative-managed-ledger-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
> [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]"
> "Apr 25, 2022 @ 12:18:31.921","19:18:31.906
> [pulsar-backlog-quota-checker-35-1] ERROR
> org.apache.pulsar.broker.service.BacklogQuotaManager -
> [persistent://platform/system/merge-response-publish-odd-partition-2] Error
> resetting cursor for slowest consumer
> [platform_merge-response-publish-odd_queue]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:129)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$72(BrokerService.java:1700)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.Optional.ifPresent(Optional.java:183) ~[?:?]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1682)
> ~[io.streamnative-pulsar-broker-2.8.2.0.jar:2.8.2.0]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> [?:?]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> [io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]"
> "Apr 25, 2022 @ 12:18:31.921"," at
> java.lang.Thread.run(Thread.java:829) [?:?]"
> "Apr 25, 2022 @ 12:18:31.888","19:18:31.888
> [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
> org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble:
> [pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local
>
> Thanks
>
>
>
>
> On Fri, Mar 18, 2022 at 9:15 PM PengHui Li <peng...@apache.org> wrote:
>
>> It means you have other reset cursor options when doing the ledger
>> trimming
>>
>> On Sat, Mar 19, 2022 at 5:52 AM Tecno Brain <cerebrotecnolog...@gmail.com>
>> wrote:
>>
>>> I found this in the logs:
>>> * Failed to mark delete while trimming data ledgers: Reset cursor in
>>> progress - unable to mark delete position 18377:-1*
>>> but although the messages indicate different topics, the time matches
>>> when we started to see messages replayed.
>>>
>>> What causes that message? What are the consequences of it?
>>>
>>> On Fri, Mar 18, 2022 at 1:31 PM Tecno Brain <
>>> cerebrotecnolog...@gmail.com> wrote:
>>>
>>>> I wonder what would I have seen in the logs if someone had done
>>>> something like this:
>>>>
>>>> bin/pulsar-admin persistent reset-cursor --time 3h
>>>> persistent://tenant/namespace/topic
>>>>
>>>>
>>>> On Thu, Mar 17, 2022 at 1:54 PM Tecno Brain <
>>>> cerebrotecnolog...@gmail.com> wrote:
>>>>
>>>>> Hi Penghui,
>>>>> No, we are not seeing messages "disappear" because of TTL.
>>>>> What we observed is that messages from 3 out of 20 topics were
>>>>> reprocessed.
>>>>> We initially thought that the messages were written again into the
>>>>> topic by our own applications, but we did not find any evidence of that
>>>>> happening. Our applications logs are pretty good and we would have found
>>>>> some evidence.
>>>>> We couldn't find the reason.
>>>>> Our hypothesis was that the cursor was lost and I was trying to
>>>>> find a way to verify that hypothesis through the Pulsar logs...looking if
>>>>> we had lost a broker or a bookie.
>>>>> Initially, I thought that perhaps those 3 topics belonged to the
>>>>> same "bundle" and whenever the broker changed, the cursor was lost.
>>>>> But Pulsar stores the cursor in the bookies, not the broker....so, a
>>>>> broker change shouldn't affect the cursor for the subscription (a shared
>>>>> subscription)
>>>>> We haven't observed the same issue again.
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 12, 2022 at 7:21 AM PengHui Li <peng...@apache.org> wrote:
>>>>>
>>>>>> If you have TTL, the messages will be expired.
>>>>>> You mean "cursor was lost", how do you verify this?
>>>>>> to list subscriptions or not able to consume the message?
>>>>>> If it is the latter, I think it should be related to the message TTL.
>>>>>>
>>>>>> And how about the "brokerDeleteInactiveTopicsMode" in your broker
>>>>>> settings?
>>>>>> If you are using "delete_when_subscriptions_caught_up", after all the
>>>>>> message
>>>>>> been expired, the topic will be deleted automatically by default.
>>>>>>
>>>>>> Penghui
>>>>>>
>>>>>> On Sat, Mar 12, 2022 at 5:28 AM Tecno Brain <
>>>>>> cerebrotecnolog...@gmail.com> wrote:
>>>>>>
>>>>>>> We do have a backlog quota and messageTTL
>>>>>>>
>>>>>>> Our namespace is configured as follows:
>>>>>>>
>>>>>>> Backlog quota:
>>>>>>>
>>>>>>>
>>>>>>> *"message_age BacklogQuotaImpl(limit=-1, limitSize=-1,
>>>>>>> limitTime=180000, policy=consumer_backlog_eviction)"*
>>>>>>> Retention:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *{ "retentionTimeInMinutes" : 0, "retentionSizeInMB" : 0}*
>>>>>>>
>>>>>>> Message TTL:
>>>>>>>
>>>>>>> *172800*
>>>>>>>
>>>>>>> All topics are partitioned.
>>>>>>>
>>>>>>> *{
>>>>>>> "partitions" : 3
>>>>>>> }*
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 10, 2022 at 11:24 PM PengHui Li <codelipeng...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Have you changed the backlog quota policy or enabled the message
>>>>>>>> TTL?
>>>>>>>> Pulsar will not remove any cursors or skip messages by default.
>>>>>>>>
>>>>>>>> Penghui
>>>>>>>> On Mar 11, 2022, 1:25 PM +0800, Tecno Brain <
>>>>>>>> cerebrotecnolog...@gmail.com>, wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>> I have an application using Pulsar just as JMS (we get single
>>>>>>>> messages, acknowledge them when we are done processing it)
>>>>>>>> The entire system, composed of several types of apps, uses about
>>>>>>>> 40 different topics.
>>>>>>>>
>>>>>>>> Yesterday, an application that subscribes to about 20 queues,
>>>>>>>> suddenly was inundated with thousands of messages from two of the
>>>>>>>> queues
>>>>>>>> but I could not track those messages to an application producing them.
>>>>>>>> We
>>>>>>>> found that the messages were *duplicates*.
>>>>>>>> So it seems that the cursor to these two topics was lost, and
>>>>>>>> messages from 3 hours earlier were consumed again. I found the
>>>>>>>> following
>>>>>>>> paragraph :
>>>>>>>>
>>>>>>>> "Each subscription stores a cursor. The cursor is the current
>>>>>>>> offset in the log. Subscriptions store their cursor in BookKeeper in
>>>>>>>> ledgers. This makes cursor tracking scalable just like topics."
>>>>>>>> (
>>>>>>>> https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
>>>>>>>> )
>>>>>>>>
>>>>>>>> My guess is that the cursor was lost.
>>>>>>>> How could I verify that this was the case? I can't find anything
>>>>>>>> relevant in the logs.
>>>>>>>>
>>>>>>>> The only message I found occurring around the same time is
>>>>>>>>
>>>>>>>> New ensemble:
>>>>>>>> [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>>>>>> pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181,
>>>>>>>> pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181] is not
>>>>>>>> adhering to Placement Policy
>>>>>>>>
>>>>>>>> Any pointers are appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
