michalcukierman commented on issue #20968: URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1680437807
@codelipenghui I was able to reproduce the error with `pulsar-perf` and have some new observations: - Data loss occurs on a single topic, it's harder to reproduce (does not happen every time) - Messages producing is not required to reproduce the issue. A single, active consumer is needed to recreate the error. - (Sometimes?) Partitions are lost after the test, what could be the cause of the lack of messages I've created a project with two scripts (one for partitioned, one for non-partitioned topics) and put in a GH repository: https://github.com/websight-io/pulsar-chaos-test-perf The reproduction steps (simplified, see the project for the one you would like to execute): ``` TOPIC="test-partitioned-0" # Preparation kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics create-partitioned-topic $TOPIC -p 12 kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics set-retention -s -1 -t -1 $TOPIC kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics get-retention $TOPIC kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf produce -bm 1 -r 100 -m 10000 -s 102400 $TOPIC # Execution kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf consume -ioThreads 4 -c 4 -sp Earliest $TOPIC & CONSUMER_PID=$! sleep 15 # wait for the producer to start sending messages kubectl delete pod pulsar-broker-0 -n pulsar kubectl delete pod pulsar-broker-1 -n pulsar kubectl delete pod pulsar-broker-2 -n pulsar sleep 90 # wait for the messages to be received and brokers to stabilized kill -15 $CONSUMER_PID # Verification kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf read $TOPIC ``` Data loss on partitioned topic takes some time, you may be read all the messages at the first attempt, they may be lost after some time (try to re-run the test). You may also try to run tests several times, or change the parameters like `sleep` time or messages number/size. Alternative verification that can replace reader (I see consistent results in sum of `numberOfEntries` on partitions and read messages): ``` for i in {0..11} do kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal $TOPIC-partition-$i done ```` The output of internal stats for partitioned topic is sometimes: ``` kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-partitioned-topic-partition-0` Partitioned Topic not found: persistent://public/default/test-partitioned-topic-partition-0 has zero partitions ``` Example internal-stats taken after one of my run: ``` chaos-test-pulsar-perf git:(main) ✗ for i in {0..11} do kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-partitioned-10-partition-$i done { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.644Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2022:841", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2110, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2022:841", "readPosition" : "2022:842", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:02:26.861Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2022:-1", "readPosition" : "2110:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -842, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.347Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.491Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2042:818", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2174, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2042:818", "readPosition" : "2042:819", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.725Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2042:-1", "readPosition" : "2174:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -819, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.382Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.064Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2026:849", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2138, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2026:849", "readPosition" : "2026:850", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.244Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2026:-1", "readPosition" : "2138:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -850, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.415Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.179Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2029:817", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2164, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2029:817", "readPosition" : "2029:818", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.304Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2029:-1", "readPosition" : "2164:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -818, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.409Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.658Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2024:843", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2112, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2024:843", "readPosition" : "2024:844", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:02:26.868Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2024:-1", "readPosition" : "2112:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -844, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.369Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.526Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2041:833", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2178, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2041:833", "readPosition" : "2041:834", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.727Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2041:-1", "readPosition" : "2178:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -834, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.412Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.1Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2025:830", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2150, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2025:830", "readPosition" : "2025:831", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.464Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2025:-1", "readPosition" : "2150:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -831, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.403Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.109Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2030:831", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2158, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2030:831", "readPosition" : "2030:832", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.183Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2030:-1", "readPosition" : "2158:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -832, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.392Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.727Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2023:832", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2126, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2023:832", "readPosition" : "2023:833", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:02:27.185Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2023:-1", "readPosition" : "2126:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -833, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.362Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.437Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2037:826", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2169, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2037:826", "readPosition" : "2037:827", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.713Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2037:-1", "readPosition" : "2169:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -827, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.395Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.074Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2028:841", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2157, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2028:841", "readPosition" : "2028:842", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.172Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2028:-1", "readPosition" : "2157:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -842, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.384Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } { "entriesAddedCounter" : 0, "numberOfEntries" : 0, "totalSize" : 0, "currentLedgerEntries" : 0, "currentLedgerSize" : 0, "lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.068Z", "waitingCursorsCount" : 2, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "2027:827", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 2139, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "sub" : { "markDeletePosition" : "2027:827", "readPosition" : "2027:828", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.224Z", "state" : "NoLedger", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "multiTopicsReader-883d8df4ce" : { "markDeletePosition" : "2027:-1", "readPosition" : "2139:0", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : -828, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.373Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } } ``` The logs for `public/default/persistent/test-partitioned-10-partition-0` <img width="2927" alt="Screenshot 2023-08-16 at 13 11 36" src="https://github.com/apache/pulsar/assets/4356553/f19f1499-268d-46e1-abce-e7d010bc4b55";> You may see that the ledger 2022 (which was holding the data) was deleted The logs for `persistent://public/default/test-partitioned-10-partition-0` <img width="2919" alt="Screenshot 2023-08-16 at 13 23 29" src="https://github.com/apache/pulsar/assets/4356553/4d26181e-fd52-471f-854c-8d74fe1c6d34";> Notice that `policies` to the topic are applied after the ledger is removed. It also runing on different threads. Note that all 3 brokers were managing the topic (log entries per broker: broker-0: 23, broker-1: 23, broker-2: 1). (please verify if it may be relevant): <img width="2922" alt="Screenshot 2023-08-16 at 13 28 29" src="https://github.com/apache/pulsar/assets/4356553/29aa7849-1c90-46c2-9edf-1b4d5ac1340b";> The logs for ledger `#2022` <img width="2924" alt="Screenshot 2023-08-16 at 13 15 19" src="https://github.com/apache/pulsar/assets/4356553/7d95eab7-80e5-4618-ac2a-e6a2ce2590c1";> The logs for current ledger `#2110`: <img width="2918" alt="Screenshot 2023-08-16 at 13 16 36" src="https://github.com/apache/pulsar/assets/4356553/6bd9cbc6-d4fa-4abf-bb72-23f103e7f5a6";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
