TakaHiR07 opened a new issue, #19460: URL: https://github.com/apache/pulsar/issues/19460
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version 1. pulsar-server-version: 2.9.3.0.5 based on 2.9.3 2. client-producer-version: c++ 2.9.3 3. client-consumer-version: go-client ### Minimal reproduce step Continuously send message, receive message and ack. ### What did you expect to see? deserializing message metadata successfully ### What did you see instead? deserializing message metadata error when PersistentMessageExpiryMonitor#expireMessages, which result in topic message can not be cleaned. markDeletePosition is 48516, so expireMessages need to get the timestamp of 48517. In our production environment, bookie read entry `69134924:48517` successfully, but broker fail in `parseMessageMetadata`. entry is 3 replication. message retention time is 3 day. However, the near position such as 48516, 48518, 48519 can read successfully, only 48517 fail. The stack is as follow ``` 14:12:57.694 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://like/baina/like_user_event_0501016-partition-5][cksinker_like_user_event_0501016] Error deserializing message for expiry check java.lang.IllegalArgumentException: Invalid unknonwn tag type: 7 at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:270) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.pulsar.common.api.proto.MessageMetadata.parseFrom(MessageMetadata.java:1370) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:435) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:422) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.pulsar.client.impl.MessageImpl.getEntryTimestamp(MessageImpl.java:284) ~[org.apache.pulsar-pulsar-client-original-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor.lambda$expireMessages$0(PersistentMessageExpiryMonitor.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.bookkeeper.mledger.impl.OpFindNewest.readEntryComplete(OpFindNewest.java:89) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5] at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.lambda$asyncReadEntry0$0(EntryCacheImpl.java:222) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] ``` group stats is as follow ``` "group_xxx" : { "markDeletePosition" : "69134924:48516", "readPosition" : "79441322:45498", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 465871946, "cursorLedger" : 79430404, "cursorLedgerLastEntry" : 5047, "individuallyDeletedMessages" : "[(69134924:48518..69134924:49999],(69136302:-1..69136302:49999] ``` ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
