chenxu80 opened a new issue, #10668: URL: https://github.com/apache/rocketmq/issues/10668
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment ubuntu 24.04 ### RocketMQ version branch: develop version: 5.5.0 Git commit id: a6fb9e2fa0d4e446c88b0051cd4fac233dcee9ec ### JDK Version JDK 8u202 ### Describe the Bug After a slave truncates its CommitLog to a smaller offset, Timer-related data structures such as TimerLog and TimerWheel may still retain records that reference CommitLog offsets beyond the truncation point. These stale references can cause the Timer dequeue pipeline to read an unrelated message and retry it indefinitely. Assume TimerLog contains a record like: ```text TimerLog entry -> CommitLog offset P ``` Originally, offset `P` contains a valid `TIMER_TOPIC` message. After CommitLog is truncated to offset `T`, where `T < P`, the TimerLog entry is not removed. When CommitLog data is transferred again, offset `P` may be overwritten by a completely different message. The resulting state is: ```text TimerLog entry -> CommitLog offset P CommitLog[P] -> non-TIMER_TOPIC message ``` When Timer dequeue processes this TimerLog entry, it reads the new message at `P` and treats it as the original Timer wrapper message. The new message may not contain required Timer properties such as: ```text PROPERTY_REAL_TOPIC PROPERTY_REAL_QUEUE_ID ``` This can cause an exception such as: ```text java.lang.NumberFormatException: null at java.lang.Integer.parseInt(Integer.java:542) at java.lang.Integer.parseInt(Integer.java:615) at org.apache.rocketmq.store.timer.TimerMessageStore.convertMessage(TimerMessageStore.java) at org.apache.rocketmq.store.timer.TimerMessageStore.convert(TimerMessageStore.java) at org.apache.rocketmq.store.timer.TimerMessageStore$TimerDequeuePutMessageService.run(TimerMessageStore.java) ``` With `timerSkipUnknownError=false`, `TimerDequeuePutMessageService` keeps retrying the same `TimerRequest`. The TimerLog record is not repeatedly read. Instead, the already-created `TimerRequest` remains in the internal retry loop and fails continuously. ## Impact A single stale TimerLog reference can prevent the current Timer slot from completing. As a result: - Timer dequeue progress cannot advance. - Subsequent scheduled messages cannot be delivered. - Retry messages using the Timer mechanism, including POP revive retry delivery, may remain blocked. - The broker continuously reports the same exception. ### Steps to Reproduce This issue can be reproduced with a simulated `MessageStore` without constructing the complete HA failure sequence. 1. Create a `TimerMessageStore` with a temporary TimerLog/TimerWheel directory and a mocked `MessageStore`. 2. Keep the default configuration: ```properties timerSkipUnknownError=false ``` 3. Create a valid Timer message and append a TimerLog record that references a simulated CommitLog offset: ```text offsetPy = P sizePy = S ``` Ensure the corresponding TimerWheel slot references this TimerLog record. 4. Configure the mocked `MessageStore` so that reading `(P, S)` now returns a valid ordinary message instead of the original `TIMER_TOPIC` message. The replacement message should: - Have a non-`TIMER_TOPIC` topic. - Not contain `PROPERTY_REAL_TOPIC`. - Not contain `PROPERTY_REAL_QUEUE_ID`. This simulates the state after CommitLog truncation and subsequent HA data overwrite. 5. Enable Timer dequeue and advance the Timer clock until the slot containing the simulated stale record is processed. 6. Run `TimerDequeueGetMessageService` and `TimerDequeuePutMessageService` in separate threads with a test timeout. 7. Observe that the replacement message is passed to `convertMessage()` and triggers: ```text java.lang.NumberFormatException: null at java.lang.Integer.parseInt(Integer.java) at org.apache.rocketmq.store.timer.TimerMessageStore.convertMessage(TimerMessageStore.java) at org.apache.rocketmq.store.timer.TimerMessageStore.convert(TimerMessageStore.java) at org.apache.rocketmq.store.timer.TimerMessageStore$TimerDequeuePutMessageService.run(TimerMessageStore.java) ``` 8. Verify that: - The same `TimerRequest` is retried repeatedly. - The request latch is not completed. - Timer read progress does not advance. - A valid Timer message scheduled after this record is not delivered. 9. Stop the Timer service explicitly at the end of the test to avoid leaving the retry thread running. ### What Did You Expect to See? The message referenced by TimerLog should be validated before conversion. If it is no longer a valid `TIMER_TOPIC` message, the stale request should be released and Timer dequeue should continue. ### What Did You See Instead? The ordinary message is treated as a Timer wrapper message. Conversion fails repeatedly, and the current Timer slot cannot complete. ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
