Silencesk opened a new issue, #10288: URL: https://github.com/apache/rocketmq/issues/10288
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment Linux 4.19 x86_64, physical memory 16GB. ### RocketMQ version Reproduced on both v4.9.4 and v5.3.3. The affected code still exists on develop. ### JDK Version 1.8.0_192-b12 (HotSpot 64-Bit Server VM, mixed mode). Runtime args: -Xms8g -Xmx8g -Xmn4g -XX:+UseG1GC -XX:MaxDirectMemorySize=15g -XX:+AlwaysPreTouch ### Describe the Bug Broker crashes with a fatal JVM error (SIGSEGV / SEGV_MAPERR) while ReputMessageService is writing a dispatch entry to a ConsumeQueue, or while AdminBrokerThread is executing cleanExpiredConsumeQueue. **Case 1 — writer side (v4.9.4):** Problematic frame: v ~StubRoutines::jint_disjoint_arraycopy siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x7fec2674a064 Current thread: JavaThread "ReputMessageService" J C2 org.apache.rocketmq.store.ConsumeQueue.putMessagePositionInfo(JIJJ)Z J C2 org.apache.rocketmq.store.ConsumeQueue.putMessagePositionInfoWrapper J C2 DefaultMessageStore$CommitLogDispatcherBuildConsumeQueue.dispatch J C2 DefaultMessageStore$ReputMessageService.doReput The faulting address sits in an un-mapped hole between two ConsumeQueue mmap regions in /proc/self/maps, proving the target MappedByteBuffer has already been munmapped. **Case 2 — cleaner side (v5.3.3):** Problematic frame: J C2 org.apache.rocketmq.store.queue.ConsumeQueueStore.cleanExpired(J)V Current thread: JavaThread "AdminBrokerThread_12" J C1 DefaultMessageStore.cleanExpiredConsumerQueue J C1 AdminBrokerProcessor.cleanExpiredConsumeQueue Root cause is the same: the ConsumeQueue write/read paths access MappedFile.mappedByteBuffer without holding the ReferenceResource reference count, so a concurrent destroy() can unmap the buffer while a JIT-compiled Unsafe.copyMemory is still writing to it. **Code references (5.3.3 / develop)** Write path has no hold()/release(): // store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java (putMessagePositionInfo) MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile(expectLogicOffset); if (mappedFile != null) { ... this.setMaxPhysicOffset(offset + size); if (messageStore.getMessageStoreConfig().isPutConsumeQueueDataByFileChannel()) { appendResult = mappedFile.appendMessageUsingFileChannel(...); } else { appendResult = mappedFile.appendMessage(this.byteBufferIndex.array()); // writes to mmap } return appendResult; } // DefaultMappedFile.appendMessage(byte[], offset, length) ByteBuffer buf = this.mappedByteBuffer.slice(); // not guarded by hold() buf.position(currentPos); buf.put(data, offset, length); // JIT-inlined Unsafe.copyMemory Concurrent destroyer: // store/.../queue/ConsumeQueueStore.java:cleanExpired } else if (maxCLOffsetInConsumeQueue < minCommitLogOffset) { removeTopicQueueTable(...); this.destroy(nextQT.getValue()); // -> MappedFileQueue.destroy -> MappedFile.destroy(3000) itQT.remove(); } ReferenceResource.shutdown(intervalForcibly) decrements the initial reference on the first call, so if the writer side never calls hold() the counter immediately drops to 0, cleanup() runs, UtilAll.cleanBuffer(mappedByteBuffer) triggers munmap, and the in-flight write hits an unmapped page → SIGSEGV. The intervalForcibly grace window only protects callers that actually do reference counting. Trigger surfaces (any of them reproduces the crash): 1. AdminBrokerProcessor.cleanExpiredConsumeQueue RPC (external management call). 2. DefaultMessageStore$CleanConsumeQueueService periodic deleteExpiredFile – safe for the tail file but still unmaps non-tail files that the dispatcher may be writing to when reput lags behind. 3. DefaultMessageStore.deleteTopics / cleanUnusedTopic (topic removal). 4. Recover / truncate paths (truncateDirty, slave switch). ### Steps to Reproduce 1. Start a broker under sustained write pressure with many topics, so that CommitLog rolls quickly and minCommitLogOffset advances. 2. Either: - Send the CLEAN_EXPIRED_CONSUMEQUEUE admin command repeatedly while dispatch is active, or - Set a small fileReservedTime so CleanConsumeQueueService keeps calling deleteExpiredFileByOffset while ReputMessageService is behind the latest CommitLog offset and still writing to non-tail ConsumeQueue files. 3. Wait for C2 to compile ConsumeQueue.putMessagePositionInfo and ConsumeQueueStore.cleanExpired (a few minutes of load). 4. JVM crashes with SIGSEGV in jint_disjoint_arraycopy / ConsumeQueueStore.cleanExpired; the hs_err_pid*.log memory map shows the faulting address lying in a hole between two consumequeue/... mmap regions, or immediately past the end of one. ### What Did You Expect to See? ConsumeQueue write/read paths and ConsumeQueueStore.cleanExpired cooperate correctly with MappedFile.destroy() via the existing ReferenceResource protocol, so concurrent destroy is safe and at worst causes the current dispatch to be retried. ### What Did You See Instead? Broker JVM crashes with SIGSEGV (SEGV_MAPERR). The target address is an already-munmapped MappedByteBuffer belonging to a ConsumeQueue file that was just destroyed ("cleanExpiredConsumerQueue: ... consumer queue destroyed" log entries immediately before the crash). ### Additional Context If you need the full JVM crash log, I can provide it. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
