suiyuzeng opened a new issue #495: The phyOffset in the consume queue is not match the commitlog URL: https://github.com/apache/rocketmq/issues/495 I meet a issue that the phyOffset in the consume queue is not match the commitlog after shutdown abnormally twice. Process is as follows: 1.Master shutdown abnormally. After restarting, commitlog was truncated to 70780951160. One consume queue offset of a topic was truncated from 46297686 to 46275814; 2018-10-16 14:14:58.637 INFO main - topic:role_change, queue:1, queue offset after truncate:46275814, origin:46297686 (the log was add by me) 2.After a while, shutdown abnormally again; 3.After restarting, commitlog was truncated to 70798634224. And the consuem queue offset was truncated from 46297686 to 46287387; 2018-10-16 14:20:36.666 INFO main - topic:role_change, queue:1, queue offset after truncate:46287387, origin:46297686 I write a tool to read the commitlog and consume queue log. The last offset of the consme queue before the commitlog offset 70798634224 should be 46287378; grep the consume queue of commitlog: physicOffset:70798629831, topic:role_change, queueId:1, queueOffset:46287375, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670520000:86840457 physicOffset:70798631359, topic:role_change, queueId:1, queueOffset:46287376, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670520000:86840465 physicOffset:70798632887, topic:role_change, queueId:1, queueOffset:46287377, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670520000:86840473 physicOffset:70798633842, topic:role_change, queueId:1, queueOffset:46287378, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670520000:86840657 physicOffset > 70798634224. The consume queue offset is not discontinuous. physicOffset:70798637468, topic:role_change, queueId:1, queueOffset:46287387, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670875000:88586414 physicOffset:70798638996, topic:role_change, queueId:1, queueOffset:46287388, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670875000:88586422 physicOffset:70798640524, topic:role_change, queueId:1, queueOffset:46287389, bodyLen:37, bodyStr:HELLO ROCKETMQ:1539670875000:88586430 consume queue: offset:46287374, phyoffset:70798628303, size:191, tagsCode:0 offset:46287375, phyoffset:70798629831, size:191, tagsCode:0 offset:46287376, phyoffset:70798631359, size:191, tagsCode:0 offset:46287377, phyoffset:70798632887, size:191, tagsCode:0 offset:46287378, phyoffset:70798633842, size:191, tagsCode:0-------this offset is right offset:46287379, phyoffset:70798623138, size:191, tagsCode:0-------the phyoffset is smaller than last offset:46287380, phyoffset:70798624666, size:191, tagsCode:0 offset:46287381, phyoffset:70798626194, size:191, tagsCode:0 offset:46287382, phyoffset:70798627722, size:191, tagsCode:0 offset:46287383, phyoffset:70798629250, size:191, tagsCode:0 offset:46287384, phyoffset:70798630778, size:191, tagsCode:0 offset:46287385, phyoffset:70798632306, size:191, tagsCode:0 offset:46287386, phyoffset:70798633834, size:191, tagsCode:0------the phyoffset(70798633834) is smaller than 70798634224 offset:46287387, phyoffset:70798637468, size:191, tagsCode:0 offset:46287388, phyoffset:70798638996, size:191, tagsCode:0 offset:46287389, phyoffset:70798640524, size:191, tagsCode:0 The cause is the truncating offset of consume queue. In the first truncating, the offset is reset to 46275814 while the data after the offset was not cleared. In the second restarting , the commitlog offset was truncate to 70798634224. And the real last offset of the consume queue is 46287378(in the commitlog). But after consume queue truncating, the offset is 46287387. In the consume queue log, the last consume queue offset whose phyoffset is smaller than 70798634224 is 46287386. The offset from 46287379 to 46287386 was created before the first tuncating. And in the log, the consume queue offset truncated from are both 46297686; In the commit log, there are some checks(crc,length). But the is no checks in consume queue log. As the consume queue log file is not very large, how about clearing the data after the last offset in consume queue truncating?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
