zongtanghu opened a new pull request #597: [ISSUE #525] Support the message track,from develop branch to msg_track URL: https://github.com/apache/rocketmq/pull/597 What is the purpose of the change Message track trace refers to the consumption processing of a message which is sent from the producer instance has arrived the broker,and then has been consumed by the consumer instance.In the whole process, the information of time, location and other related services(including Client and Broker) is aggregated into the complete link information of message communication. In MQ system, the complete link of a message contains three roles: producer, broker server and consumer.In the process of message communicaton, each role adds relevant information to the track trace link.By aggregating these information,it can provide some powerful support to user. About the Apache RocketMQ project,I will add the feature of message track trace which can help users query every complete link data of a message exactly.This funcation is important to RocketMQ,especially in some financial application fields. Brief changelog Look at the client/broker/common/tools/example/logappender/test module. Look at the rocketmq/distribution/conf/2m-noslave (1)broker-a.properties file. (2)broker-b.properties file. (3)broker-trace.properties file. Verifying this change (1)the special storing for message track trace broker configuration files Join the flag as follows: autoTraceBrokerEnable=true (2)the normal broker configuration files Join the flag as follows: autoTraceBrokerEnable=false ##Architecture and design We plan to design the function of message track trace including its store part and client instance collecting.The total architecture has two part below: （1）Client instance collects the infomation of message track trace The design of client architecture above as follows: （a）."producer and consumer multi-thread module, and Blocking Queue"：here, in client,as a producer model,we can collect the infomation(such as,sending and consuming message cost time,broker store time,broker stored ip address and so on) and put this information into the Blocking Queue.And as s consumer model,use a thread called "MQ-AsyncArrayDispatcher" to take the message track trace infomation from Blocking Queue.Then this asynchronous thread( called "MQ-AsyncArrayDispatcher") pack the message track trace element as AsyncAppenderRequest task and submit to the thead pool. Last,the main execution process of AsyncAppenderRequest task is sending the message track trace infomation which is collected above from client side to a special broker node which is redefined in below chapter. （b）.define a new pair hook which is implementation of the“SendMessageHook/ConsumeMessageHook”,from this,we can collect message track trace data before and after publishing and subscribing messages. （2）Redefine a special broker node to store message track trace data As shown in the above picture,we can define a specail broker service node which can store the message track trace data in a common RocketMQ cluster.Here,we can add a flag(such as autoTraceBrokerEnable) in broker.properties file and use this variable to define Whether this broker is a specail node for storing message track trace data. （a）autoTraceBrokerEnable is false.This indicates this broker is an ordinary node,then "Trace_Topic" will not be created in this node.And it will still follow the original processing flow without any change. （b）autoTraceBrokerEnable is true.This indicates broker is an special node,which stores the message track trace data specailly.And The "Trace_Topic" is automatically created during the Broker's startup phase,this node automatically registers its owned set of topics in memory to nameserver(including Trace_Topic).Thus,in a RocketMQ cluster,there is only a specail broker node which stores message track trace datas.And clients(including publishing and subscribing messages) knows which broker node to send message trace data after they try fetch topic routing info from nameserver. （3）How to solve the query message track trace data by original topic For example,Topic which saves message track trace data is called "RMQ_SYS_TRACE_DATA_XXX" is instead of Topic of original data.But，message query(By messageId + topic,topic+ key or topic) which still uses original data on RocketMQ console side can't query the expected result for us.Here, we can fill in the keyset field of the message track trace data by using msgId (not offset MsgId) or Key of the original message content when send the message track trace data collected by the client, so the IndexFile index of the Broker end can be built according to the msgId or Key of the original message. And at the broker side,we can invokes the queryMessage() method of the storage part through the QueryMessageProcessor Business processor. （4）Finishing querying messages track trace datas in console Fisrtly,the console project is based on spring boot technology,so we can finish querying messages track track datas in this project.The basic design is using thread pool to send a RPC request to the redefined broker node to get the message track trace data. Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR. Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue. Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body. Write a pull request description that is detailed enough to understand what the pull request does, how, and why. Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module. Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass. If this contribution is large, please file an Apache Individual Contributor License Agreement.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
