Hi aaron.ai, What kind of specific performance profiling data do you expect? Basically, this RIP is changing single thread ReputMessageService to multi-stage-multi-threaded version. Something similar in the Linux world is multiple-queuing block IO mechanism https://docs.kernel.org/block/blk-mq.html
@[email protected] <[email protected]> Can we, preliminarily, decompose the dispatch request time into multiple slices, like Linux Kernel IO stages(await time, service time...)? This way, guys would roughly get goals of the optimization. For example, prepare a handful of commit log files without associated consume queues, run the recovery process. During the recovery, collect statistics of each DispatchRequest, including duration in queue, time for writing consume queue logs Zhanhui Li On Mon, Oct 17, 2022 at 5:03 PM aaron ai <[email protected]> wrote: > Hi, Could you provide more data(performance profiling or another way) to > support this proposal ? > > fuyou <[email protected]> 于2022年10月17日周一 15:33写道： > > > good idea. > > > > but i think Introducing bitmap is complicated, so what's about using hash > > for topic and share some queue,ReputService access queue to build Consume > > Queue. > > > > Zhanhui Li <[email protected]> 于2022年10月17日周一 15:04写道： > > > > > Hey, RocketMQ developers and users, > > > > > > It's glad and encouraging to see improvement in the building of > > > ConsumeQueue, which is essential to keep end-to-end latency low, even > if > > > workload of RocketMQ broker is pretty heavy. This is one of the places > > > where RocketMQ really shines among other messaging/streaming products. > > > > > > Questions, suggestions and comments are all welcome! > > > > > > Best Regards! > > > > > > Zhanhui Li > > > > > > On Mon, Oct 17, 2022 at 2:55 PM [email protected] <[email protected]> > > wrote: > > > > > > > Hi, RocketMQ Community > > > > > > > > The generation of messages depend on the ReputMessageService single > > > > thread. When the message production traffic is heavy, the > ConsumeQueue > > > > messages are generated slowly, resulting in a delay in message > > > consumption. > > > > > > > > We need to process CommitLog messages concurrently to speed up the > > > > generation of ConsumeQueue messages. > > > > > > > > [RIP-52 Optimize Building ConsumeQueue] > > > > Wiki < > > > > > > > > > > https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue > > > > > > > > > Chinese version < > > > > https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM> > > > > English version < > > > > https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f> > > > > > > > > Please reply to this email if you have any questions > > > > Best Regards > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > -- > > ============================================= > > > > fuyou001 > > Best Regards > > >
