Hello RocketMQ Community, This is a vote to improve the implementation of tiered storage. According to the plan proposed in RIP-65, the improvements in this submission involve bug fixes and a metadata management API, but they will not "break data format compatibility". In the next PR, I will add more features to the tiered storage module.
We strongly hope that this feature can be stabilized and made available for production. Any other checkpoints are welcome, and please reply to this email, we sincerely hope for your feedback. The vote will be open for at least 72 hours or until the necessary number of votes is reached. Please vote accordingly: [ ] +1 Approve [ ] +0 No opinion [ ] -1 Disapprove with a reason ---------- Forwarded message --------- 发件人： zhimin li <lizhim...@gmail.com> Date: 2023年5月14日周日 14:14 Subject: Re: [DISCUSS] [RIP-65] Tiered Storage Optimization To: <dev@rocketmq.apache.org> Hi, Mock Zhou, Thank you for providing specific suggestions for this project. 1. This advice has been addressed in the new metadata API. 2. It's a great suggestion. I will improve this in future PRs. 3. Could you please describe the specific problem you have found? Currently, I have only observed some asynchronous usage and waiting in the upload and download processes. 4. CompositeAccess is used to combine access to the ConsumeQueue and CommitLog files, and this API is not exposed to users. Regarding the testing report, are you referring to a report on correctness or performance? This suggestion seems unrelated to this topic, and we can create a detailed discussion on this point on GitHub. Here are my brief answers: 1. Although some of the features in rocketmq 5.0 version could have been better designed for code quality, correctness itself can be ensured through test cases, and the community's CICD process can guarantee this very well. To quote an expert, "Testing is critical throughout the entire automation process, and thorough automated testing can greatly increase confidence in delivery." We should believe them. 2. If you are referring to performance testing. The community has plans to incorporate performance baselines into the CICD process. Best Wishes, Zhimin Li Mock Zhou <6199...@qq.com.invalid> 于2023年5月12日周五 22:25写道： > > 看了下文档和代码，几个地方不理解 > > > 1. 比如说这段在 Meta 里面，用 fileSegment 来更新 Meta 很奇怪，Meta 还要关心具体的文件 > 获取或者更新都是用 FileSegmentMeta 或者 Key 确实更合理一点。 > > > FileSegmentMetadata getFileSegment(TieredFileSegment fileSegment); > FileSegmentMetadata updateFileSegment(TieredFileSegment fileSegment); > > > 2. 当时试着跑了下，我的测试服务器是64C256G的，跑了 mysql nginx 等服务，打开这个功能，broker > 就会多四五百个线程，executor 这里不建议静态类初始化。 > > > 3. 存储格式很像 kafka 的 .log 和 .index 文件，其实这里本地 filechannel 或者 mmap 或者 sdk > 写远端都挺快的，上传代码中有一些看起来不是很必要的异步，一个线程等待另一个线程是很危险的，非常容易死锁。 > > > 4. 能否具体讲下 CompositeAccess 在解决什么问题，直接屏蔽访问细节是否更好？ > > > 我们公司关注 5.0 版本很久了，业务方非常希望使用逻辑队列这些新的特性，但是新增的功能代码质量下降太多了，导致我们迟迟不敢上线 5.0 版本。 > 如果有对应的测试报告和数据，能否也给我们参考下，这样（向老板）比较有说服力。 > > > 设计文档比较长，大家都是国人，讲英语的时候把中文也带上吧，实习生觉得你们这个太长就直接跳过了。 > > > > 上海东财信息技术有限公司基础设施团队，六险一金，实线带团队，感兴趣请把简历发我邮箱。 > > > 莫周（联系邮箱 6199...@qq.com） > > > ------------------ 原始邮件 ------------------ > 发件人: > "dev" > > <lizhim...@gmail.com>; > 发送时间: 2023年5月12日(星期五) 晚上9:15 > 收件人: "dev"<dev@rocketmq.apache.org>; > > 主题: Re: [DISCUSS] [RIP-65] Tiered Storage Optimization > > > > As you described, the current implementation fails to meet the > production requirements. It's not just about "just to increase the > level of abstraction". In my opinion, design and quality of the code > are very important in software engineering. Therefore, we should > discuss and improve these codes as soon as possible. At the same time, > I am enthusiastic about adding more feature support in the future. > Also there are some bugs in the implementation as mentioned in the > documentation. Lastly, I suggest that you carefully review the > improvement suggestions outlined in the document, then we can discuss > specific API design. > > Best Wishes, > Zhimin Li > > SSpirits <ad...@lv5.moe> 于2023年5月12日周五 16:37写道： > > > > There is no benefit in doing such a big refactoring at the current stage > just to increase the level of abstraction. This proposal neither facilitates > the participation of new contributors (because there is no optimization of > FileSegment) nor does it have much-needed new capabilities such as new > backend support or timed message support. We can do the refactoring described > in this proposal when there are definite reasons. I hope we can focus on > improving the feature of tiered storage so that it can be production > available as soon as possible. > > > > Your response would be highly appreciated. > > SSpirits > > Email：ad...@lv5.moe<mailto:ad...@lv5.moe> > > On 11 May 2023 at 3:26 PM +0800, zhimin li <lizhim...@gmail.com>, > wrote: > > Hello RocketMQ Community: > > > > I have reviewed the code related to tiered storage. And I hope to > > improve the code quality of the tiered storage. Actually turned out to > > hava a couple of small bugs in it too. > > > > There are some interface improvements in metadata management for > > tiered storage. Here is my detailed design document. > > > https://github.com/apache/rocketmq/wiki/RIP-65-Tiered-Storage-Optimization > > > > Please welcome to reply to this email or comment on the proposal if > > you have any questions or suggestions. > > > > Thanks, > > Zhimin Li
