Hey all, Just wanna follow up on multi-raft feature progress. I’ve collect some feedbacks from Xiaoyu, Anu and Sid (https://docs.google.com/document/d/1NxCiHhn0u9BqgjuUXB8zxGtny69Qek4yTFe1QqUHiqM/edit) and address them all in HDDS-2913. Shout out to Xiaoyu, Anu and Sid for the feedbacks and help on resolving them as well. Also would like to know if there are other comments and reviews.
We at Tencent has already deployed the multi-raft version to our internal production cluster and it’s serving reasonable amount of traffic now. So far there are over 16K times of write into our Ozone cluster and I compare with the single-raft version’s performance. Both are measured in similar pattern of traffic on daily basis. Write finishes in: Single raft Multi raft > 3s 0.009% 0.006% 2s ~ 3s 27.4% 1.46% 1s ~ 2s 1.64% 0.07% 0.2s ~ 1s 2.7% 0.53% < 0.2s 68.2% 97.9% Our internal customer writes to ozone every day and there are schedules jobs as well as on-demand jobs. Size could be from KB to GB every write, but every daes y’s traffic share the same pattern. Therefore, we see that multi-raft version makes ~98% of write finish within 0.2s, which is 20% more than what single-raft version can do. At the same time, those who finishes from 2s to 3s reduces from 27.4% to 1.46%. Multi-raft has made our internal cluster more stable and the latency fluctuates way less, which is pretty helpful. Cheers, Li 发件人: "timmycheng(程力)" <[email protected]> 日期: 2020年1月13日 星期一 下午4:24 收件人: "[email protected]" <[email protected]> 主题: [DISCUSS] - Merge Multi-Raft Support - HDDS-1564 Hey all, Happy to present the multi-raft feature to ozone community (https://issues.apache.org/jira/browse/HDDS-1564). This feature is to allow every datanode to host more than 1 pipeline based on user config to better utilize every datanode’s disks IO. All dev work have been done and I’ve conducted performance tests in different scenarios. Based on my testing, multi-raft ozone cluster can help to make writing latency as low as 1/3 of single-raft’s one. Please check the attachment in the above JIRA for test brief and more details as well as the code patch. I would like to use this thread to discuss about this feature and it’s merge back to master. -Li
