Hi Xinyu, Welcome to Apache Ratis. Thanks a lot for joining us working on it!
- One idea is to use Streaming to do appendEntries. gRPC seems not suitable to do appendEntries when the data size is large since it requires buffer allocation and buffer copying. It is better to use Netty zero buffer copying, i.e. Streaming. - For the data stream feature, there is a missing component -- Currently, it requires all nodes to be available during streaming. AppendEntries won't include streamed data. So if a follower does not have the stream data, it must install a snapshot in order to catch up. In Ozone Write Pipeline, we do have an assumption that all datanodes are available in the pipeline. When one of the datanodes fails, it closes the pipeline (so that it does not have to take care of resending streamed data). Since you seem to be interested in the data stream feature, you may consider working on this missing component. - Another idea is to support reading using Streaming. Then, reading can also have Netty zero buffer copying. These are my wish list items. Tsz-Wo On Tue, Feb 21, 2023 at 8:45 AM Xinyu Tan <[email protected]> wrote: > Hi, Ratis CommunityRatis Community Community > > I'm Xinyu Tan from the Apache IoTDB community. My Github ID is > OneSizeFitsQuorum, and I'm interested in consensus algorithm, distributed > storage system, time-series database and distributed transaction. > > In the past nearly a year, the consensus layer of Apache IoTDB has > integrated with Apache Ratis, which makes Apache IoTDB capable of strong > consistency. Many thanks to Apache Ratis community especially Tsz Wo Sze > for your patient help and the efforts of William. > > In the next year, apart from William, another student and I will also > devote our energy to the stability, performance, memory control, snapshot > transmission and other issues of Apache Ratis, hoping to make Apache Ratis > and Apache IoTDB better and better. > > Finally, I would like to know if there is any overall roadmap for Apache > Ratis in 2023 or release/3.0, so that we can review some of the issues that > Apache IoTDB is having with Apache Ratis to do something more beneficial > for both of us. > > Thanks > ---------------------- > Xinyu Tan >
