Hi Shilun, Xinyu, I was one of the contributors of the previous gRPC zero-copy feature in Ratis. There were two reasons the code was very complicated:
1) The zero-copy feature in gRPC itself was a kind of a hack -- That part of gRPC API was not easy to use. 2) The data flow in Ratis Leader is quite involved. When the Leader receives a request from a client, we don't want to copy the client data. But the client data has to be written to the RaftLog and also be sent out to followers. In-between, there is also a cache for caching the log entries. One more complication is the state machine data since the state machine may choose to cache it (a different cache). As a result, the gRPC zero-copy feature in Ratis makes the mode significantly more complicated. I agree that some applications (maybe IoTDB), which do not have large client requests, probably do not want the zero-copy feature. One typical solution to this case is to make the feature configurable. However, even if we could disable the feature, the code will still be changed complicatedly. How about we create a new module, say ratis-gRPC-zero-copy? Currently, we try to make the ratis-grpc work well for both small and large requests at the same time. It may simply be a mission impossible task. Why not create two modules -- one for small requests and the other one for large requests? What do you think? Tsz-Wo On Thursday, July 30, 2026, slfan1989 <[email protected]> wrote: > Hi Xinyu, > > Thank you for sharing your experience from Apache IoTDB. The ownership > and lifecycle issues you described are very relevant to the concerns > around RATIS-1931. > > I agree that the goal should not be to apply zero-copy to every > possible path. A selective approach, focusing on paths with clear > ownership boundaries and measurable performance benefits, would likely > provide a better balance between performance and maintainability. > > The current implementation is already limited mainly to > `RaftClientRequestProto` and `AppendEntriesRequestProto`, while > `InstallSnapshotRequestProto` continues to use the regular marshaller. > I will use the benchmark not only to evaluate whether zero-copy is > beneficial overall, but also to identify where the benefit is > concentrated. > > I plan to organize the measurements by: > > - RPC and execution path; > - Message type and payload size; > - Concurrency level; > - TLS enabled and disabled; > - Zero-copy fast-path and fallback counts; > - Throughput, latency, CPU, allocation, and GC behavior. > > Alongside the performance results, I will also document the ownership > boundaries and lifecycle-management implications of each candidate > path. This should help us evaluate both the measurable benefit and the > corresponding correctness and maintenance cost. > > If the results show that the benefit is significant only for a small > number of large-payload paths, I agree that it would be better to keep > the implementation narrowly scoped rather than extend > reference-counting across additional modules. Any possible extension > to other paths, such as `installSnapshot`, should be considered > separately and only if the expected benefit and ownership model > justify the added complexity. > > Thank you again for this helpful perspective. I will incorporate it > into the benchmark design and share the methodology and results with > the community for further discussion. > > Best regards, > Shilun Fan > > On Thu, Jul 30, 2026 at 10:19 AM Xinyu Tan <[email protected]> wrote: > > > > Hi Shilun, > > > > Thank you for the detailed response and for putting together such a > thoughtful benchmark plan. I think the proposed A/B comparison—using the > same codebase and configuration while changing only the zero-copy > setting—is a good way to isolate the actual benefit. Testing with both TLS > enabled and disabled, together with allocation, GC, direct-memory, > fallback, and unreleased-message metrics, should give the community a much > clearer picture. > > > > One additional thought is that the final design may not necessarily need > to apply zero-copy to every possible path. In my view, the key question is > how to make a reasonable trade-off between performance improvement and > implementation complexity. > > > > I previously worked on memory pooling for large-value workloads in the > Apache IoTDB community. The goal was similar: to reduce the allocation and > GC overhead caused by large objects. Java normally hides memory allocation > and release from application code, but once object pooling is introduced, > the application also becomes responsible for managing the lifecycle of > those objects correctly. > > > > In our case, pooling was relatively straightforward around the RPC layer > because the ownership boundary was clear and there were few coupled > dependencies. The much more difficult part was the in-memory MemTable. A > MemTable or its underlying buffers could be referenced concurrently by > write threads, query threads, and asynchronous flush threads. Once the same > pooled object could be retained by multiple components, it became difficult > to determine when it was truly safe to release or reuse the memory. > > > > Passing such references across module boundaries made the ownership > model even harder to reason about. Supporting it everywhere would have > required manual reference and lifecycle management across a large part of > the Java codebase, introducing a correctness and maintenance burden similar > to manual memory management in C++. > > > > We therefore made a deliberate trade-off: we reused the memory pool in > paths where ownership was clear and the performance gain was significant, > while retaining the original allocation or copy-based approach in paths > where lifecycle management would introduce too much complexity. For > large-value workloads, this selective approach still produced a substantial > improvement, with performance reaching roughly one to two times the > original level in our tests, while keeping the additional complexity > contained. > > > > I think a similar approach may be worth considering for Ratis. The > benchmark could help answer not only whether zero-copy is beneficial > overall, but also which message types, payload ranges, and execution paths > provide enough benefit to justify the added lifecycle-management > complexity. It may be better to optimize a smaller number of high-value and > clearly owned paths than to apply zero-copy broadly and make the entire > implementation harder to maintain. > > > > Thanks again for preparing such a careful evaluation plan. I am looking > forward to seeing the benchmark methodology and results. > > > > Best, > > Xinyu > > > > On 2026/07/30 01:05:34 slfan1989 wrote: > > > Hi Xinyu, > > > > > > Thank you for raising this important point. I agree that we should > > > first quantify the actual performance benefit before investing > > > significant effort in rebuilding the zero-copy branch. This would help > > > us evaluate the benefit more objectively against the implementation > > > complexity and long-term maintenance cost. > > > > > > As you noted, the existing tests mainly focus on correctness and > > > reference release, and do not provide sufficient performance data. > > > > > > After taking another look at the implementation, I found that the > > > current zero-copy marshaller is applied to RaftClientRequestProto and > > > AppendEntriesRequestProto, but not to InstallSnapshotRequestProto. > > > Therefore, I plan to focus the benchmark primarily on large-payload > > > client writes and appendEntries replication. Since installSnapshot is > > > not currently covered by the zero-copy implementation, I do not think > > > it would be appropriate to use it to quantify the direct benefit at > > > this stage. > > > > > > To minimize the impact of unrelated differences between the old branch > > > and master, I plan to perform an A/B comparison using the same > > > codebase and configuration, changing only the zero-copy setting. I > > > will also run the comparison with TLS both enabled and disabled. > > > > > > Where practical, I will collect the following metrics: > > > > > > - Throughput and latency, including percentile latency; > > > - CPU utilization; > > > - Memory allocation rate and GC activity; > > > - Relevant off-heap memory and direct-buffer metrics; > > > - Zero-copy, fallback, and unreleased-message counters. > > > > > > When sharing the results, I will also include the test machine > > > specifications, operating system and JDK versions, commit SHA, > > > relevant Ratis/gRPC settings, payload sizes, concurrency levels, > > > cluster size, warm-up procedure, measurement duration, and number of > > > runs. I hope this will make the results easier for the community to > > > understand and reproduce. > > > > > > I will first prepare the benchmark methodology and share the > > > performance results with the community. Based on the results, we can > > > then discuss whether the expected benefit justifies the full migration > > > and its ongoing maintenance cost. > > > > > > Thank you again for the thoughtful suggestion! Please let me know if > > > you have any recommendations regarding the benchmark design or > > > additional metrics that would be useful. > > > > > > Best regards, > > > Shilun Fan > > > > > > On Tue, Jul 28, 2026 at 2:20 PM Xinyu Tan <[email protected]> wrote: > > > > > > > > Hi Shilun, > > > > > > > > Thanks for driving this forward, and for laying out the rebuild plan > so clearly. I'm supportive of landing zero-copy on master; before the > community commits to the rebuild, though, I'd like to raise a more > ROI-oriented question. > > > > > > > > The overall complexity here is genuinely non-trivial. Zero-copy > introduces manual buffer reference counting on the gRPC deserialization > side — every message must be explicitly released across all async > completion paths (onNext / onCompleted / onError / cancel / timeout), and > the branch history alone took 9+ follow-ups just to enumerate all the > release points (RATIS-2007 / 2018 / 2151 / 2173, etc.). So the change > surface is large and the maintenance cost is real — yet the corresponding > benefit, as far as I can tell, hasn't been quantified. > > > > > > > > I had a quick look: the existing TestGrpcZeroCopy is > correctness/release-focused, with no throughput/latency assertions, and I > couldn't find any measured gRPC zero-copy numbers in the repo or the docs. > So I'd like to check two things with you: > > > > > > > > 1. Did the version previously implemented on the old branch ever > measure an actual performance improvement (throughput / latency / CPU), or > a reduction in GC pressure? > > > > 2. If so, what did the test environment look like roughly, and what > were the specific numbers? > > > > > > > > If that benefit hasn't been measured yet, I'd suggest we first put > together a small demo / benchmark to quantify the "ideal-case" gain before > deciding how to proceed — e.g. targeting large-payload installSnapshot and > large appendEntries, measured under both TLS-on and TLS-off. With concrete > numbers in hand, we can evaluate the ROI and timeline of pushing forward > more objectively, and better balance the investment across the community. > > > > > > > > Would appreciate your thoughts on this. > > > > > > > > Best, > > > > Xinyu Tan > > > > > > > > On 2026/07/19 13:01:18 slfan1989 wrote: > > > > > Hi all, > > > > > > > > > > I would like to continue improving the RATIS-1931 gRPC zero-copy > > > > > feature and ultimately merge it into master. > > > > > > > > > > The existing RATIS-1931_grpc-zero-copy branch has fallen > significantly > > > > > behind master. The two branches now differ substantially in their > > > > > APIs, module structure, test framework, and Raft/LogAppender > > > > > implementations. > > > > > > > > > > Directly merging or rebasing the entire branch would result in a > large > > > > > number of conflicts. It could also reintroduce historical commits > that > > > > > are already in master, have been superseded by newer > implementations, > > > > > or are unrelated to zero-copy. > > > > > > > > > > Therefore, I would like to discuss the following proposal: > > > > > > > > > > 1. Create a new development branch based on the current master and > > > > > re-audit all commits that are unique to the old branch. > > > > > > > > > > 2. Migrate only the commits required for the gRPC zero-copy > > > > > implementation, correctness, tests, configuration, and metrics. > > > > > Commits that are already in master, have been superseded, or are > > > > > unrelated to zero-copy will not be migrated. > > > > > > > > > > 3. Review the selected commits in their original order and > cherry-pick > > > > > them where practical. We will preserve the original commit > attribution > > > > > and logical granularity as much as possible. When a clean > cherry-pick > > > > > is not appropriate because of API changes, partial inclusion in > > > > > master, or obsolete dependencies, the change may be adapted, split, > > > > > combined, or reimplemented on top of the current master. Its > > > > > relationship to the original commit will be documented in the > commit > > > > > message. > > > > > > > > > > 4. Continue developing and improving gRPC zero-copy on the new > branch, > > > > > including reference lifecycles, release handling on exceptional > paths, > > > > > leak detection, metrics, and regression tests. We will also keep > the > > > > > branch synchronized with master to avoid another large divergence. > > > > > > > > > > 5. Once the functionality, correctness, and compatibility have been > > > > > sufficiently validated, create a new pull request and work toward > > > > > merging the reorganized gRPC zero-copy implementation into master. > > > > > > > > > > I would appreciate the community’s feedback on this proposal, > > > > > particularly regarding: > > > > > > > > > > - Whether rebuilding the feature branch on top of the current > master > > > > > is the right approach; > > > > > - Suggestions regarding the commit audit and migration scope; > > > > > - Historical commits that should be specifically included or > excluded; > > > > > - Suggestions for validation and the eventual merge strategy. > > > > > > > > > > Best regards, > > > > > Shilun Fan > > > > > > > > >
