*Zero-Copy Utilization in Storm:* Does Apache Storm fully leverage Netty’s zero-copy mechanism for all message transmissions between workers, especially in the context of tuple data transfer? If not, are there any plans to incorporate this further, or specific reasons why it may not be
> feasible? Looking at the codebase, ByteBuf <https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ByteBuf.java> is being used in some flows and it supports transparent zery-copy. > *Performance Benchmarks:* Are there any performance comparisons > available that highlight the impact of zero-copy versus traditional > buffer > copying within Storm’s communication layer? Understanding where > zero-copy > is most beneficial would be helpful for users with high throughput > requirements. > I don't believe there is. > *Configuring Zero-Copy:* Are there any configuration settings that allow > users to tune Netty's communication layer for optimal zero-copy > performance, such as enabling direct buffer allocation or customizing > the > behavior for specific workloads? The configurations being offered are here: https://github.com/apache/storm/blob/master/conf/defaults.yaml On Wed, 9 Oct 2024 at 11:15, Karthick <ibmkarthickma...@gmail.com> wrote: > I have a question regarding the use of Netty for inter-worker communication > in Apache Storm, particularly around the efficiency of data transfer. As > you may know, Netty supports a zero-copy mechanism (e.g., FileRegion and > direct buffer usage in ByteBuf), which can improve performance by > minimizing unnecessary data copying between user and kernel space. > > Could anyone provide insights into the following points? > > 1. > > *Zero-Copy Utilization in Storm:* Does Apache Storm fully leverage > Netty’s zero-copy mechanism for all message transmissions between > workers, > especially in the context of tuple data transfer? If not, are there any > plans to incorporate this further, or specific reasons why it may not be > feasible? > 2. > > *Performance Benchmarks:* Are there any performance comparisons > available that highlight the impact of zero-copy versus traditional > buffer > copying within Storm’s communication layer? Understanding where > zero-copy > is most beneficial would be helpful for users with high throughput > requirements. > 3. > > *Configuring Zero-Copy:* Are there any configuration settings that allow > users to tune Netty's communication layer for optimal zero-copy > performance, such as enabling direct buffer allocation or customizing > the > behavior for specific workloads? > > Any guidance or insights would be greatly appreciated. We are planning to > fine tune the Netty configurations in our clusters. > > Thank you for your time and looking forward to your response. > > Best regards, > Karthick. >
