yandrey321 commented on PR #10536:
URL: https://github.com/apache/ozone/pull/10536#issuecomment-4937321161
> > ... NioByteString and JVM use slower memory copy function ...
>
> > ... gRPC/Netty wire buffer ...
>
> For NioByteString (i.e. direct buffer), there is no buffer copying to
gRPC/Netty wire buffer.
Just checked async-profiler to check, and the data shows a copy does happen
for NioByteString — in fact two copies, versus one for the heap-backed
BoundedByteString.
Setup: the client write path with mocked container RPCs, where the request
is serialized via request.writeTo(...) into a pooled direct Netty ByteBuf — the
same OutputStreamEncoder + framing gRPC's MessageFramer uses. Same benchmark
run twice, only flipping ChunkBuffer allocation (allocateDirect → NioByteString
vs allocateHeap → BoundedByteString). CPU event, 1 MB writes, 2 threads.
Direct / NioByteString.writeTo — 15.7% of total CPU:
```
CodedOutputStream$OutputStreamEncoder.writeTo
└─ NioByteString.writeTo
├─ slice → slice → <init>×4 → jbyte_disjoint_arraycopy #
copy #1: off-heap → temp heap byte[]
└─ writeLazy → ByteBufOutputStream.write
→ AbstractByteBuf.setBytes →
Unsafe.copyMemory # copy #2: heap → direct wire ByteBuf
```
Heap / BoundedByteString(LiteralByteString).writeTo — 0.7% of total CPU:
```
CodedOutputStream$OutputStreamEncoder.writeTo
└─ LiteralByteString.writeTo → writeLazy → ByteBufOutputStream.write
→
AbstractByteBuf.setBytes → Unsafe.copyMemory # single copy: backing byte[] →
wire ByteBuf
```
Both paths terminate in setBytes → Unsafe.copyMemory into the Netty wire
buffer, so neither is zero-copy here. The difference is that
OutputStreamEncoder cannot bulk-transfer an array-less (direct) buffer, so for
NioByteString it first stages the off-heap bytes into a temporary heap buffer
(slice/arraycopy) and only then copies into the pooled direct ByteBuf. That
extra staging copy is why the NioByteString serialization path costs ~22× more
CPU (15.7% vs 0.7% of total), which lines up with the ~11% end-to-end
throughput difference I measured (10,409 vs 11,637 MB/s).
So the direct-buffer path isn't avoiding the wire-buffer copy — it's adding
one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
