Github user liyezhang556520 commented on the pull request:
https://github.com/apache/spark/pull/12038#issuecomment-203715485
@vanzin
>That's better, but is it needed at all? I don't see any comments about why
consolidating the buffers is a win in the source for CompositeByteBuf.
Traversing the single buffer should be slightly faster because there's less
bookkeeping, but there's the cost of copying that data in the first place.
If so we can just set the `maxNumComponents` with `Integer.Max_VALUE` for
`compositeByteBuffer`.
>When testing this code, I remember that during large transfers packets
would arrive in 64k chunks at the most, so that means that once you're
transferring more than 1MB, you'd have to copy things.
In my test, the chunk sizes are mainly around 20~30 KB.
>Have you tried not consolidating to see whether there's any negative
side-effect?
I tested previously with
`buffers.getFirst().alloc().compositeBuffer(Integer.MAX_VALUE);`, and with
frame size over 1GB, which consist of about 40000 chunks, I didn't see negative
side-effect.
Ok, let's solve this issue without copy for any case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
