Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5234#issuecomment-87134037
This seems to me a pretty serious problem with our current implementation
and a limitation from Netty w.r.t. the use of zero-copy.
If I am understanding this correctly, the problem is that Netty Channel
cannot accept a composite buffer that includes a ByteBuf and a FileRegion at
the same time. As a result, we separated a data transfer message into two
messages: one for header, and one for data itself using transferTo.
Netty already went most of the way to make sure users don't have to worry
about linearizability and concurrency, but in this case we'd have to resort to
locking in order to guarantee atomicity of a message that gets divided into
two.
If Netty can have some support structure for this, it'd greatly simplify
our programming model and eliminate the locking required at our level. Looks
like an extra if statement that checks whether the "msg" is a special object
that contain a ByteBuf and a FileRegion:
https://github.com/netty/netty/blob/33f75d374091946058fc334c3cdbcd0f0a59d9b3/transport/src/main/java/io/netty/channel/nio/AbstractNioByteChannel.java#L189
cc @normanmaurer and @trustin any comment on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
