Github user squito commented on the pull request:
https://github.com/apache/spark/pull/5400#issuecomment-123011644
@tgravescs sorry I missed your comment earlier. I only did a proof of
concept for network transfer & shuffles, so I never did any performance
testing. I did some light testing w/ caching, there was no noticeable change
in performance -- I can bring that up to date and test it out again.
fyi, the major complication w/ changing the network layers is that we are
now using [netty's
`LengthFieldBasedFrameDecoder`](https://github.com/apache/spark/blob/master/network/common/src/main/java/org/apache/spark/network/util/NettyUtils.java#L98)
which has a limit of 2G. so we'd need to implement our own frame decoder for
larger frames.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
