[
https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758488#comment-16758488
]
Jose Torres commented on SPARK-24541:
-------------------------------------
I'm not gonna lie, I didn't put a tremendous amount of thought into the title
of the Jira ticket. There's a strong argument that using Netty is indeed the
right decision here. (Although we have to keep scalability in mind; we'll
eventually need to do some kind of multiplexing to support even moderately
sized N to N shuffles, so we should probably stay compatible with that.)
I'd guess that the RPC framework does carry a performance penalty from things
such as extra headers, but I'd argue the major disadvantage is that it's not
the right abstraction layer. RPCs normally live exclusively in the control
plane.
> TCP based shuffle
> -----------------
>
> Key: SPARK-24541
> URL: https://issues.apache.org/jira/browse/SPARK-24541
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jose Torres
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
