[
https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162351#comment-14162351
]
Patrick Wendell commented on SPARK-3797:
----------------------------------------
For the dependencies issue - the plan is to create a separate build module that
only contains the jar for the shuffle service so we can produce a jar with only
this service and not the rest of Spark's dependency graph. This won't have any
dependencies except for netty which is already a dependency of YARN and we are
using the same version, and potentially the scala library jar (though we've
even discussed writing this particular component in Java). I think that fully
solves the issues Sandy has mentioned.
BTW in general I don't think we are going to require this to run Spark-on-YARN
in the future - it will just be a mode that people can run in if they want to
have better elasticity.
> Run the shuffle service inside the YARN NodeManager as an AuxiliaryService
> --------------------------------------------------------------------------
>
> Key: SPARK-3797
> URL: https://issues.apache.org/jira/browse/SPARK-3797
> Project: Spark
> Issue Type: Sub-task
> Components: YARN
> Reporter: Patrick Wendell
> Assignee: Andrew Or
>
> It's also worth considering running the shuffle service in a YARN container
> beside the executor(s) on each node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
