Hi Neal, >From my understanding, the reason I think the shuffle is not available in maven central is that it is an external component and dependent on the cluster manager version - atleast in case of yarn.
For example, it would require the appropriate hadoop profile based on your underlying infra to build the right shuffle component. Publishing all of the different combinations to maven doesn't make sense. So you would have to store it externally - probably an internal central repo in your company. -Dhruve On Mon, Dec 12, 2016 at 11:56 PM, Neal Yin <neal....@workday.com> wrote: > Hi, > > For dynamic allocation feature, I need spark-xxx-yarn-shuffle.jar. In my > local spark build, I can see it. But in maven central, I can’t find it. My > build script pulls all jars from maven central. The only option is to check > in this jar into git? > > Thanks, > > -Neal > -- -Dhruve Ashar
