We are running CDH 5.4 with Spark 1.3 as our main version and that version is configured to use the external shuffling service. We have also installed Spark 1.5 and have configured it not to use the external shuffling service and that works well for us so far. I would be interested myself how to configure multiple versions to use the same shuffling service.
*Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>* On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler < andreas.fritz...@gmail.com> wrote: > Hi Steve, Alex, > > how do you handle the distribution and configuration of > the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2 > different Spark versions? > > Regards, > Andreas > > On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> >> > On 5 Oct 2015, at 16:48, Alex Rovner <alex.rov...@magnetic.com> wrote: >> > >> > Hey Steve, >> > >> > Are you referring to the 1.5 version of the history server? >> > >> >> >> Yes. I should warn, however, that there's no guarantee that a history >> server running the 1.4 code will handle the histories of a 1.5+ job. In >> fact, I'm fairly confident it won't, as the events to get replayed are >> different. >> > >
