Hi all, Both the history server and the shuffle service are backward compatible, but not forward compatible. This means as long as you have the latest version of history server / shuffle service running in your cluster then you're fine (you don't need multiple of them).
That said, an old shuffle service (e.g. 1.2) also happens to work with say Spark 1.4 because the shuffle file formats haven't changed. However, there are no guarantees that this will remain the case. -Andrew 2015-10-05 16:37 GMT-07:00 Alex Rovner <alex.rov...@magnetic.com>: > We are running CDH 5.4 with Spark 1.3 as our main version and that version > is configured to use the external shuffling service. We have also installed > Spark 1.5 and have configured it not to use the external shuffling service > and that works well for us so far. I would be interested myself how to > configure multiple versions to use the same shuffling service. > > *Alex Rovner* > *Director, Data Engineering * > *o:* 646.759.0052 > > * <http://www.magnetic.com/>* > > On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler < > andreas.fritz...@gmail.com> wrote: > >> Hi Steve, Alex, >> >> how do you handle the distribution and configuration of >> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2 >> different Spark versions? >> >> Regards, >> Andreas >> >> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <ste...@hortonworks.com> >> wrote: >> >>> >>> > On 5 Oct 2015, at 16:48, Alex Rovner <alex.rov...@magnetic.com> wrote: >>> > >>> > Hey Steve, >>> > >>> > Are you referring to the 1.5 version of the history server? >>> > >>> >>> >>> Yes. I should warn, however, that there's no guarantee that a history >>> server running the 1.4 code will handle the histories of a 1.5+ job. In >>> fact, I'm fairly confident it won't, as the events to get replayed are >>> different. >>> >> >> >
