Hi Andrew, thanks a lot for the clarification!
Regards, Andreas On Tue, Oct 6, 2015 at 2:23 AM, Andrew Or <and...@databricks.com> wrote: > Hi all, > > Both the history server and the shuffle service are backward compatible, > but not forward compatible. This means as long as you have the latest > version of history server / shuffle service running in your cluster then > you're fine (you don't need multiple of them). > > That said, an old shuffle service (e.g. 1.2) also happens to work with say > Spark 1.4 because the shuffle file formats haven't changed. However, there > are no guarantees that this will remain the case. > > -Andrew > > 2015-10-05 16:37 GMT-07:00 Alex Rovner <alex.rov...@magnetic.com>: > >> We are running CDH 5.4 with Spark 1.3 as our main version and that >> version is configured to use the external shuffling service. We have also >> installed Spark 1.5 and have configured it not to use the external >> shuffling service and that works well for us so far. I would be interested >> myself how to configure multiple versions to use the same shuffling service. >> >> *Alex Rovner* >> *Director, Data Engineering * >> *o:* 646.759.0052 >> >> * <http://www.magnetic.com/>* >> >> On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler < >> andreas.fritz...@gmail.com> wrote: >> >>> Hi Steve, Alex, >>> >>> how do you handle the distribution and configuration of >>> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2 >>> different Spark versions? >>> >>> Regards, >>> Andreas >>> >>> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <ste...@hortonworks.com> >>> wrote: >>> >>>> >>>> > On 5 Oct 2015, at 16:48, Alex Rovner <alex.rov...@magnetic.com> >>>> wrote: >>>> > >>>> > Hey Steve, >>>> > >>>> > Are you referring to the 1.5 version of the history server? >>>> > >>>> >>>> >>>> Yes. I should warn, however, that there's no guarantee that a history >>>> server running the 1.4 code will handle the histories of a 1.5+ job. In >>>> fact, I'm fairly confident it won't, as the events to get replayed are >>>> different. >>>> >>> >>> >> >
