Hi guys
1. >> Add this jar to the classpath of all NodeManagers in your cluster. A related question on configuration of the auxillary shuffle service. *How do i find the classpath for NodeManager?* I tried finding all places where the existing mapreduce shuffle jars are present and place the spark yarn shuffle jar in the same location, but with no success. $ find . -name *shuffle*.jar ./hadoop/client/hadoop-mapreduce-client-shuffle.jar ./hadoop/client/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar ./hadoop/client/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar ./hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar ./hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar ./falcon/client/lib/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar ./oozie/libserver/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar ./oozie/libtools/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar ./spark/lib/spark-1.4.1.2.3.2.0-2950-yarn-shuffle.jar Regards Deenar On 7 October 2015 at 01:27, Alex Rovner <alex.rov...@magnetic.com> wrote: > Thank you all for your help. > > *Alex Rovner* > *Director, Data Engineering * > *o:* 646.759.0052 > > * <http://www.magnetic.com/>* > > On Tue, Oct 6, 2015 at 11:17 AM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> >> On 6 Oct 2015, at 01:23, Andrew Or <and...@databricks.com> wrote: >> >> Both the history server and the shuffle service are backward compatible, >> but not forward compatible. This means as long as you have the latest >> version of history server / shuffle service running in your cluster then >> you're fine (you don't need multiple of them). >> >> >> FWIW I've just created a JIRA on tracking/reporting version mismatch on >> history server playback better: >> https://issues.apache.org/jira/browse/SPARK-10950 >> >> Even though the UI can't be expected to playback later histories, it >> could be possible to report the issue in a way that users can act on "run a >> later version", rather than raise support calls. >> >> >
