> On 18 Apr 2016, at 23:05, Marcelo Vanzin <van...@cloudera.com> wrote: > > On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <r...@databricks.com> wrote: >> The bigger problem is that it is much easier to maintain backward >> compatibility rather than dictating forward compatibility. For example, as >> Marcin said, if we come up with a slightly different shuffle layout to >> improve shuffle performance, we wouldn't be able to do that if we want to >> allow Spark 1.6 shuffle service to read something generated by Spark 2.1. > > And I think that's really what Mark is proposing. Basically, "don't > intentionally break backwards compatibility unless it's really > required" (e.g. SPARK-12130). That would allow option B to work. > > If a new shuffle manager is created, then neither option A nor option > B would really work. Moving all the shuffle-related classes to a > different package, to support option A, would be really messy. At that > point, you're better off maintaining the new shuffle service outside > of YARN, which is rather messy too. >
There's a WiP in YARN to move Aux NM services into their own CP, though that doesn't address shared native libs, such as the leveldb support that went into 1.6 There's already been some fun with Jackson versions and that of Hadoop — SPARK-12807; something that per-service classpaths would fix. would having separate CPs allow multiple spark shuffle JARs to be loaded, as long as everything bonded to the right one?
