In the Spark runner the user provides the core spark dependencies at runtime and we assume that backwards compatibility is kept (in upstream Spark). We support the whole 2.x line but we try to keep the version close to the latest stable release.
Notice however that we lack tests to validate that all versions do work, I remember some issues with metrics during the migration to spark 2.x with older versions of spark (<= 2.1). Those worked flawlessly with more recent versions. I don't know if Flink could do something like this (become a provided dep) in particular for the current case where there seems not to be API breaking changes. In any case +1 to try to get a bit the act together on this. On Mon, Sep 17, 2018 at 9:31 AM Robert Bradshaw <rober...@google.com> wrote: > > On Mon, Sep 17, 2018 at 2:02 AM Austin Bennett <whatwouldausti...@gmail.com> > wrote: >> >> Do we currently maintain a finer grained list of compatibility between >> execution/runner versions and beam versions? Is this only really a concern >> with recent Flink (sounded like at least Spark jump, too)? I see the >> capability matrix: >> https://beam.apache.org/documentation/runners/capability-matrix/, but some >> sort of compatibility between runner versions with beam releases might be >> useful. >> >> I see compatibility matrix as far as beam features, but not for underlying >> runners. Ex: something like this would save a user trying to get Beam >> working on recent Flink 1.6 and then subsequently hitting a (potentially not >> well documented) wall given known issues. > > > +1. I was bitten by this as well. > > I don't know if it's worth having a compatibility matrix for each version (as > the overlap is likely to be all or nothing in most cases), but it should be > prominently displayed here and elsewhere. Want to send out a PR?
