Hi, As part of our POC, we are testing Spark runner. We tried to submit a submit job to a spark(2.2) cluster running on a K8s cluster. As per our findings during this POC, the beam capability matrix https://beam.apache.org/documentation/runners/capability-matrix/ is not updated.
Below features are actually supported in SparkRunner: 1) GroupByKey 2) Event-time triggers (watermark crosses the end of window) 3) Count triggers 4) Allowed lateness 5) Accumulating fired panes. *Thanks & Regards,* *Vishwas * On Mon, Sep 17, 2018 at 1:18 PM Ismaël Mejía <[email protected]> wrote: > In the Spark runner the user provides the core spark dependencies at > runtime and > we assume that backwards compatibility is kept (in upstream Spark). We > support > the whole 2.x line but we try to keep the version close to the latest > stable > release. > > Notice however that we lack tests to validate that all versions do work, I > remember some issues with metrics during the migration to spark 2.x with > older > versions of spark (<= 2.1). Those worked flawlessly with more recent > versions. > > I don't know if Flink could do something like this (become a provided > dep) in particular for the current case where there seems not to be > API breaking changes. > > In any case +1 to try to get a bit the act together on this. > > On Mon, Sep 17, 2018 at 9:31 AM Robert Bradshaw <[email protected]> > wrote: > > > > On Mon, Sep 17, 2018 at 2:02 AM Austin Bennett < > [email protected]> wrote: > >> > >> Do we currently maintain a finer grained list of compatibility between > execution/runner versions and beam versions? Is this only really a concern > with recent Flink (sounded like at least Spark jump, too)? I see the > capability matrix: > https://beam.apache.org/documentation/runners/capability-matrix/, but > some sort of compatibility between runner versions with beam releases might > be useful. > >> > >> I see compatibility matrix as far as beam features, but not for > underlying runners. Ex: something like this would save a user trying to > get Beam working on recent Flink 1.6 and then subsequently hitting a > (potentially not well documented) wall given known issues. > > > > > > +1. I was bitten by this as well. > > > > I don't know if it's worth having a compatibility matrix for each > version (as the overlap is likely to be all or nothing in most cases), but > it should be prominently displayed here and elsewhere. Want to send out a > PR? >
