FWIW there's an overhaul JIRA at https://issues.apache.org/jira/browse/BEAM-2888. Anyone want to pick up some of it? (and add the stuff from this thread?)
Kenn On Fri, Sep 21, 2018 at 9:06 AM Jean-Baptiste Onofré <[email protected]> wrote: > Catcha. > > Yes, it makes sense to me, and I agree with Tim: we can use something > similar to the IOs. > > Regards > JB > > On 21/09/2018 16:06, Maximilian Michels wrote: > >> not sure I fully follow you there. > > > > @JB I simply meant to ask whether it make sense to have Runners in the > > matrix whose code/documentation is not part of Beam. For it to become a > > part of Beam, it could be as easy as adding a link to the external > > Runner page. > > > >> I don't know that we need to limit the matrix to runners in the Beam > >> codebase > > > > @Robert I think we used to only allow Runners in the matrix which were > > in the Beam code base. However, you are right, it is not necessary for > > Runners to live in the Beam repo. But IMHO they should be documented and > > linked before entries to the matrix are made. > > > >> +1 but perhaps we should having a table listing Runners under > >> development like we do for IOs. > > > > @Tim Yes, I didn't even know the Hadoop Runner was in a branch. > > > > > > I don't want to remove any Runners from the matrix but I propose to > > require some form of documentation on the Beam website in addition to > > the compatibility matrix entry. > > > > The current state: > > > > Runners documented > > ================== > > > > Direct Runner > > Apache Apex > > Apache Flink > > Apache Gearpump > > Apache Samza > > Apache Spark > > Google Cloud Dataflow > > > > Runners according to the matrix > > =============================== > > > > Apache Apex > > Apache Flink > > Apache Gearpump > > Apache Samza > > Apache Spark > > Google Cloud Dataflow > > Apache Hadoop MapReduce > > JStorm > > IBM Streams > > > > > > If we can fix the diff between these two lists, I'd feel more > > comfortable the next time somebody asks about a Runner I haven't used > > yet :) > > > > Thanks, > > Max > > > > On 21.09.18 14:51, Thomas Weise wrote: > >> The MapReduce runner IMHO should not be in the matrix. > >> > >> For the external runners, is there any public documentation available > >> that explains how they can be used and how they are supported? > >> > >> Thanks, > >> Thomas > >> > >> > >> On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> You are right Tim. > >> > >> M/R runner is on a branch (in stale for now to be honest ;)). > >> > >> I think I got Max's remark. > >> > >> So, agree to focus only Beam coverage in the runner compatibility > >> matrix. However, it's also important for the community to have some > >> insights about runners generally speaking. > >> > >> Regards > >> JB > >> > >> On 21/09/2018 14:00, Tim Robertson wrote: > >> > "what do you think about limiting the matrix to Runners in the > >> Beam code > >> > base" > >> > > >> > +1 but perhaps we should having a table listing Runners under > >> > development like we do for IOs. > >> > > >> > As a concrete example we have MapReduce listed in the matrix [1], > >> a page > >> > documenting it [2] stating it is in Beam 2.6.0 but unless I'm > >> mistaken > >> > the code exists only on a branch [3] and hasn't been touched for > >> a while. > >> > > >> > Thanks, > >> > Tim > >> > > >> > [1] > >> https://beam.apache.org/documentation/runners/capability-matrix/ > >> > [2] https://beam.apache.org/documentation/runners/mapreduce/ > >> > [3] https://github.com/apache/beam/tree/mr-runner > >> > > >> > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré > >> <[email protected] <mailto:[email protected]> > >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > > >> > Hi Max, > >> > > >> > not sure I fully follow you there. You mean that we would > >> have kind of > >> > compability matrix on dedicated page of each runner ? > >> > > >> > Regards > >> > JB > >> > > >> > On 21/09/2018 10:57, Maximilian Michels wrote: > >> > > Hi Beamers, > >> > > > >> > > There have been occasions where people asked me about > >> Runner XY and I > >> > > had to find out that it only exists in the compatibility > >> matrix, > >> > but not > >> > > as part of our code base. More interestingly, I couldn't > >> even find its > >> > > code or documentation via my favorite search engine. > >> > > > >> > > This seems to be the case for multiple Runners in the > >> matrix. > >> > > > >> > > The compatibility matrix will need an overhaul anyways > >> with the > >> > > portability changes, but what do you think about limiting > >> the > >> > matrix to > >> > > Runners in the Beam code base? > >> > > > >> > > Thanks, > >> > > Max > >> > > >> > -- > >> > Jean-Baptiste Onofré > >> > [email protected] <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>> > >> > http://blog.nanthrax.net > >> > Talend - http://www.talend.com > >> > > >> > >> -- Jean-Baptiste Onofré > >> [email protected] <mailto:[email protected]> > >> http://blog.nanthrax.net > >> Talend - http://www.talend.com > >> > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
