Catcha. Yes, it makes sense to me, and I agree with Tim: we can use something similar to the IOs.
Regards JB On 21/09/2018 16:06, Maximilian Michels wrote: >> not sure I fully follow you there. > > @JB I simply meant to ask whether it make sense to have Runners in the > matrix whose code/documentation is not part of Beam. For it to become a > part of Beam, it could be as easy as adding a link to the external > Runner page. > >> I don't know that we need to limit the matrix to runners in the Beam >> codebase > > @Robert I think we used to only allow Runners in the matrix which were > in the Beam code base. However, you are right, it is not necessary for > Runners to live in the Beam repo. But IMHO they should be documented and > linked before entries to the matrix are made. > >> +1 but perhaps we should having a table listing Runners under >> development like we do for IOs. > > @Tim Yes, I didn't even know the Hadoop Runner was in a branch. > > > I don't want to remove any Runners from the matrix but I propose to > require some form of documentation on the Beam website in addition to > the compatibility matrix entry. > > The current state: > > Runners documented > ================== > > Direct Runner > Apache Apex > Apache Flink > Apache Gearpump > Apache Samza > Apache Spark > Google Cloud Dataflow > > Runners according to the matrix > =============================== > > Apache Apex > Apache Flink > Apache Gearpump > Apache Samza > Apache Spark > Google Cloud Dataflow > Apache Hadoop MapReduce > JStorm > IBM Streams > > > If we can fix the diff between these two lists, I'd feel more > comfortable the next time somebody asks about a Runner I haven't used > yet :) > > Thanks, > Max > > On 21.09.18 14:51, Thomas Weise wrote: >> The MapReduce runner IMHO should not be in the matrix. >> >> For the external runners, is there any public documentation available >> that explains how they can be used and how they are supported? >> >> Thanks, >> Thomas >> >> >> On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré <[email protected] >> <mailto:[email protected]>> wrote: >> >> You are right Tim. >> >> M/R runner is on a branch (in stale for now to be honest ;)). >> >> I think I got Max's remark. >> >> So, agree to focus only Beam coverage in the runner compatibility >> matrix. However, it's also important for the community to have some >> insights about runners generally speaking. >> >> Regards >> JB >> >> On 21/09/2018 14:00, Tim Robertson wrote: >> > "what do you think about limiting the matrix to Runners in the >> Beam code >> > base" >> > >> > +1 but perhaps we should having a table listing Runners under >> > development like we do for IOs. >> > >> > As a concrete example we have MapReduce listed in the matrix [1], >> a page >> > documenting it [2] stating it is in Beam 2.6.0 but unless I'm >> mistaken >> > the code exists only on a branch [3] and hasn't been touched for >> a while. >> > >> > Thanks, >> > Tim >> > >> > [1] >> https://beam.apache.org/documentation/runners/capability-matrix/ >> > [2] https://beam.apache.org/documentation/runners/mapreduce/ >> > [3] https://github.com/apache/beam/tree/mr-runner >> > >> > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré >> <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >> > Hi Max, >> > >> > not sure I fully follow you there. You mean that we would >> have kind of >> > compability matrix on dedicated page of each runner ? >> > >> > Regards >> > JB >> > >> > On 21/09/2018 10:57, Maximilian Michels wrote: >> > > Hi Beamers, >> > > >> > > There have been occasions where people asked me about >> Runner XY and I >> > > had to find out that it only exists in the compatibility >> matrix, >> > but not >> > > as part of our code base. More interestingly, I couldn't >> even find its >> > > code or documentation via my favorite search engine. >> > > >> > > This seems to be the case for multiple Runners in the >> matrix. >> > > >> > > The compatibility matrix will need an overhaul anyways >> with the >> > > portability changes, but what do you think about limiting >> the >> > matrix to >> > > Runners in the Beam code base? >> > > >> > > Thanks, >> > > Max >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> >> -- Jean-Baptiste Onofré >> [email protected] <mailto:[email protected]> >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
