> not sure I fully follow you there.
@JB I simply meant to ask whether it make sense to have Runners in the
matrix whose code/documentation is not part of Beam. For it to become a
part of Beam, it could be as easy as adding a link to the external
Runner page.
I don't know that we need to limit the matrix to runners in the Beam codebase
@Robert I think we used to only allow Runners in the matrix which were
in the Beam code base. However, you are right, it is not necessary for
Runners to live in the Beam repo. But IMHO they should be documented and
linked before entries to the matrix are made.
+1 but perhaps we should having a table listing Runners under development like
we do for IOs.
@Tim Yes, I didn't even know the Hadoop Runner was in a branch.
I don't want to remove any Runners from the matrix but I propose to
require some form of documentation on the Beam website in addition to
the compatibility matrix entry.
The current state:
Runners documented
==================
Direct Runner
Apache Apex
Apache Flink
Apache Gearpump
Apache Samza
Apache Spark
Google Cloud Dataflow
Runners according to the matrix
===============================
Apache Apex
Apache Flink
Apache Gearpump
Apache Samza
Apache Spark
Google Cloud Dataflow
Apache Hadoop MapReduce
JStorm
IBM Streams
If we can fix the diff between these two lists, I'd feel more
comfortable the next time somebody asks about a Runner I haven't used yet :)
Thanks,
Max
On 21.09.18 14:51, Thomas Weise wrote:
The MapReduce runner IMHO should not be in the matrix.
For the external runners, is there any public documentation available
that explains how they can be used and how they are supported?
Thanks,
Thomas
On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré <[email protected]
<mailto:[email protected]>> wrote:
You are right Tim.
M/R runner is on a branch (in stale for now to be honest ;)).
I think I got Max's remark.
So, agree to focus only Beam coverage in the runner compatibility
matrix. However, it's also important for the community to have some
insights about runners generally speaking.
Regards
JB
On 21/09/2018 14:00, Tim Robertson wrote:
> "what do you think about limiting the matrix to Runners in the
Beam code
> base"
>
> +1 but perhaps we should having a table listing Runners under
> development like we do for IOs.
>
> As a concrete example we have MapReduce listed in the matrix [1],
a page
> documenting it [2] stating it is in Beam 2.6.0 but unless I'm
mistaken
> the code exists only on a branch [3] and hasn't been touched for
a while.
>
> Thanks,
> Tim
>
> [1] https://beam.apache.org/documentation/runners/capability-matrix/
> [2] https://beam.apache.org/documentation/runners/mapreduce/
> [3] https://github.com/apache/beam/tree/mr-runner
>
> On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi Max,
>
> not sure I fully follow you there. You mean that we would
have kind of
> compability matrix on dedicated page of each runner ?
>
> Regards
> JB
>
> On 21/09/2018 10:57, Maximilian Michels wrote:
> > Hi Beamers,
> >
> > There have been occasions where people asked me about
Runner XY and I
> > had to find out that it only exists in the compatibility
matrix,
> but not
> > as part of our code base. More interestingly, I couldn't
even find its
> > code or documentation via my favorite search engine.
> >
> > This seems to be the case for multiple Runners in the matrix.
> >
> > The compatibility matrix will need an overhaul anyways with the
> > portability changes, but what do you think about limiting the
> matrix to
> > Runners in the Beam code base?
> >
> > Thanks,
> > Max
>
> --
> Jean-Baptiste Onofré
> [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
http://blog.nanthrax.net
Talend - http://www.talend.com
