On Thu, Apr 28, 2016 at 5:41 AM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
> Hi all,
>
> regarding the recent threads on the mailing list, I would like to start a
> format discussion around the IO.
> As we can expect the first contributions on this area (I already have some
> work in progress around this ;)), I think it's a fair discussion to have.
>
> Now, we have two kinds of IO: the one "generic" to Beam, the one "local"
> to the runners.
>
> For example, let's take Kafka: we have the KafkaIO (in IO), and for
> instance, we have the spark-streaming kafka connector (in Spark Runner).
>
> Right now, we have two approaches for the user:
> 1. In the pipeline, we use KafkaIO from Beam: it's the preferred approach
> for sure. However, the user may want to use the runner specific IO for two
> reasons:
> * Beam doesn't provide the IO yet (for instance, spark cassandra
> connector is available whereas we don't have yet any CassandraIO (I'm
> working on it anyway ;))
> * The runner native IO is optimized or contain more features that
> the Beam native IO
> 2. So, for the previous reasons, the user could want to use the native
> runner IO. The drawback of this approach is that the pipeline will be tight
> to a specific runner, which is completely against the Beam design.
>
> I wonder if it wouldn't make sense to add flag on the IO API (and related
> on Runner API) like .useNative().
>
> For instance, the user would be able to do:
>
>
> pipeline.apply(KafkaIO.read().withBootstrapServers("...").withTopics("...").useNative(true);
>
> then, if the runner has a "native" IO, it will use it, else, if
> useNative(false) (the default), it won't use any runner native IO.
>
I think the runner should substitute its native IO whenever it can
(assuming it's actually an improvement). The user should not have to (or,
IMHO, even be able to) request this.
The point there is for the configuration: assuming the Beam IO and the
> runner IO can differ, it means that the "Beam IO" would have to populate
> all runner specific IO configuration.
>
None of this configuration should be a semantic change. There may be other
options that the runner specialization could provide (or require?) but I
think this should be handled by a generic mechanism that allows the user to
pass arbitrary, opaque configuration options to a runner, but importantly
*any* runner is free to ignore these configuration options if it does not
understand them without impacting the semantics of the pipeline.
> Of course, it's always possible to use a PTransform to wrap the runner
> native IO, but we are back on the same concern: the pipeline will be couple
> to a specific runner.
>
> The purpose of the useNative() flag is to "automatically" inform the
> runner to use a specific IO if it has one: the pipeline stays decoupled
> from the runners. <http://www.talend.com>
It's a bit unclear why the runner would not want to use the specific IO if
it has one.
