On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Thanks for your replies Robert and Cham.
What I had in mind was a generic Wrapper that would easily allow users
to use IO from Java. Such wrapper could start as an experimental
feature
and then, through URN versioning, become stable eventually.
UDFs are needed, though they are a special case. Most users (including
Matthias) just want to specify a few String options which do not
require
UDFs but something along the lines what I proposed here.
Sounds good let's start documenting/implementing the "easy" case and
think bit more regarding UDFs.
Robert wrote:
> UDFs that are called from within an IO as part of its operation is
> still an open question.
Exactly. How about we solve the easier case first, unblock users, and
then think more about solving the general case?
Cham wrote:
> I'm happy to work with you to realize this.
Would be great to exchange more ideas on this! I can compile the
current
ideas we have in a document and we move from there.
Great. Thanks.
Thanks,
Max
On 05.02.19 17:56, Chamikara Jayalath wrote:
>
>
> On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>> wrote:
>
> Good points Cham.
>
> JSON seemed like the most intuitive way to specify a
configuration map.
> We already use JSON in other places, e.g. to specify the
environment
> configuration. It is not necessarily a contradiction to have JSON
> inside
> Protobuf. From the perspective of IO authors, the
user-friendliness
> plays a role because they wouldn't have to deal with Protobuf.
>
>
> It's a good point that JSON will make this more user-friendly for IO
> transforms authors. Probably we should do a bit of
experimentation and
> keep this experimental in case we hit a performance snag.
>
>
> I agree that the configuration format is an implementation
detail that
> will be hidden to users via easy-to-use wrappers.
>
> Do we have to support UDFs for expanding existing IO? Users
would still
> be able to apply UDFs via ParDo on the IO output collections.
Generally
> speaking, I can see how for cross-language transforms UDF
support would
> be good. For example, a Combine implementation in Java, where the
> combine UDFs come from Python.
>
>
> I think we should try to support UDFs in the first version unless
> there's a major blocker that hinders realizing this. Many IO
transforms
> available today expect users to pass UDFs to realize various
features
> (for example, dynamic destinations for BigqueryIO and FileIO,
timestamp
> function for KafkaIO). I think without support for UDFs usability of
> cross-language transforms feature will be significantly limited.
>
> I'm happy to work with you to realize this.
>
> Thanks,
> Cham
>
>
> I suppose the question is, do we try to solve the general
case, or
> do we
> go with a simpler approach for enabling the use of existing
IO first?
> Lack of IO seems to be the most pressing issue for the
adoption of Beam
> Python. I imagine that a backwards-compatible incremental
support for
> cross-language transforms (IOs first, later other transforms)
would be
> possible.
>
> -Max
>
> On 05.02.19 03:07, Chamikara Jayalath wrote:
> >
> >
> > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>> wrote:
> >
> > Yes, I imagine sources to implement a JsonConfigurable
> interface (e.g.
> > on their builders):
> >
> > JsonConfigurable {
> > // Either a json string or Map<String, String>
> > apply(String jsonConfig);
> > }
> >
> > In Python we would create this transform:
> >
> > URN: JsonConfiguredSource:v1
> > payload: {
> > environment: environment_id, // Java/Python/Go
> > resourceIdentifier: string, //
> "org.apache.beam.io.PubSubIO"
> > configuration: json config, // { "topic" :
> "my_pubsub_topic" }
> > }
> >
> >
> > Thanks Max, this is a great first step towards defining to
API for
> > cross-language transforms.
> > Is there a reason why you would want to use JSON instead
of a proto
> > here. I guess we'll be providing a more user friendly language
> wrapper
> > (for example, Python) for end-users here, so
> user-friendliness-wise, the
> > format we choose won't matter much (for pipeline authors).
> > If we don't support UDFs, performance difference will be
> negligible, but
> > UDFs might require a callback to original SDK (per-element
worst
> case).
> > So might make sense to choose the more efficient format.
> >
> > Also, probably we need to define a more expanded definition
> (proto/JSON)
> > to support UDFs. For example, a payload + a set of parameter
> definitions
> > so that the target SDK (for example, Java) can call back the
> original
> > SDK where the pipeline was authored in (for example,
Python) to
> resolve
> > UDFs at runtime.
> >
> > Thanks,
> > Cham
> >
> > That's more generic and could be used for other languages
> where we
> > might
> > have sources/sinks.
> >
> > > (FWIW, I was imagining PubSubIO already had a
translation into
> > BeamFnApi protos that fully specified it, and we use
that same
> > format to translate back out.)
> >
> > Not that I know of.
> >
> > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > Are you suggesting something akin to a generic
> > >
> > > urn: JsonConfiguredJavaSource
> > > payload: some json specifying which source and
which
> parameters
> > >
> > > which would expand to actually constructing and
applying
> that source?
> > >
> > > (FWIW, I was imagining PubSubIO already had a
translation into
> > BeamFnApi
> > > protos that fully specified it, and we use that
same format to
> > translate
> > > back out.)
> > >
> > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
> <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>> wrote:
> > >
> > > Recaping here:
> > >
> > > We all agree that SDF is the way to go for future
> > implementations of
> > > sources. It enables us to get rid of the source
> interfaces.
> > However,
> > > SDF
> > > does not solve the lack of streaming sources in
Python.
> > >
> > > The expansion PR (thanks btw!) solves the
problem of
> > > expanding/translating URNs known to an
> ExpansionService. That
> > is a more
> > > programmatic way of replacing language-specific
performs,
> > instead of
> > > relying on translators directly in the Runner.
> > >
> > > What is unsolved is the configuration of
sources from
> a foreign
> > > environment. In my opinion this is the most
pressing issue
> > for Python
> > > sources, because what is PubSubIO worth in
Python if
> you cannot
> > > configure it?
> > >
> > > What about this:
> > >
> > > I think it is worth adding a JSON configuration
option for
> > all existing
> > > Java sources. That way, we could easily
configure them as
> > part of the
> > > expansion request (which would contain a JSON
> configuration).
> > I'll
> > > probably fork a thread to discuss this in more
detail,
> but would
> > > like to
> > > hear your thoughts.
> > >
> > > -Max
> > >
> > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian
Michels
> > > <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>> wrote:
> > > >
> > > > Ah, I thought you meant native Flink
transforms.
> > > >
> > > > Exactly! The translation code is already
there.
> The main
> > > challenge
> > > > is how to
> > > > programmatically configure the BeamIO from
> Python. I
> > suppose
> > > that is
> > > > also an
> > > > unsolved problem for cross-language
transforms
> in general.
> > > >
> > > >
> > > > This is what
> https://github.com/apache/beam/pull/7316 does.
> > > >
> > > > For a particular source, one would want to
define a
> URN and
> > > > corresponding payload, then (probably) a
> CompositeTransform in
> > > Python
> > > > that takes the users arguments, packages
them into the
> > payload,
> > > applies
> > > > the ExternalTransform, and returns the
results. How
> to handle
> > > arbitrary
> > > > UDFs embedded in sources is still TBD.
> > > >
> > > > For Matthias' pipeline with PubSubIO we
can build
> > something
> > > > specific, but for
> > > > the general case there should be way to
> initialize a
> > Beam IO
> > > via a
> > > > configuration
> > > > map provided by an external environment.
> > > >
> > > >
> > > > I thought quite a bit about how we could
represent
> expansions
> > > statically
> > > > (e.g. have some kind of expansion template that
> could be
> > used, at
> > > least
> > > > in many cases, as data without firing up a
separate
> > process. May be
> > > > worth doing eventually, but we run into the
same issues
> > that were
> > > > discussed at
> > > >
> >
https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > >
> > > > If one is already using a portable runner
like Flink,
> > having the job
> > > > service process automatically also serve up an
> expansion
> > service for
> > > > various URNs it knows and cares about is
probably a
> pretty
> > low bar.
> > > > Flink could serve up things it would rather
get back
> > untouched in a
> > > > transform with a special flink runner urn.
> > > >
> > > > As Ahmet mentions, SDF is better solution. I
hope
> it's not
> > that far
> > > > away, but even once it comes we'll likely
want the
> above
> > > framework to
> > > > invoke the full suite of Java IOs even after
they're
> > running on SDF
> > > > themselves.
> > > >
> > > > - Robert
> > > >
> > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > Exactly, that's what I had in mind.
> > > > >
> > > > > A Flink runner native transform would
make
> the existing
> > > unbounded
> > > > sources
> > > > > available, similar to:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 8:18 AM
Maximilian
> Michels
> > > > <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>> wrote:
> > > > >
> > > > > Wouldn't it be even more useful
for the
> transition
> > > period if
> > > > we enabled Beam IO
> > > > > to be used via Flink (like in the
legacy
> Flink
> > Runner)? In
> > > > this particular
> > > > > example, Matthias wants to use
PubSubIO,
> which
> > is not even
> > > > available as a
> > > > > native
> > > > > Flink transform.
> > > > >
> > > > > On 31.01.19 16:21, Thomas Weise
wrote:
> > > > > > Until SDF is supported, we
could also add
> > Flink runner
> > > > native transforms for
> > > > > > selected unbounded sources [1].
> > > > > >
> > > > > > That might be a reasonable
option to
> unblock
> > users that
> > > > want to try Python
> > > > > > streaming on Flink.
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 6:51 AM
> Maximilian
> > Michels
> > > > <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>
> > > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>>> wrote:
> > > > > >
> > > > > > > I have a hard time to
imagine
> how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers into
the existing
> > > > Bounded/UnboundedSource, so I would
> > > > > > love to hear more about
the details.
> > > > > >
> > > > > > Isn't it the other way around?
> The SDF is a
> > > > generalization of
> > > > > UnboundedSource.
> > > > > > So we would wrap
UnboundedSource
> using
> > SDF. I'm not
> > > > saying it is
> > > > > trivial, but
> > > > > > SDF offers all the
functionality that
> > > UnboundedSource
> > > > needs.
> > > > > >
> > > > > > For example, the
> @GetInitialRestriction
> > method
> > > would
> > > > call split on the
> > > > > > UnboundedSource and the
restriction
> > trackers would
> > > > then be used to
> > > > > process the
> > > > > > splits.
> > > > > >
> > > > > > On 31.01.19 15:16, Ismaël
Mejía
> wrote:
> > > > > > >> Not necessarily. This
would
> be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for UnboundedSource.
> Probably the
> > > easier path
> > > > for migration.
> > > > > > >
> > > > > > > That would be fantastic, I
> have heard
> > about such
> > > > wrapper multiple
> > > > > > > times but so far there
is not any
> > realistic
> > > > proposal. I have a hard
> > > > > > > time to imagine how can
we map
> in a
> > generic way
> > > > RestrictionTrackers
> > > > > > > into the existing
> > Bounded/UnboundedSource, so I
> > > > would love to hear
> > > > > > > more about the details.
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at
3:07 PM
> > Maximilian
> > > Michels
> > > > <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>
> > > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>>> wrote:
> > > > > > >>
> > > > > > >> > In addition to have
> support in
> > the runners,
> > > > this will require a
> > > > > > >> > rewrite of PubsubIO to
> use the
> > new SDF API.
> > > > > > >>
> > > > > > >> Not necessarily. This
would
> be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for
> > > > > > >> UnboundedSource.
Probably the
> easier
> > path for
> > > > migration.
> > > > > > >>
> > > > > > >> On 31.01.19 14:03, Ismaël
> Mejía wrote:
> > > > > > >>>> Fortunately, there
is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use
Java
> IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>
> > > > > > >>> In addition to have
support
> in the
> > > runners, this
> > > > will require a
> > > > > > >>> rewrite of PubsubIO
to use
> the new
> > SDF API.
> > > > > > >>>
> > > > > > >>> On Thu, Jan 31, 2019
at 12:23 PM
> > > Maximilian Michels
> > > > > <[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>
> > > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>>> wrote:
> > > > > > >>>>
> > > > > > >>>> Hi Matthias,
> > > > > > >>>>
> > > > > > >>>> This is already
reflected
> in the
> > > compatibility
> > > > matrix, if you look
> > > > > > under SDF.
> > > > > > >>>> There is no
UnboundedSource
> > interface for
> > > > portable pipelines.
> > > > > That's a
> > > > > > legacy
> > > > > > >>>> abstraction that will be
> replaced
> > with SDF.
> > > > > > >>>>
> > > > > > >>>> Fortunately, there
is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use
Java
> IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> Max
> > > > > > >>>>
> > > > > > >>>> On 31.01.19 12:06,
Matthias
> > Baetens wrote:
> > > > > > >>>>> Hey Ankur,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the
swift reply.
> > Should I change
> > > > this in the
> > > > > capability matrix
> > > > > > >>>>>
> > > >
> >
<https://s.apache.org/apache-beam-portability-support-table>
> > > then?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>> On Thu, 31 Jan 2019
at 09:31,
> > Ankur Goenka
> > > > <[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>
> > > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>>
> > > > > > >>>>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
> > > <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
> > <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi Matthias,
> > > > > > >>>>>
> > > > > > >>>>> Unfortunately,
> unbounded reads
> > > including
> > > > pubsub are not yet
> > > > > > supported for
> > > > > > >>>>> portable runners.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>> Ankur
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan
31, 2019
> at 2:44 PM
> > > Matthias
> > > > Baetens
> > > > > > <[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> <mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>>>
> > > > > > >>>>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>>
> > > > > >
<mailto:[email protected] <mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>
> > > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>
> > > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> > > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
> > <mailto:[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi everyone,
> > > > > > >>>>>
> > > > > > >>>>> Last few
days I
> have been
> > > trying to
> > > > run a streaming
> > > > > > pipeline (code on
> > > > > > >>>>> Github
> > > >
<https://github.com/matthiasa4/beam-demo>) on a
> > > > > > Flink Runner.
> > > > > > >>>>>
> > > > > > >>>>> I am
running a Flink
> > cluster
> > > locally
> > > > (v1.5.6
> > > > > > >>>>>
> > > > <https://flink.apache.org/downloads.html>)
> > > > > > >>>>> I have
built the SDK
> > Harness
> > > > Container: /./gradlew
> > > > > > >>>>>
> > > :beam-sdks-python-container:docker/
> > > > > > >>>>> and
started the
> JobServer:
> > > /./gradlew
> > > > > > >>>>>
> > > >
:beam-runners-flink_2.11-job-server:runShadow
> > > > > > >>>>>
> > -PflinkMasterUrl=localhost:8081./
> > > > > > >>>>>
> > > > > > >>>>> I run my
> pipeline with:
> > > /env/bin/python
> > > > > streaming_pipeline.py
> > > > > > >>>>>
> --runner=PortableRunner
> > > > --job_endpoint=localhost:8099
> > > > > > --output xxx
> > > > > > >>>>>
> --input_subscription xxx
> > > > --output_subscription xxx/
> > > > > > >>>>> /
> > > > > > >>>>> /
> > > > > > >>>>> All this
is running
> > inside a
> > > Ubuntu
> > > > (Bionic) in a
> > > > > Virtualbox.
> > > > > > >>>>>
> > > > > > >>>>> The job
submits
> fine, but
> > > > unfortunately fails after
> > > > > a few
> > > > > > seconds with
> > > > > > >>>>> the error
attached.
> > > > > > >>>>>
> > > > > > >>>>> Anything I am
> missing
> > or doing
> > > wrong?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
