Thanks Luke and Max for the information. We have the use case that inside a DoFn, we will need to call external services to trigger some other flows. The calls to other services are REST based sync calls, and it will take 150 milliseconds plus to return. We are using Flink as the runner and I came across this Async I/O operator from flink, trying to figure out if this is the right approach and if Beam provides any similar concept for it.
Thanks! Eleanore On Wed, Jul 8, 2020 at 2:55 AM Maximilian Michels <[email protected]> wrote: > Just to clarify: We could make the AsnycIO operator also available in > Beam but the operator has to be represented by a concept in Beam. > Otherwise, there is no way to know when to produce it as part of the > translation. > > On 08.07.20 11:53, Maximilian Michels wrote: > > Flink's AsycIO operator is useful for processing io-bound operations, > > e.g. sending network requests. Like Luke mentioned, it is not available > > in Beam. > > > > -Max > > > > On 07.07.20 22:11, Luke Cwik wrote: > >> Beam is a layer that sits on top of execution engines like Flink and > >> provides its own programming model thus native operators like Flink's > >> async IO operator are not exposed. > >> > >> Most people use a DoFn to do all their IO and sometimes will compose > >> it with another transform such as GroupIntoBatches[1] to simplify > >> their implementation. > >> > >> Why do you need async? > >> > >> 1: > >> > https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/ > >> > >> > >> > >> On Tue, Jul 7, 2020 at 11:03 AM Eleanore Jin <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> Hi community, > >> > >> I cannot find any documentation for Beam supporting Flink async IO > >> operator > >> > >> ( > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html), > > >> > >> just wonder is this not supported right now? > >> > >> Thanks a lot! > >> Eleanore > >> >
