That is correct. On Mon, Jul 13, 2020 at 4:33 PM Eleanore Jin <eleanore....@gmail.com> wrote:
> Hi Kaymak, > > Sorry for the late reply and thanks for sharing the blog, I went through > it. > > here is my understanding: > > timely processing could `buffer` data and send them to the external > system in a batch fashion, but in order for it to work `similar` flink > async IO operator it also requires the external system to be able to accept > input data in bulk and return back the response synchronously. Otherwise it > would still like making multiple sync calls to the external system and get > back responses one by one. > > Thanks a lot for sharing! > > Best, > Eleanore > > On Thu, Jul 9, 2020 at 1:56 AM Kaymak, Tobias <tobias.kay...@ricardo.ch> > wrote: > >> Hi Eleanore, >> >> Maybe batched RPC is what you are looking for? >> https://beam.apache.org/blog/timely-processing/ >> >> On Wed, Jul 8, 2020 at 6:20 PM Eleanore Jin <eleanore....@gmail.com> >> wrote: >> >>> Thanks Luke and Max for the information. >>> >>> We have the use case that inside a DoFn, we will need to call external >>> services to trigger some other flows. The calls to other services are REST >>> based sync calls, and it will take 150 milliseconds plus to return. We are >>> using Flink as the runner and I came across this Async I/O operator from >>> flink, trying to figure out if this is the right approach and if Beam >>> provides any similar concept for it. >>> >>> Thanks! >>> Eleanore >>> >>> On Wed, Jul 8, 2020 at 2:55 AM Maximilian Michels <m...@apache.org> >>> wrote: >>> >>>> Just to clarify: We could make the AsnycIO operator also available in >>>> Beam but the operator has to be represented by a concept in Beam. >>>> Otherwise, there is no way to know when to produce it as part of the >>>> translation. >>>> >>>> On 08.07.20 11:53, Maximilian Michels wrote: >>>> > Flink's AsycIO operator is useful for processing io-bound operations, >>>> > e.g. sending network requests. Like Luke mentioned, it is not >>>> available >>>> > in Beam. >>>> > >>>> > -Max >>>> > >>>> > On 07.07.20 22:11, Luke Cwik wrote: >>>> >> Beam is a layer that sits on top of execution engines like Flink and >>>> >> provides its own programming model thus native operators like >>>> Flink's >>>> >> async IO operator are not exposed. >>>> >> >>>> >> Most people use a DoFn to do all their IO and sometimes will compose >>>> >> it with another transform such as GroupIntoBatches[1] to simplify >>>> >> their implementation. >>>> >> >>>> >> Why do you need async? >>>> >> >>>> >> 1: >>>> >> >>>> https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/ >>>> >> >>>> >> >>>> >> >>>> >> On Tue, Jul 7, 2020 at 11:03 AM Eleanore Jin <eleanore....@gmail.com >>>> >> <mailto:eleanore....@gmail.com>> wrote: >>>> >> >>>> >> Hi community, >>>> >> >>>> >> I cannot find any documentation for Beam supporting Flink async >>>> IO >>>> >> operator >>>> >> >>>> >> ( >>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html), >>>> >>>> >> >>>> >> just wonder is this not supported right now? >>>> >> >>>> >> Thanks a lot! >>>> >> Eleanore >>>> >> >>>> >>>
