That is correct.
On Mon, Jul 13, 2020 at 4:33 PM Eleanore Jin wrote:
> Hi Kaymak,
>
> Sorry for the late reply and thanks for sharing the blog, I went through
> it.
>
> here is my understanding:
>
> timely processing could `buffer` data and send them to the external
> system in a batch fashion,
Hi Kaymak,
Sorry for the late reply and thanks for sharing the blog, I went through
it.
here is my understanding:
timely processing could `buffer` data and send them to the external
system in a batch fashion, but in order for it to work `similar` flink
async IO operator it also requires the
Hi Eleanore,
Maybe batched RPC is what you are looking for?
https://beam.apache.org/blog/timely-processing/
On Wed, Jul 8, 2020 at 6:20 PM Eleanore Jin wrote:
> Thanks Luke and Max for the information.
>
> We have the use case that inside a DoFn, we will need to call external
> services to
Thanks Luke and Max for the information.
We have the use case that inside a DoFn, we will need to call external
services to trigger some other flows. The calls to other services are REST
based sync calls, and it will take 150 milliseconds plus to return. We are
using Flink as the runner and I
Just to clarify: We could make the AsnycIO operator also available in
Beam but the operator has to be represented by a concept in Beam.
Otherwise, there is no way to know when to produce it as part of the
translation.
On 08.07.20 11:53, Maximilian Michels wrote:
Flink's AsycIO operator is
Flink's AsycIO operator is useful for processing io-bound operations,
e.g. sending network requests. Like Luke mentioned, it is not available
in Beam.
-Max
On 07.07.20 22:11, Luke Cwik wrote:
Beam is a layer that sits on top of execution engines like Flink and
provides its own programming
Beam is a layer that sits on top of execution engines like Flink and
provides its own programming model thus native operators like Flink's async
IO operator are not exposed.
Most people use a DoFn to do all their IO and sometimes will compose it
with another transform such as GroupIntoBatches[1]
Hi community,
I cannot find any documentation for Beam supporting Flink async IO operator
(
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html),
just wonder is this not supported right now?
Thanks a lot!
Eleanore
