+1 to this proposal. (Interestingly, this was how the very first prototype of the FnAPI worked, back before we had a working data plane :).
On Wed, Nov 3, 2021 at 12:10 PM Robert Burke <[email protected]> wrote: > > Nit: The FnAPI can also send particularly large iterable values over the > State API instead, (so called State Backed Iterables) but that doesn't apply > to the Small Bundle case being described here. Actually, there's no reason one couldn't send a state backed iterable element embedded in the process bundle request, though of course there would be less benefit there. > On Wed, Nov 3, 2021, 11:49 AM Pablo Estrada <[email protected]> wrote: >> >> Thanks for sharing Yichi. Do you have an idea of the potential gains / the >> existing cost on small bundles? >> Thanks >> -P. >> >> On Wed, Nov 3, 2021 at 11:00 AM Yichi Zhang <[email protected]> wrote: >>> >>> Hi, beam dev community, >>> >>> Today the Fn API only uses the data plane for data transferring in bundle >>> processing, it is observed that the protocol has some extent of >>> inefficiency when dealing with small cheap bundles (mostly seen in >>> streaming pipelines). I put up a proposal to enable embedding of a small >>> amount of data in ProcessBundleRequest and ProcessBundleResponse in order >>> to circumvent such inefficiency. >>> >>> Proposal Doc: >>> https://docs.google.com/document/d/14p8Y_n4IY5n9L_I9l5x9lVGgml4ZzdCw645HldndCrw/edit?usp=sharing >>> >>> PTAL. >>> >>> Thanks, >>> Yichi
