Nit: The FnAPI can also send particularly large iterable values over the State API instead, (so called State Backed Iterables) but that doesn't apply to the Small Bundle case being described here.
On Wed, Nov 3, 2021, 11:49 AM Pablo Estrada <[email protected]> wrote: > Thanks for sharing Yichi. Do you have an idea of the potential gains / the > existing cost on small bundles? > Thanks > -P. > > On Wed, Nov 3, 2021 at 11:00 AM Yichi Zhang <[email protected]> wrote: > >> Hi, beam dev community, >> >> Today the Fn API only uses the data plane for data transferring in bundle >> processing, it is observed that the protocol has some extent of >> inefficiency when dealing with small cheap bundles (mostly seen in >> streaming pipelines). I put up a proposal to enable embedding of a small >> amount of data in ProcessBundleRequest and ProcessBundleResponse in order >> to circumvent such inefficiency. >> >> Proposal Doc: >> https://docs.google.com/document/d/14p8Y_n4IY5n9L_I9l5x9lVGgml4ZzdCw645HldndCrw/edit?usp=sharing >> >> PTAL. >> >> Thanks, >> Yichi >> >
