*~Vincent*
On Fri, Aug 6, 2021 at 3:33 PM Robert Bradshaw <[email protected]> wrote: > For a one-off wrapper, you could manually instantiate and invoke the > wrapped DoFn methods (setup, startBundle, process, etc.) Even if I knew which process to call, is the idea i'd write my own OutputReceiver and somehow read out all outputs? That doesn't seem very easy to do. > Unfortunately > doing this generically in Java is harder to do due to the complexity > of Java's DoFn invoking logic (see, e.g. > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java > and related classes). > > On Fri, Aug 6, 2021 at 3:18 PM Vincent Marquez > <[email protected]> wrote: > > > > > > > > On Fri, Aug 6, 2021 at 2:42 PM Reuven Lax <[email protected]> wrote: > >> > >> Is there a reason you have to have the B->C be a DoFn? Could it be a > simply function or lambda? > > > > > > It's just so we can reuse existing PTransforms such as readAll. If I > have a readAll DoFn that already manages redis connections/lifecycle it > makes sense to me to reuse it instead of writing a custom DoFn that takes > KV<A,B>. > > > > Does that make sense? > > > > > > > > > >> > >> > >> On Fri, Aug 6, 2021 at 2:24 PM Vincent Marquez < > [email protected]> wrote: > >>> > >>> I asked on user a while ago and didn't get an answer, so I'm wondering > if anyone has an idea if this would be hard to add in a PR: > >>> > >>> Given a DoFn<B,C> and PCollection<KV<A,B>> can we make it so we can > get back a PCollection<KV<A<C>>? Is there a way to make it generic so it > works for PCollection<T<A,B>>? > >>> > >>> The use cases are numerous. Instead of using side inputs in a > streaming pipeline, we can enrich some data with readAll from various > sources. Itt would make a lot of sense to be able to reuse those readAll > transforms instead of writing a custom DoFn to grab data from Redis, Sql, > etc. This also seems fairly compostable as well so you could continue to > enrich data with multiple sources. > >>> > >>> ~Vincent > > > > > > ~Vincent > > >
