On Fri, Aug 6, 2021 at 2:42 PM Reuven Lax <[email protected]> wrote: > Is there a reason you have to have the B->C be a DoFn? Could it be a > simply function or lambda? >
It's just so we can reuse existing PTransforms such as readAll. If I have a readAll DoFn that already manages redis connections/lifecycle it makes sense to me to reuse it instead of writing a custom DoFn that takes KV<A,B>. Does that make sense? > > On Fri, Aug 6, 2021 at 2:24 PM Vincent Marquez <[email protected]> > wrote: > >> I asked on user a while ago and didn't get an answer, so I'm wondering if >> anyone has an idea if this would be hard to add in a PR: >> >> Given a DoFn<B,C> and PCollection<KV<A,B>> can we make it so we can get >> back a PCollection<KV<A<C>>? Is there a way to make it generic so it works >> for PCollection<T<A,B>>? >> >> The use cases are numerous. Instead of using side inputs in a streaming >> pipeline, we can enrich some data with readAll from various sources. Itt >> would make a lot of sense to be able to reuse those readAll transforms >> instead of writing a custom DoFn to grab data from Redis, Sql, etc. This >> also seems fairly compostable as well so you could continue to enrich data >> with multiple sources. >> >> *~Vincent* >> > ~Vincent
