On Wed, Mar 24, 2021 at 10:58 AM Alexey Romanenko <aromanenko....@gmail.com> wrote:
> Robert, could you elaborate a bit why or point me out if it was already > discussed? > I will speak to this from a Dataflow perspective. As Alex pointed out above Dataflow's flex templates allows for a more flexible (:p) way of creating templates and is most ways supersedes Dataflow templates. (There is a comparison here [1]). One advantage is that there is no more a need for propagating ValueProvider across the code base in order to support Dataflow's template mechanism. I might be missing context on how ValueProvider's are used by other runners and it might still be useful in those cases. My understanding was ValueProvider was only supported by Dataflow but I might be wrong. [1] https://cloud.google.com/dataflow/docs/concepts/dataflow-templates#evaluating-which-template-type-to-use > > On 24 Mar 2021, at 00:11, Robert Bradshaw <rober...@google.com> wrote: > > I would encourage flex templates over further proliferation of > ValueProviders. > > On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >> I think you are right - now with SDF support in KafkaIO it should be >> possible to determine the number of splits in the runtime and support >> ValueProviders in that way. >> >> CC: Boyuan Zhang >> >> On 23 Mar 2021, at 18:18, Vincent Marquez <vincent.marq...@gmail.com> >> wrote: >> >> Hello. I was looking at how to use the KafkaIO from a Dataflow Template, >> which requires all configuration options to be ValueProviders, which >> KafkaIO doesn't support currently. I saw this old PR: >> https://github.com/apache/beam/pull/6636/files >> >> I believe the reason this was never merged was there wasn't a good way to >> determine the number of splits to fire up in the UnboundedSource for >> KafkaIO. However, now that KafkaIO is based on SplittableDoFn which >> handles splitting in a dynamic fashion, are there still any blockers to >> this? >> >> Could we change all the KafkaIO parameters to ValueProviders now? >> >> *~Vincent* >> >> >> >
