Robert, could you elaborate a bit why or point me out if it was already discussed?
> On 24 Mar 2021, at 00:11, Robert Bradshaw <rober...@google.com> wrote: > > I would encourage flex templates over further proliferation of > ValueProviders. > > On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko <aromanenko....@gmail.com > <mailto:aromanenko....@gmail.com>> wrote: > I think you are right - now with SDF support in KafkaIO it should be possible > to determine the number of splits in the runtime and support ValueProviders > in that way. > > CC: Boyuan Zhang > >> On 23 Mar 2021, at 18:18, Vincent Marquez <vincent.marq...@gmail.com >> <mailto:vincent.marq...@gmail.com>> wrote: >> >> Hello. I was looking at how to use the KafkaIO from a Dataflow Template, >> which requires all configuration options to be ValueProviders, which KafkaIO >> doesn't support currently. I saw this old PR: >> https://github.com/apache/beam/pull/6636/files >> <https://github.com/apache/beam/pull/6636/files> >> >> I believe the reason this was never merged was there wasn't a good way to >> determine the number of splits to fire up in the UnboundedSource for >> KafkaIO. However, now that KafkaIO is based on SplittableDoFn which handles >> splitting in a dynamic fashion, are there still any blockers to this? >> >> Could we change all the KafkaIO parameters to ValueProviders now? >> >> ~Vincent >
