Robert, could you elaborate a bit why or point me out if it was already discussed?
> On 24 Mar 2021, at 00:11, Robert Bradshaw <[email protected]> wrote: > > I would encourage flex templates over further proliferation of > ValueProviders. > > On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko <[email protected] > <mailto:[email protected]>> wrote: > I think you are right - now with SDF support in KafkaIO it should be possible > to determine the number of splits in the runtime and support ValueProviders > in that way. > > CC: Boyuan Zhang > >> On 23 Mar 2021, at 18:18, Vincent Marquez <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello. I was looking at how to use the KafkaIO from a Dataflow Template, >> which requires all configuration options to be ValueProviders, which KafkaIO >> doesn't support currently. I saw this old PR: >> https://github.com/apache/beam/pull/6636/files >> <https://github.com/apache/beam/pull/6636/files> >> >> I believe the reason this was never merged was there wasn't a good way to >> determine the number of splits to fire up in the UnboundedSource for >> KafkaIO. However, now that KafkaIO is based on SplittableDoFn which handles >> splitting in a dynamic fashion, are there still any blockers to this? >> >> Could we change all the KafkaIO parameters to ValueProviders now? >> >> ~Vincent >
