+1 from my side too :-) And ideally I would want to have some hooks to let me know the extra partitions have been picked up (or a way to query it).
Although if that can't be provided I can work around it myself by sending some specific message to the partition that somewhere results in a visible state change in the pipeline. Also, as a quick (semi related) heads up: I will very likely soon contribute a change to the LogAppendTimePolicy so that the idle partition behavior (automatic watermark generation) can be disabled. (of course all related to my streamy-db project) Kind regards, Jan On Tue, 1 Jan 2019 at 08:19, Ramesh Nethi <ramesh.ne...@gmail.com> wrote: > +1 for this capability. This would enable pipelines to continue to run > when such changes need to be made. > > regards > Ramesh > > On Fri, 23 Nov 2018 at 00:40 Raghu Angadi <rang...@google.com> wrote: > >> On Thu, Nov 22, 2018 at 10:10 AM Raghu Angadi <rang...@google.com> wrote: >> >>> - New partitions will be ignored during runtime. >>> - Update will not succeed either. Error message on the workers should >>> explain the mismatch. >>> >> >> This is the current state. Supporting changes to number of partition is >> quite doable if there is enough user interested (even in the current >> UnnoundedSource API framework). >> >>> >>> On Thu, Nov 22, 2018 at 2:15 AM Jozef Vilcek <jozo.vil...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> just wanted to check how does Beam KafkaIO behaves when partitions are >>>> added to the topic. >>>> Will they be picked up or ignored during the runtime? >>>> Will they be picked up on restart with state restore? >>>> >>>> Thanks, >>>> Jozef >>>> >>>
