Hey Marton! Sorry for the late answer.
I think this would be a cool feature. It would definitely require work in the representation and transfer of intermediate results. Ufuk is reworking those. He is currently on vacation and will probably comment when he is back. We can definitely keep that in mind when working on the network stack, though I feel we have to wait for the base functionality there to be in place. A few questions on how that should work: - Would this have to be a coordinated action across workers, or can workers individually do that? - How does it work with stateful operators? The state needs repartitioning as well. - Do all workers need to quiesce their inputs, and trigger a finer partitioning? Or do you partition rather fine before, and change the assignment of worker on partitions? Greetings, Stephan On Tue, Aug 19, 2014 at 12:43 PM, Márton Balassi <[email protected]> wrote: > Hey, > > Any comments on this please? :) > > Thanks. > > > On Tue, Aug 12, 2014 at 8:10 AM, Márton Balassi <[email protected]> > wrote: > > > Hi, > > > > Apache Storm has this cool feature that one can "rebalance a topology" > > during runtime to handle the changes in the streaming workload. > Practically > > this means adjusting the number of worker machines and the parallelism of > > the spouts and bolts. > > > > Translated to Flink terms this would mean changing the numberOfSubtasks > > field of the AbstractJobVertices of the JobGraph during runtime (amongst > > others, e.g. updating the numberOfOutputChannels in the partitioners > > accordingly). > > > > Is there any support for achieving this currently? > > > > Cheers, > > > > Marton > > > > >
