Hi Ben, You can not share slots across jobs. Flink adopts a two-level slot scheduling mechanism. Slots are firstly allocated to each job, then the JobMaster decides which tasks should be executed in which slots, i.e. slot sharing.
I think what you are looking for is Pipelined Region Restart Strategy [1], which restarts only the tasks connected by pipelined edges instead of the whole job graph. Thank you~ Xintong Song [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/task_failure_recovery.html#restart-pipelined-region-failover-strategy On Mon, Feb 24, 2020 at 11:28 PM Benoît Paris < benoit.pa...@centraliens-lille.org> wrote: > Hello all! > > I have a setup composed of several streaming pipelines. These have > different deployment lifecycles: I want to be able to modify and redeploy > the topology of one while the other is still up. I am thus putting them in > different jobs. > > The problem is I have a Co-Location constraint between one subtask of each > pipeline; I'd like them to run on the same TaskSlots, much like if they > were sharing a TaskSlot; or at least have them on the same JVM. > > A semi-official feature > "DataStream.getTransformation().setCoLocationGroupKey(stringKey)" [1] > exists for this, but seem to be tied to the Sub-Tasks actually being able > to be co-located on the same Task Slot. > > The documentation mentions [2] that it might be impossible to do ("Flink > allows subtasks to share slots even if they are subtasks of different > tasks, so long as they are *from the same job*"). > > The streaming pipelines are numerous (about 10), and I can't afford to > increase the number of TaskSlots per TaskManager. I also would like to > avoid putting all the pipelines in the same job, restarting it every time a > single one changes. > > I'd like to have mailing list's informed opinion about this, if there are > workarounds, or if I could reconsider my problem under another angle. > > Cheers > Ben > > [1] > https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/translation/FlinkUniverse.java#L116 > > [2] > https://ci.apache.org/projects/flink/flink-docs-master/concepts/runtime.html#task-slots-and-resources >
