Hi Xintong Thank you for your answer. This seems promising, I'll look into it.
Do you believe the code of the operators of the restarted Region can be changed between restarts? Best Benoît On Tue, Feb 25, 2020 at 2:30 AM Xintong Song <tonysong...@gmail.com> wrote: > Hi Ben, > > You can not share slots across jobs. Flink adopts a two-level slot > scheduling mechanism. Slots are firstly allocated to each job, then the > JobMaster decides which tasks should be executed in which slots, i.e. slot > sharing. > > I think what you are looking for is Pipelined Region Restart Strategy [1], > which restarts only the tasks connected by pipelined edges instead of the > whole job graph. > > Thank you~ > > Xintong Song > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/task_failure_recovery.html#restart-pipelined-region-failover-strategy > > > > On Mon, Feb 24, 2020 at 11:28 PM Benoît Paris < > benoit.pa...@centraliens-lille.org> wrote: > >> Hello all! >> >> I have a setup composed of several streaming pipelines. These have >> different deployment lifecycles: I want to be able to modify and redeploy >> the topology of one while the other is still up. I am thus putting them in >> different jobs. >> >> The problem is I have a Co-Location constraint between one subtask of >> each pipeline; I'd like them to run on the same TaskSlots, much like if >> they were sharing a TaskSlot; or at least have them on the same JVM. >> >> A semi-official feature >> "DataStream.getTransformation().setCoLocationGroupKey(stringKey)" [1] >> exists for this, but seem to be tied to the Sub-Tasks actually being able >> to be co-located on the same Task Slot. >> >> The documentation mentions [2] that it might be impossible to do ("Flink >> allows subtasks to share slots even if they are subtasks of different >> tasks, so long as they are *from the same job*"). >> >> The streaming pipelines are numerous (about 10), and I can't afford to >> increase the number of TaskSlots per TaskManager. I also would like to >> avoid putting all the pipelines in the same job, restarting it every time a >> single one changes. >> >> I'd like to have mailing list's informed opinion about this, if there are >> workarounds, or if I could reconsider my problem under another angle. >> >> Cheers >> Ben >> >> [1] >> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/translation/FlinkUniverse.java#L116 >> >> [2] >> https://ci.apache.org/projects/flink/flink-docs-master/concepts/runtime.html#task-slots-and-resources >> > -- Benoît Paris Ingénieur Machine Learning Explicable Tél : +33 6 60 74 23 00 http://benoit.paris http://explicable.ml
