> > Do you believe the code of the operators of the restarted Region can be > changed between restarts?
I'm not an expert on the restart strategies, but AFAIK the answer is probably not. Sorry I overlooked that you need to modify the job. Thank you~ Xintong Song On Tue, Feb 25, 2020 at 6:00 PM Benoît Paris < benoit.pa...@centraliens-lille.org> wrote: > Hi Xintong > > Thank you for your answer. This seems promising, I'll look into it. > > Do you believe the code of the operators of the restarted Region can be > changed between restarts? > > Best > Benoît > > > On Tue, Feb 25, 2020 at 2:30 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> Hi Ben, >> >> You can not share slots across jobs. Flink adopts a two-level slot >> scheduling mechanism. Slots are firstly allocated to each job, then the >> JobMaster decides which tasks should be executed in which slots, i.e. slot >> sharing. >> >> I think what you are looking for is Pipelined Region Restart Strategy >> [1], which restarts only the tasks connected by pipelined edges instead of >> the whole job graph. >> >> Thank you~ >> >> Xintong Song >> >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/task_failure_recovery.html#restart-pipelined-region-failover-strategy >> >> >> >> On Mon, Feb 24, 2020 at 11:28 PM Benoît Paris < >> benoit.pa...@centraliens-lille.org> wrote: >> >>> Hello all! >>> >>> I have a setup composed of several streaming pipelines. These have >>> different deployment lifecycles: I want to be able to modify and redeploy >>> the topology of one while the other is still up. I am thus putting them in >>> different jobs. >>> >>> The problem is I have a Co-Location constraint between one subtask of >>> each pipeline; I'd like them to run on the same TaskSlots, much like if >>> they were sharing a TaskSlot; or at least have them on the same JVM. >>> >>> A semi-official feature >>> "DataStream.getTransformation().setCoLocationGroupKey(stringKey)" [1] >>> exists for this, but seem to be tied to the Sub-Tasks actually being able >>> to be co-located on the same Task Slot. >>> >>> The documentation mentions [2] that it might be impossible to do ("Flink >>> allows subtasks to share slots even if they are subtasks of different >>> tasks, so long as they are *from the same job*"). >>> >>> The streaming pipelines are numerous (about 10), and I can't afford to >>> increase the number of TaskSlots per TaskManager. I also would like to >>> avoid putting all the pipelines in the same job, restarting it every time a >>> single one changes. >>> >>> I'd like to have mailing list's informed opinion about this, if there >>> are workarounds, or if I could reconsider my problem under another angle. >>> >>> Cheers >>> Ben >>> >>> [1] >>> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/translation/FlinkUniverse.java#L116 >>> >>> [2] >>> https://ci.apache.org/projects/flink/flink-docs-master/concepts/runtime.html#task-slots-and-resources >>> >> > > -- > Benoît Paris > Ingénieur Machine Learning Explicable > Tél : +33 6 60 74 23 00 > http://benoit.paris > http://explicable.ml >
