Thanks for preparing the FLIP and starting the discussion, Robert. ## Cluster vs. Job configuration As I have commented on the FLIP-160 discussion thread [1], I'm a bit unsure about activating the reactive execution mode via a cluster level configuration option. I'm aware that in the first step this feature does not support session clusters. However, I think that does not mean it won't be supported in future. In that case, the cluster configuration option will limit us from having jobs running with different execution modes in the same session cluster.
## Active resource managers According to the FLIP, this feature explicitly does not support active resource managers. IIUC, this is because when in this feature the job requests an infinite amount of resources, which would flood Kubernetes / Yarn / Mesos with unreasonably large number of resource requests. If this is the only concern, I'd like to bring the configuration option `slotmanager.number-of-slots.max` to your attention. This feature allows putting an upper limit to the total number of slots the Flink cluster uses, preventing active resource managers from allocating too many resources from Kubernetes / Yarn / Mesos. Unless there are other concerns that I overlooked, I think it would be nicer for the reactive mode to also support active resource managers, with the additional requirement to explicitly configure the max slots. Thank you~ Xintong Song [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-160-Declarative-scheduler-td48165.html On Sat, Jan 23, 2021 at 5:59 AM Steven Wu <stevenz...@gmail.com> wrote: > Thanks a lot for the proposal, Robert and Till. > > > No fixed parallelism for any of the operators > > Regarding this limitation, can the scheduler only adjust the default > parallelism? if some operators set parallelism explicitly (like always 1), > just leave them unchanged. > > > On Fri, Jan 22, 2021 at 8:42 AM Robert Metzger <rmetz...@apache.org> > wrote: > > > Hi all, > > > > Till started a discussion about FLIP-160: Declarative scheduler [1] > earlier > > today, the first major feature based on that effort will be FLIP-159: > > Reactive Mode. It allows users to operate Flink in a way that it > reactively > > scales the job up or down depending on the provided resources: adding > > TaskManagers will scale the job up, removing them will scale it down > again. > > > > Here's the link to the Wiki: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode > > > > We are very excited to hear your feedback about the proposal! > > > > Best, > > Robert > > > > [1] > > > > > https://lists.apache.org/thread.html/r604a01f739639e2a5f093fbe7894c172125530332747ecf6990a6ce4%40%3Cdev.flink.apache.org%3E > > >
