Most components are written multi-job aware. The only thing that is not in there right now is scheduling policies for fair resource sharing. This is important in shared clusters.
Since YARN implements all those things (various job queues with different priorities/policies etc), I suggest to not try and re-build it in Flink and simply declare a JobManager a "single-job-at-a-time" manager. You can still run an interactive session with many jobs one after another. On Wed, Apr 29, 2015 at 7:07 PM, Maximilian Michels <m...@apache.org> wrote: > > > > However, dropping it completely instead of improving it would make Flink > > setups on dedicated clusters quite useless, right? > > > > Not really, because you could also use YARN on dedicated clusters for > proper multi-user support. > > On Wed, Apr 29, 2015 at 5:51 PM, Fabian Hueske <fhue...@gmail.com> wrote: > > > I agree that Flink's multi-user support is not very good at the moment. > > However, dropping it completely instead of improving it would make Flink > > setups on dedicated clusters quite useless, right? > > > > > > 2015-04-29 17:33 GMT+02:00 Maximilian Michels <m...@apache.org>: > > > > > Hi everyone, > > > > > > Currently Flink accepts jobs from multiple clients and executes them > > > concurrently if the resource limits are not exceeded. However, the > > > multi-user support is very poor. We don't support queuing of jobs and > > > concurrent jobs have to share resources in a nice way. Otherwise, jobs > > will > > > fail. > > > > > > Using YARN, we circumvent these problems because it provides a proper > > user > > > and session management. I'm wondering now, should we get rid of the > > pseudo > > > multi-user mode and just support one user per Flink cluster instance? > > > > > > Best, > > > Max > > > > > > PS: > > > This question came up when I was working on a pull request to support > > > backtracking intermediate results. I need to hold a copy of the full > > > previous execution graph to resume from old results. With multiple > users, > > > we have to build in some kind of session management to archive old > > > execution graphs. Otherwise, they will consume too much memory in the > job > > > manager. > > > > > >
