I agree with Max since it hard to foresee when Flip-6 is really done. Cheers, Till
On Wed, Sep 28, 2016 at 4:24 PM, Maximilian Michels <m...@apache.org> wrote: > Hi Eron, > > Great to see so much progress on the Mesos implementation! Thank you > for sharing the code with us. > > I'm not entirely sure whether we actually wait on the completion of > FLIP-6. We might complete the Mesos support for the 1.2.0 release and > port its code to the new RPC abstraction that comes with FLIP-6 > afterwards. That is still to decide. It is probably also acceptable to > have the new RPC layer and still use direct Akka code for Mesos. > Client changes should be relatively independent of FLIP-6 and be made > in such way that they are easily portable to FLIP-6. > > All in all, we are probably best of with merging your changes to the > master quickly since Mesos support will be one of the most important > new features of Flink 1.2.0. > > Best, > Max > > > > > On Mon, Sep 26, 2016 at 11:37 PM, Wright, Eron <ewri...@live.com> wrote: > > > > Hello, > > > > The code I presented at FF2016 represents a 'status-quo' approach to > realizing a specific scenario - "mesos-session.sh". But the final > solution will involve CLI changes and the full realization of a dispatcher, > which conflicts with FLIP-6. We should advance the client/dispatcher > design of FLIP-6 before I make further changes. Also we must decide > whether the Mesos work should take FLIP-6 as a dependency, and/or strive to > land a useable subset into master. > > > > Here's a summary of the code I presented, as reference for ongoing > FLIP-6 design discussion. A fresh PR for convenience: > > https://github.com/EronWright/flink/pull/1/files > > > > 1. MesosDispatcher (Backend). The 'backend' acts as a Mesos framework, > to launch an AppMaster as a Mesos task. For each session, the backend > accepts "session parameters" which define the libraries, configuration, > resource profile and other parameters needed to launch the AppMaster. > The backend persists the information for recovery purposes. Leader > election is also used. Not included is a dispatcher 'frontend' - a > formalized API surface or REST server. > > > > 2. SessionParameters. Captures the information needed to launch a > session based on an AppMaster. A session is a stateful execution > environment for a program. The session parameters can also be understood > as the historical inputs to yarn-session.sh, plus job inputs per FLIP-6. > I acknowledge that the term 'session' is a working term. > > > > 3 . MesosDispatcherRunner. The runner for the 'remote dispatcher' > scenario, which would be started by Marathon, expose a REST API with which > to submit/manage jobs, and host the above dispatcher backend. > > > > 4. FlinkMesosSessionCli. This class mirrors the FlinkYarnSessionCli, > which is used in both the 'flink run' scenario and 'yarn-session' > scenario. I didn't fully implement the CustomCommandLine interface which > yields a ClusterClient, because the dispatcher API must be fleshed out > first. > > > > 5. SessionArtifactHelper. An attempt to consolidate logic related to > session artifacts (i.e. ship files). > > > > 6. DispatcherClient. The client interface for the dispatcher. In > concept there could be numerous implementations - a > > 'remote' impl which would make REST calls to a remote dispatcher, a > 'local' impl which would host the dispatcher directly. Seen in this PR is > only the latter but it might be throw-away code. > > > > 7. LaunchableMesosSession. This class generates the effective > container environment at launch time. > > > > 8. ContaineredJobMasterParameters. Refactored from YARN code for > sharing purposes. > > > > -Eron >
