Re: [DISCUSS] Mesos Dispatcher (FF2016)

[Till Rohrmann]

I agree with Max since it hard to foresee when Flip-6 is really done.

Cheers,
Till

On Wed, Sep 28, 2016 at 4:24 PM, Maximilian Michels  wrote:

> Hi Eron,
>
> Great to see so much progress on the Mesos implementation! Thank you
> for sharing the code with us.
>
> I'm not entirely sure whether we actually wait on the completion of
> FLIP-6. We might complete the Mesos support for the 1.2.0 release and
> port its code to the new RPC abstraction that comes with FLIP-6
> afterwards. That is still to decide. It is probably also acceptable to
> have the new RPC layer and still use direct Akka code for Mesos.
> Client changes should be relatively independent of FLIP-6 and be made
> in such way that they are easily portable to FLIP-6.
>
> All in all, we are probably best of with merging your changes to the
> master quickly since Mesos support will be one of the most important
> new features of Flink 1.2.0.
>
> Best,
> Max
>
>
>
>
> On Mon, Sep 26, 2016 at 11:37 PM, Wright, Eron  wrote:
> >
> >  Hello,
> >
> > The code I presented at FF2016 represents a 'status-quo' approach to
> realizing a specific scenario - "mesos-session.sh".   But the final
> solution will involve CLI changes and the full realization of a dispatcher,
> which conflicts with FLIP-6.   We should advance the client/dispatcher
> design of FLIP-6 before I make further changes.Also we must decide
> whether the Mesos work should take FLIP-6 as a dependency, and/or strive to
> land a useable subset into master.
> >
> > Here's a summary of the code I presented, as reference for ongoing
> FLIP-6 design discussion.  A fresh PR for convenience:
> > https://github.com/EronWright/flink/pull/1/files
> >
> > 1. MesosDispatcher (Backend).   The 'backend' acts as a Mesos framework,
> to launch an AppMaster as a Mesos task.   For each session, the backend
> accepts "session parameters" which define the libraries, configuration,
> resource profile and other parameters needed to launch the AppMaster.
> The backend persists the information for recovery purposes.Leader
> election is also used.   Not included is a dispatcher 'frontend' - a
> formalized API surface or REST server.
> >
> > 2. SessionParameters.   Captures the information needed to launch a
> session based on an AppMaster.A session is a stateful execution
> environment for a program.   The session parameters can also be understood
> as the historical inputs to yarn-session.sh, plus job inputs per FLIP-6.
>  I acknowledge that the term 'session' is a working term.
> >
> > 3 . MesosDispatcherRunner.  The runner for the 'remote dispatcher'
> scenario, which would be started by Marathon, expose a REST API with which
> to submit/manage jobs, and host the above dispatcher backend.
> >
> > 4. FlinkMesosSessionCli.   This class mirrors the FlinkYarnSessionCli,
> which is used in both the 'flink run' scenario and 'yarn-session'
> scenario.I didn't fully implement the CustomCommandLine interface which
> yields a ClusterClient, because the dispatcher API must be fleshed out
> first.
> >
> > 5. SessionArtifactHelper.   An attempt to consolidate logic related to
> session artifacts (i.e. ship files).
> >
> > 6. DispatcherClient.   The client interface for the dispatcher.  In
> concept there could be numerous implementations - a
> > 'remote' impl which would make REST calls to a remote dispatcher, a
> 'local' impl which would host the dispatcher directly.  Seen in this PR is
> only the latter but it might be throw-away code.
> >
> > 7. LaunchableMesosSession.   This class generates the effective
> container environment at launch time.
> >
> > 8. ContaineredJobMasterParameters.  Refactored from YARN code for
> sharing purposes.
> >
> > -Eron
>

Re: [DISCUSS] Mesos Dispatcher (FF2016)

[Maximilian Michels]

Hi Eron,

Great to see so much progress on the Mesos implementation! Thank you
for sharing the code with us.

I'm not entirely sure whether we actually wait on the completion of
FLIP-6. We might complete the Mesos support for the 1.2.0 release and
port its code to the new RPC abstraction that comes with FLIP-6
afterwards. That is still to decide. It is probably also acceptable to
have the new RPC layer and still use direct Akka code for Mesos.
Client changes should be relatively independent of FLIP-6 and be made
in such way that they are easily portable to FLIP-6.

All in all, we are probably best of with merging your changes to the
master quickly since Mesos support will be one of the most important
new features of Flink 1.2.0.

Best,
Max




On Mon, Sep 26, 2016 at 11:37 PM, Wright, Eron  wrote:
>
>  Hello,
>
> The code I presented at FF2016 represents a 'status-quo' approach to 
> realizing a specific scenario - "mesos-session.sh".   But the final solution 
> will involve CLI changes and the full realization of a dispatcher, which 
> conflicts with FLIP-6.   We should advance the client/dispatcher design of 
> FLIP-6 before I make further changes.Also we must decide whether the 
> Mesos work should take FLIP-6 as a dependency, and/or strive to land a 
> useable subset into master.
>
> Here's a summary of the code I presented, as reference for ongoing FLIP-6 
> design discussion.  A fresh PR for convenience:
> https://github.com/EronWright/flink/pull/1/files
>
> 1. MesosDispatcher (Backend).   The 'backend' acts as a Mesos framework, to 
> launch an AppMaster as a Mesos task.   For each session, the backend accepts 
> "session parameters" which define the libraries, configuration, resource 
> profile and other parameters needed to launch the AppMaster.The backend 
> persists the information for recovery purposes.Leader election is also 
> used.   Not included is a dispatcher 'frontend' - a formalized API surface or 
> REST server.
>
> 2. SessionParameters.   Captures the information needed to launch a session 
> based on an AppMaster.A session is a stateful execution environment for a 
> program.   The session parameters can also be understood as the historical 
> inputs to yarn-session.sh, plus job inputs per FLIP-6.   I acknowledge that 
> the term 'session' is a working term.
>
> 3 . MesosDispatcherRunner.  The runner for the 'remote dispatcher' scenario, 
> which would be started by Marathon, expose a REST API with which to 
> submit/manage jobs, and host the above dispatcher backend.
>
> 4. FlinkMesosSessionCli.   This class mirrors the FlinkYarnSessionCli, which 
> is used in both the 'flink run' scenario and 'yarn-session' scenario.I 
> didn't fully implement the CustomCommandLine interface which yields a 
> ClusterClient, because the dispatcher API must be fleshed out first.
>
> 5. SessionArtifactHelper.   An attempt to consolidate logic related to 
> session artifacts (i.e. ship files).
>
> 6. DispatcherClient.   The client interface for the dispatcher.  In 
> concept there could be numerous implementations - a
> 'remote' impl which would make REST calls to a remote dispatcher, a 'local' 
> impl which would host the dispatcher directly.  Seen in this PR is only the 
> latter but it might be throw-away code.
>
> 7. LaunchableMesosSession.   This class generates the effective container 
> environment at launch time.
>
> 8. ContaineredJobMasterParameters.  Refactored from YARN code for sharing 
> purposes.
>
> -Eron

[DISCUSS] Mesos Dispatcher (FF2016)

[Wright, Eron]

 Hello,

The code I presented at FF2016 represents a 'status-quo' approach to realizing 
a specific scenario - "mesos-session.sh".   But the final solution will involve 
CLI changes and the full realization of a dispatcher, which conflicts with 
FLIP-6.   We should advance the client/dispatcher design of FLIP-6 before I 
make further changes.Also we must decide whether the Mesos work should take 
FLIP-6 as a dependency, and/or strive to land a useable subset into master.

Here's a summary of the code I presented, as reference for ongoing FLIP-6 
design discussion.  A fresh PR for convenience:
https://github.com/EronWright/flink/pull/1/files

1. MesosDispatcher (Backend).   The 'backend' acts as a Mesos framework, to 
launch an AppMaster as a Mesos task.   For each session, the backend accepts 
"session parameters" which define the libraries, configuration, resource 
profile and other parameters needed to launch the AppMaster.The backend 
persists the information for recovery purposes.Leader election is also 
used.   Not included is a dispatcher 'frontend' - a formalized API surface or 
REST server.

2. SessionParameters.   Captures the information needed to launch a session 
based on an AppMaster.A session is a stateful execution environment for a 
program.   The session parameters can also be understood as the historical 
inputs to yarn-session.sh, plus job inputs per FLIP-6.   I acknowledge that the 
term 'session' is a working term.

3 . MesosDispatcherRunner.  The runner for the 'remote dispatcher' scenario, 
which would be started by Marathon, expose a REST API with which to 
submit/manage jobs, and host the above dispatcher backend.

4. FlinkMesosSessionCli.   This class mirrors the FlinkYarnSessionCli, which is 
used in both the 'flink run' scenario and 'yarn-session' scenario.I didn't 
fully implement the CustomCommandLine interface which yields a ClusterClient, 
because the dispatcher API must be fleshed out first.

5. SessionArtifactHelper.   An attempt to consolidate logic related to session 
artifacts (i.e. ship files).

6. DispatcherClient.   The client interface for the dispatcher.  In concept 
there could be numerous implementations - a 
'remote' impl which would make REST calls to a remote dispatcher, a 'local' 
impl which would host the dispatcher directly.  Seen in this PR is only the 
latter but it might be throw-away code.

7. LaunchableMesosSession.   This class generates the effective container 
environment at launch time.

8. ContaineredJobMasterParameters.  Refactored from YARN code for sharing 
purposes.

-Eron