On Wed, Mar 28, 2018 at 3:14 AM, Niclas Hedhman <nic...@apache.org> wrote:
> Hi, > > is there some document (or presentation) that explains the internals of > how a Job gets deployed on to the cluster? Communications, Classloading and > Serialization (if any) are the key points here I think. > I don't know of any specific presentations, but data artisans provide http://training.data-artisans.com/system-overview.html which are pretty good. The Flink documentation is comprehensive. Class-loading: https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html State serialization: https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/custom_serialization.html > > I suspect that my application modeling framework is incompatible with the > standard Flink mechanism, and I would like to learn how much effort there > is to make my own mechanism (assuming it is possible, since Yarn and Mesos > are in similar situation) > Don't know what you mean by application "modeling" framework, but if you mean that you have a Flink app (batch or streaming) that you'd want to deploy to YARN (or Mesos, which is similar), then the flow appears to be 1- Create a "Flink Cluster" (also called a YARN session) when a user does "bin/yarn-session.sh <params>" and then 2- Run the app when the user does "bin/flink run <app-class> <app-jar>". It's the user's responsibility to shut down the cluster (YARN session) by sending a "stop" command to the YARN session created in 1). The code appears to be in classes like org.apache.flink.yarn.cli.FlinkYarnSessionCli (manage the YARN session) and org.apache.flink.client.CliFrontend (submit a Flink app to the YARN session). Regards, Kedar > > Thanks in Advance > -- > Niclas Hedhman, Software Developer > http://zest.apache.org - New Energy for Java >
