rmetzger commented on a change in pull request #14258: URL: https://github.com/apache/flink/pull/14258#discussion_r532570786
########## File path: docs/deployment/resource-providers/mesos.md ########## @@ -84,113 +77,127 @@ of failures. If you don't have a running DC/OS cluster, please follow the [instructions on how to install DC/OS on the official website](https://dcos.io/install/). -Once you have a DC/OS cluster, you may install Flink through the DC/OS -Universe. In the search prompt, just search for Flink. Alternatively, you can use the DC/OS CLI: +Once you have a DC/OS cluster, you may install Flink through the DC/OS Universe. Just +search for Flink in the search prompt. Alternatively, you can use the DC/OS CLI: - dcos package install flink +{% highlight bash %} +dcos package install flink +{% endhighlight %} Further information can be found in the -[DC/OS examples documentation](https://github.com/dcos/examples/tree/master/1.8/flink). - - -## Mesos without DC/OS - -You can also run Mesos without DC/OS. - -### Installing Mesos - -Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/getting-started/). - -After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. -These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). +[DC/OS examples documentation](https://github.com/dcos/examples/tree/master/flink). -Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. -In this file, you have to define +#### Mesos without DC/OS - export MESOS_work_dir=WORK_DIRECTORY +You can also run Mesos without DC/OS. Please follow the +[instructions on how to setup Mesos on the official website](http://mesos.apache.org/getting-started/). -and it is recommended to uncommment +After installing Mesos into `${MESOS_HOME}`, you have to configure the set of master and +agent nodes by creating the files `${MESOS_HOME}/etc/mesos/masters` and +`${MESOS_HOME}/etc/mesos/slaves`. These files contain a single hostname per row pointing to the +hosts executing the respective components (assuming SSH access being available for these nodes). - export MESOS_log_dir=LOGGING_DIRECTORY +Next, you have to create `${MESOS_HOME}/etc/mesos/mesos-master-env.sh` (a template file is +provided in the same directory). In `mesos-master-env.sh`, you have to set `${MESOS_work_dir}` +pointing to a directory used by Mesos to store data. Additionally, it is recommended to specify +`${MESOS_log_dir}`: +{% highlight bash %} +export MESOS_work_dir=/path/to/mesos/working/directory +export MESOS_log_dir=/path/to/mesos/logging/directory +{% endhighlight %} -In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. -You have to configure - - export MESOS_master=MASTER_HOSTNAME:MASTER_PORT - -and uncomment - - export MESOS_log_dir=LOGGING_DIRECTORY - export MESOS_work_dir=WORK_DIRECTORY - -#### Mesos Library +The Mesos agents are configured by creating `${MESOS_HOME}/etc/mesos/mesos-agent-env.sh` (a +template file is provided in the same directory). `${MESOS_master}` needs to be configured as part +of `mesos-agent-env.sh`. Additionally, `${MESOS_work_dir}` and `${MESOS_log_dir}` should be set to +the corresponding folders: -In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so` on Linux. -Under Mac OS X you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.dylib`. +{% highlight bash %} +export MESOS_master=MASTER_HOSTNAME:MASTER_PORT +export MESOS_work_dir=/path/to/mesos/working/directory +export MESOS_log_dir=/path/to/mesos/logging/directory +{% endhighlight %} -#### Deploying Mesos +In order to start your Mesos cluster, use the deployment script `${MESOS_HOME}/sbin/mesos-start-cluster.sh`. +Stopping the Mesos cluster can be achieved through `${MESOS_HOME}/sbin/mesos-stop-cluster.sh`. -In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. -In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/). -### Installing Marathon - -Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which enables you to run Flink in [high availability (HA) mode](#high-availability). +#### Pre-installing Flink vs Docker/Mesos containers -### Pre-installing Flink vs Docker/Mesos containers +You may install Flink on all of your Mesos Master and Agent nodes. You can also pull the binaries +from the Flink website during deployment and apply your custom configuration before launching the +application master. A more convenient and easier-to-maintain approach is to use Docker containers +to manage the Flink binaries and configuration. -You may install Flink on all of your Mesos Master and Agent nodes. -You can also pull the binaries from the Flink web site during deployment and apply your custom configuration before launching the application master. -A more convenient and easier to maintain approach is to use Docker containers to manage the Flink binaries and configuration. - -This is controlled via the following configuration entries: +This is controlled via the Flink configuration entries (`mesos` being the default): mesos.resourcemanager.tasks.container.type: mesos _or_ docker -If set to 'docker', specify the image name: +If the type is set to 'docker', specify the image name: mesos.resourcemanager.tasks.container.image.name: image_name +### Starting a Flink Session on Mesos -### Flink session cluster on Mesos - -A Flink session cluster is executed as a long-running Mesos Deployment. Note that you can run multiple Flink jobs on a session cluster. Each job needs to be submitted to the cluster after the cluster has been deployed. +A Flink session cluster is a long-running Mesos deployment. Note that you can run multiple Flink +jobs on a session cluster. Each job needs to be submitted to the cluster after the cluster has been +deployed. -In the `/bin` directory of the Flink distribution, you find two startup scripts -which manage the Flink processes in a Mesos cluster: +You find two startup scripts which manage the Flink processes in a Mesos cluster under +`${FLINK_HOME}/bin/`: 1. `mesos-appmaster.sh` - This starts the Mesos application master which will register the Mesos scheduler. + This script starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. 2. `mesos-taskmanager.sh` - The entry point for the Mesos worker processes. - You don't need to explicitly execute this script. - It is automatically launched by the Mesos worker node to bring up a new TaskManager. + This script is the entrypoint for the Mesos worker processes. You don't need to explicitly + execute this script. It is automatically launched by the Mesos worker node to bring up a + new TaskManager. + +For Flink on Mesos to work properly, you have to export `${MESOS_NATIVE_JAVA_LIBRARY}` pointing to +Mesos' native Java library. The location of this library differs depending on the underlying system: +- Linux: `export MESOS_NATIVE_JAVA_LIBRARY=${MESOS_HOME}/lib/libmesos.so` +- MacOS: `export MESOS_NATIVE_JAVA_LIBRARY=${MESOS_HOME}/lib/libmesos.dylib` + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the +`flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. It is possible to +completely parameterize a Mesos application through Java properties passed to the Mesos application +master: +{% highlight bash %} +${FLINK_HOME}/bin/mesos-appmaster.sh \ + -Dmesos.master=localhost:5050 \ + -Dparallelism.default=10 \ + -Dtaskmanager.numberOfTaskSlots=2 +{% endhighlight %} +This will create a JobManager on the machine that is used for executing the script. +In contrast to that, the TaskManagers will run as Mesos tasks in the Mesos cluster. -In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +## Deployment Modes Supported by Flink on Mesos -When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. -In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. +For production use, we recommend deploying Flink Applications in the +[Per-Job]({% link deployment/index.md %}#per-job-mode), as this mode provides a better isolation Review comment: ```suggestion [Per-Job]({% link deployment/index.md %}#per-job-mode) mode, as this mode provides a better isolation ``` ########## File path: docs/deployment/resource-providers/mesos.md ########## @@ -84,113 +77,127 @@ of failures. If you don't have a running DC/OS cluster, please follow the [instructions on how to install DC/OS on the official website](https://dcos.io/install/). -Once you have a DC/OS cluster, you may install Flink through the DC/OS -Universe. In the search prompt, just search for Flink. Alternatively, you can use the DC/OS CLI: +Once you have a DC/OS cluster, you may install Flink through the DC/OS Universe. Just +search for Flink in the search prompt. Alternatively, you can use the DC/OS CLI: - dcos package install flink +{% highlight bash %} +dcos package install flink +{% endhighlight %} Further information can be found in the -[DC/OS examples documentation](https://github.com/dcos/examples/tree/master/1.8/flink). - - -## Mesos without DC/OS - -You can also run Mesos without DC/OS. - -### Installing Mesos - -Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/getting-started/). - -After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. -These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). +[DC/OS examples documentation](https://github.com/dcos/examples/tree/master/flink). -Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. -In this file, you have to define +#### Mesos without DC/OS - export MESOS_work_dir=WORK_DIRECTORY +You can also run Mesos without DC/OS. Please follow the +[instructions on how to setup Mesos on the official website](http://mesos.apache.org/getting-started/). -and it is recommended to uncommment +After installing Mesos into `${MESOS_HOME}`, you have to configure the set of master and +agent nodes by creating the files `${MESOS_HOME}/etc/mesos/masters` and +`${MESOS_HOME}/etc/mesos/slaves`. These files contain a single hostname per row pointing to the +hosts executing the respective components (assuming SSH access being available for these nodes). - export MESOS_log_dir=LOGGING_DIRECTORY +Next, you have to create `${MESOS_HOME}/etc/mesos/mesos-master-env.sh` (a template file is +provided in the same directory). In `mesos-master-env.sh`, you have to set `${MESOS_work_dir}` +pointing to a directory used by Mesos to store data. Additionally, it is recommended to specify +`${MESOS_log_dir}`: +{% highlight bash %} +export MESOS_work_dir=/path/to/mesos/working/directory +export MESOS_log_dir=/path/to/mesos/logging/directory +{% endhighlight %} -In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. -You have to configure - - export MESOS_master=MASTER_HOSTNAME:MASTER_PORT - -and uncomment - - export MESOS_log_dir=LOGGING_DIRECTORY - export MESOS_work_dir=WORK_DIRECTORY - -#### Mesos Library +The Mesos agents are configured by creating `${MESOS_HOME}/etc/mesos/mesos-agent-env.sh` (a +template file is provided in the same directory). `${MESOS_master}` needs to be configured as part +of `mesos-agent-env.sh`. Additionally, `${MESOS_work_dir}` and `${MESOS_log_dir}` should be set to +the corresponding folders: -In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so` on Linux. -Under Mac OS X you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.dylib`. +{% highlight bash %} +export MESOS_master=MASTER_HOSTNAME:MASTER_PORT +export MESOS_work_dir=/path/to/mesos/working/directory +export MESOS_log_dir=/path/to/mesos/logging/directory +{% endhighlight %} -#### Deploying Mesos +In order to start your Mesos cluster, use the deployment script `${MESOS_HOME}/sbin/mesos-start-cluster.sh`. +Stopping the Mesos cluster can be achieved through `${MESOS_HOME}/sbin/mesos-stop-cluster.sh`. -In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. -In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/). -### Installing Marathon - -Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which enables you to run Flink in [high availability (HA) mode](#high-availability). +#### Pre-installing Flink vs Docker/Mesos containers -### Pre-installing Flink vs Docker/Mesos containers +You may install Flink on all of your Mesos Master and Agent nodes. You can also pull the binaries +from the Flink website during deployment and apply your custom configuration before launching the +application master. A more convenient and easier-to-maintain approach is to use Docker containers +to manage the Flink binaries and configuration. -You may install Flink on all of your Mesos Master and Agent nodes. -You can also pull the binaries from the Flink web site during deployment and apply your custom configuration before launching the application master. -A more convenient and easier to maintain approach is to use Docker containers to manage the Flink binaries and configuration. - -This is controlled via the following configuration entries: +This is controlled via the Flink configuration entries (`mesos` being the default): mesos.resourcemanager.tasks.container.type: mesos _or_ docker -If set to 'docker', specify the image name: +If the type is set to 'docker', specify the image name: mesos.resourcemanager.tasks.container.image.name: image_name +### Starting a Flink Session on Mesos -### Flink session cluster on Mesos - -A Flink session cluster is executed as a long-running Mesos Deployment. Note that you can run multiple Flink jobs on a session cluster. Each job needs to be submitted to the cluster after the cluster has been deployed. +A Flink session cluster is a long-running Mesos deployment. Note that you can run multiple Flink +jobs on a session cluster. Each job needs to be submitted to the cluster after the cluster has been +deployed. -In the `/bin` directory of the Flink distribution, you find two startup scripts -which manage the Flink processes in a Mesos cluster: +You find two startup scripts which manage the Flink processes in a Mesos cluster under +`${FLINK_HOME}/bin/`: 1. `mesos-appmaster.sh` - This starts the Mesos application master which will register the Mesos scheduler. + This script starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. 2. `mesos-taskmanager.sh` - The entry point for the Mesos worker processes. - You don't need to explicitly execute this script. - It is automatically launched by the Mesos worker node to bring up a new TaskManager. + This script is the entrypoint for the Mesos worker processes. You don't need to explicitly + execute this script. It is automatically launched by the Mesos worker node to bring up a + new TaskManager. + +For Flink on Mesos to work properly, you have to export `${MESOS_NATIVE_JAVA_LIBRARY}` pointing to +Mesos' native Java library. The location of this library differs depending on the underlying system: +- Linux: `export MESOS_NATIVE_JAVA_LIBRARY=${MESOS_HOME}/lib/libmesos.so` +- MacOS: `export MESOS_NATIVE_JAVA_LIBRARY=${MESOS_HOME}/lib/libmesos.dylib` + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the +`flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. It is possible to +completely parameterize a Mesos application through Java properties passed to the Mesos application +master: +{% highlight bash %} +${FLINK_HOME}/bin/mesos-appmaster.sh \ + -Dmesos.master=localhost:5050 \ + -Dparallelism.default=10 \ + -Dtaskmanager.numberOfTaskSlots=2 +{% endhighlight %} +This will create a JobManager on the machine that is used for executing the script. +In contrast to that, the TaskManagers will run as Mesos tasks in the Mesos cluster. -In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +## Deployment Modes Supported by Flink on Mesos -When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. -In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. +For production use, we recommend deploying Flink Applications in the +[Per-Job]({% link deployment/index.md %}#per-job-mode), as this mode provides a better isolation +for each job. -### Flink job cluster on Mesos +### Application Mode -A Flink job cluster is a dedicated cluster which runs a single job. -There is no extra job submission needed. +Flink on Mesos does not support Application Mode. -In the `/bin` directory of the Flink distribution, you find one startup script -which manage the Flink processes in a Mesos cluster: +### Per-Job Cluster Mode -1. `mesos-appmaster-job.sh` - This starts the Mesos application master which will register the Mesos scheduler, retrieve the job graph and then launch the task managers accordingly. +A job which is executed in Per-Job Cluster Mode spins up a dedicated Flink cluster that only being Review comment: ```suggestion A job which is executed in Per-Job Cluster Mode spins up a dedicated Flink cluster that is only being ``` ########## File path: docs/deployment/resource-providers/mesos.md ########## @@ -26,55 +26,48 @@ under the License. * This will be replaced by the TOC {:toc} -## Background +## Getting Started Review comment: In my mind, the `Getting Started` section should be an opinionated (make choices for the user) and short (ideally doable in 15 minutes) path to having a running Flink cluster. In particular, I would shorten the Introduction by removing the detailed description of the components, and move them to the reference. Secondly, I would only describe how to set up on dcos, and again move the manual setup to the reference. This way, it'll just be a few commands and little text to read to success. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
