tillrohrmann commented on a change in pull request #14258:
URL: https://github.com/apache/flink/pull/14258#discussion_r532734546
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -201,51 +208,50 @@ try (FileOutputStream output = new
FileOutputStream(jobGraphFile);
}
{% endhighlight %}
-<span class="label label-info">Note</span> Make sure that all Mesos processes
have the user code jar on the classpath. There are two ways:
+<span class="label label-info">Note</span> Make sure that all the classpath's
of all Mesos
+processes contain the reference to the user code jar. There are two ways to
achieve this:
+1. One way is to put them in the `${FLINK_HOME}/lib/` directory, which will
result in the user
+ code jar being loaded by the system classloader.
+1. Another option is to create a `${FLINK_HOME}/usrlib/` directory and to put
the user code jar
Review comment:
```suggestion
2. Another option is to create a `${FLINK_HOME}/usrlib/` directory and to
put the user code jar
```
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -84,113 +77,127 @@ of failures.
If you don't have a running DC/OS cluster, please follow the
[instructions on how to install DC/OS on the official
website](https://dcos.io/install/).
-Once you have a DC/OS cluster, you may install Flink through the DC/OS
-Universe. In the search prompt, just search for Flink. Alternatively, you can
use the DC/OS CLI:
+Once you have a DC/OS cluster, you may install Flink through the DC/OS
Universe. Just
+search for Flink in the search prompt. Alternatively, you can use the DC/OS
CLI:
- dcos package install flink
+{% highlight bash %}
+dcos package install flink
+{% endhighlight %}
Further information can be found in the
-[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/1.8/flink).
-
-
-## Mesos without DC/OS
-
-You can also run Mesos without DC/OS.
-
-### Installing Mesos
-
-Please follow the [instructions on how to setup Mesos on the official
website](http://mesos.apache.org/getting-started/).
-
-After installation you have to configure the set of master and agent nodes by
creating the files `MESOS_HOME/etc/mesos/masters` and
`MESOS_HOME/etc/mesos/slaves`.
-These files contain in each row a single hostname on which the respective
component will be started (assuming SSH access to these nodes).
+[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/flink).
Review comment:
The documentation here installs Flink 1.3.1
(https://github.com/dcos/examples/tree/master/flink/1.10). Do we really want to
promote this? If there is no newer DC/OS package, then I'd suggest to go for
the option w/o DC/OS and the latest Flink version.
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -84,113 +77,127 @@ of failures.
If you don't have a running DC/OS cluster, please follow the
[instructions on how to install DC/OS on the official
website](https://dcos.io/install/).
-Once you have a DC/OS cluster, you may install Flink through the DC/OS
-Universe. In the search prompt, just search for Flink. Alternatively, you can
use the DC/OS CLI:
+Once you have a DC/OS cluster, you may install Flink through the DC/OS
Universe. Just
+search for Flink in the search prompt. Alternatively, you can use the DC/OS
CLI:
- dcos package install flink
+{% highlight bash %}
+dcos package install flink
+{% endhighlight %}
Further information can be found in the
-[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/1.8/flink).
-
-
-## Mesos without DC/OS
-
-You can also run Mesos without DC/OS.
-
-### Installing Mesos
-
-Please follow the [instructions on how to setup Mesos on the official
website](http://mesos.apache.org/getting-started/).
-
-After installation you have to configure the set of master and agent nodes by
creating the files `MESOS_HOME/etc/mesos/masters` and
`MESOS_HOME/etc/mesos/slaves`.
-These files contain in each row a single hostname on which the respective
component will be started (assuming SSH access to these nodes).
+[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/flink).
-Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the
template found in the same directory.
-In this file, you have to define
+#### Mesos without DC/OS
- export MESOS_work_dir=WORK_DIRECTORY
+You can also run Mesos without DC/OS. Please follow the
+[instructions on how to setup Mesos on the official
website](http://mesos.apache.org/getting-started/).
-and it is recommended to uncommment
+After installing Mesos into `${MESOS_HOME}`, you have to configure the set of
master and
+agent nodes by creating the files `${MESOS_HOME}/etc/mesos/masters` and
+`${MESOS_HOME}/etc/mesos/slaves`. These files contain a single hostname per
row pointing to the
+hosts executing the respective components (assuming SSH access being available
for these nodes).
- export MESOS_log_dir=LOGGING_DIRECTORY
+Next, you have to create `${MESOS_HOME}/etc/mesos/mesos-master-env.sh` (a
template file is
+provided in the same directory). In `mesos-master-env.sh`, you have to set
`${MESOS_work_dir}`
+pointing to a directory used by Mesos to store data. Additionally, it is
recommended to specify
+`${MESOS_log_dir}`:
+{% highlight bash %}
+export MESOS_work_dir=/path/to/mesos/working/directory
+export MESOS_log_dir=/path/to/mesos/logging/directory
+{% endhighlight %}
-In order to configure the Mesos agents, you have to create
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same
directory.
-You have to configure
-
- export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
-
-and uncomment
-
- export MESOS_log_dir=LOGGING_DIRECTORY
- export MESOS_work_dir=WORK_DIRECTORY
-
-#### Mesos Library
+The Mesos agents are configured by creating
`${MESOS_HOME}/etc/mesos/mesos-agent-env.sh` (a
+template file is provided in the same directory). `${MESOS_master}` needs to
be configured as part
+of `mesos-agent-env.sh`. Additionally, `${MESOS_work_dir}` and
`${MESOS_log_dir}` should be set to
+the corresponding folders:
-In order to run Java applications with Mesos you have to export
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so` on Linux.
-Under Mac OS X you have to export
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.dylib`.
+{% highlight bash %}
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+export MESOS_work_dir=/path/to/mesos/working/directory
+export MESOS_log_dir=/path/to/mesos/logging/directory
+{% endhighlight %}
-#### Deploying Mesos
+In order to start your Mesos cluster, use the deployment script
`${MESOS_HOME}/sbin/mesos-start-cluster.sh`.
+Stopping the Mesos cluster can be achieved through
`${MESOS_HOME}/sbin/mesos-stop-cluster.sh`.
Review comment:
Is this information maybe contained in the getting started guide for
Mesos itself?
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -201,51 +208,50 @@ try (FileOutputStream output = new
FileOutputStream(jobGraphFile);
}
{% endhighlight %}
-<span class="label label-info">Note</span> Make sure that all Mesos processes
have the user code jar on the classpath. There are two ways:
+<span class="label label-info">Note</span> Make sure that all the classpath's
of all Mesos
+processes contain the reference to the user code jar. There are two ways to
achieve this:
+1. One way is to put them in the `${FLINK_HOME}/lib/` directory, which will
result in the user
+ code jar being loaded by the system classloader.
+1. Another option is to create a `${FLINK_HOME}/usrlib/` directory and to put
the user code jar
+ in that directory. The user code jar will be loaded by the user code
classloader after launching
+ a Per-Job Cluster via `${FLINK_HOME}/bin/mesos-appmaster-job.sh ...`.
Review comment:
Have you tried it out whether it works?
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -201,51 +208,50 @@ try (FileOutputStream output = new
FileOutputStream(jobGraphFile);
}
{% endhighlight %}
-<span class="label label-info">Note</span> Make sure that all Mesos processes
have the user code jar on the classpath. There are two ways:
+<span class="label label-info">Note</span> Make sure that all the classpath's
of all Mesos
+processes contain the reference to the user code jar. There are two ways to
achieve this:
+1. One way is to put them in the `${FLINK_HOME}/lib/` directory, which will
result in the user
+ code jar being loaded by the system classloader.
+1. Another option is to create a `${FLINK_HOME}/usrlib/` directory and to put
the user code jar
+ in that directory. The user code jar will be loaded by the user code
classloader after launching
+ a Per-Job Cluster via `${FLINK_HOME}/bin/mesos-appmaster-job.sh ...`.
+
+### Session Mode
-1. One way is putting them in the `lib/` directory, which will result in the
user code jar being loaded by the system classloader.
-1. The other way is creating a `usrlib/` directory in the parent directory of
`lib/` and putting the user code jar in the `usrlib/` directory.
-After launching a job cluster via `bin/mesos-appmaster-job.sh ...`, the user
code jar will be loaded by the user code classloader.
+The [Getting Started](#starting-a-flink-session-on-mesos) guide at the top of
this page describes
+deploying Flink in Session Mode.
-#### General configuration
+## Flink on Mesos Reference
-It is possible to completely parameterize a Mesos application through Java
properties passed to the Mesos application master.
-This also allows to specify general Flink configuration parameters.
-For example:
+### Configuring Flink on Mesos
- bin/mesos-appmaster.sh \
- -Dmesos.master=master.foobar.org:5050 \
- -Djobmanager.memory.process.size=1472m \
- -Djobmanager.rpc.port=6123 \
- -Drest.port=8081 \
- -Dtaskmanager.memory.process.size=3500m \
- -Dtaskmanager.numberOfTaskSlots=2 \
- -Dparallelism.default=10
+For a list of Mesos-specific configuration parameters, refer to the
+[Mesos section]({% link deployment/config.md %}#mesos) of the configuration
documentation.
-### High Availability
+### High Availability on Mesos
-You will need to run a service like Marathon or Apache Aurora which takes care
of restarting the JobManager process in case of node or process failures.
-In addition, Zookeeper needs to be configured like described in the [High
Availability section of the Flink docs]({% link deployment/ha/index.md %}).
+You will need to run a service like Marathon or Apache Aurora which takes care
of restarting the
+JobManager process in case of node or process failures. In addition, Zookeeper
needs to be
+configured as described in the
+[High Availability section of the Flink docs]({% link deployment/ha/index.md
%}).
#### Marathon
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
-In particular, it should also adjust any configuration parameters for the
Flink cluster.
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
particular, it should
+also adjust any configuration parameters for the Flink cluster.
Here is an example configuration for Marathon:
+{% highlight javascript %}
{
"id": "flink",
"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh
-Djobmanager.memory.process.size=1472m -Djobmanager.rpc.port=6123
-Drest.port=8081 -Dtaskmanager.memory.process.size=1024m
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2
-Dmesos.resourcemanager.tasks.cpus=1",
"cpus": 1.0,
"mem": 1024
}
Review comment:
I think this yaml is not correct. We allocate `mem: 1024` but set
`-Djobmanager.memory.process.size=1472m`. Maybe we can put the simplest
definition possible here (e.g. w/o specifying the resources and the ports, for
example) or we completely remove it and refer to the documentation of Marathon.
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -84,113 +77,127 @@ of failures.
If you don't have a running DC/OS cluster, please follow the
[instructions on how to install DC/OS on the official
website](https://dcos.io/install/).
-Once you have a DC/OS cluster, you may install Flink through the DC/OS
-Universe. In the search prompt, just search for Flink. Alternatively, you can
use the DC/OS CLI:
+Once you have a DC/OS cluster, you may install Flink through the DC/OS
Universe. Just
+search for Flink in the search prompt. Alternatively, you can use the DC/OS
CLI:
- dcos package install flink
+{% highlight bash %}
+dcos package install flink
+{% endhighlight %}
Further information can be found in the
-[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/1.8/flink).
-
-
-## Mesos without DC/OS
-
-You can also run Mesos without DC/OS.
-
-### Installing Mesos
-
-Please follow the [instructions on how to setup Mesos on the official
website](http://mesos.apache.org/getting-started/).
-
-After installation you have to configure the set of master and agent nodes by
creating the files `MESOS_HOME/etc/mesos/masters` and
`MESOS_HOME/etc/mesos/slaves`.
-These files contain in each row a single hostname on which the respective
component will be started (assuming SSH access to these nodes).
+[DC/OS examples
documentation](https://github.com/dcos/examples/tree/master/flink).
-Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the
template found in the same directory.
-In this file, you have to define
+#### Mesos without DC/OS
- export MESOS_work_dir=WORK_DIRECTORY
+You can also run Mesos without DC/OS. Please follow the
+[instructions on how to setup Mesos on the official
website](http://mesos.apache.org/getting-started/).
-and it is recommended to uncommment
+After installing Mesos into `${MESOS_HOME}`, you have to configure the set of
master and
+agent nodes by creating the files `${MESOS_HOME}/etc/mesos/masters` and
+`${MESOS_HOME}/etc/mesos/slaves`. These files contain a single hostname per
row pointing to the
+hosts executing the respective components (assuming SSH access being available
for these nodes).
- export MESOS_log_dir=LOGGING_DIRECTORY
+Next, you have to create `${MESOS_HOME}/etc/mesos/mesos-master-env.sh` (a
template file is
+provided in the same directory). In `mesos-master-env.sh`, you have to set
`${MESOS_work_dir}`
+pointing to a directory used by Mesos to store data. Additionally, it is
recommended to specify
+`${MESOS_log_dir}`:
+{% highlight bash %}
+export MESOS_work_dir=/path/to/mesos/working/directory
+export MESOS_log_dir=/path/to/mesos/logging/directory
+{% endhighlight %}
-In order to configure the Mesos agents, you have to create
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same
directory.
-You have to configure
-
- export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
-
-and uncomment
-
- export MESOS_log_dir=LOGGING_DIRECTORY
- export MESOS_work_dir=WORK_DIRECTORY
-
-#### Mesos Library
+The Mesos agents are configured by creating
`${MESOS_HOME}/etc/mesos/mesos-agent-env.sh` (a
+template file is provided in the same directory). `${MESOS_master}` needs to
be configured as part
+of `mesos-agent-env.sh`. Additionally, `${MESOS_work_dir}` and
`${MESOS_log_dir}` should be set to
+the corresponding folders:
-In order to run Java applications with Mesos you have to export
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so` on Linux.
-Under Mac OS X you have to export
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.dylib`.
+{% highlight bash %}
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+export MESOS_work_dir=/path/to/mesos/working/directory
+export MESOS_log_dir=/path/to/mesos/logging/directory
+{% endhighlight %}
-#### Deploying Mesos
+In order to start your Mesos cluster, use the deployment script
`${MESOS_HOME}/sbin/mesos-start-cluster.sh`.
+Stopping the Mesos cluster can be achieved through
`${MESOS_HOME}/sbin/mesos-stop-cluster.sh`.
-In order to start your mesos cluster, use the deployment script
`MESOS_HOME/sbin/mesos-start-cluster.sh`.
-In order to stop your mesos cluster, use the deployment script
`MESOS_HOME/sbin/mesos-stop-cluster.sh`.
More information about the deployment scripts can be found
[here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
-### Installing Marathon
-
-Optionally, you may also [install
Marathon](https://mesosphere.github.io/marathon/docs/) which enables you to run
Flink in [high availability (HA) mode](#high-availability).
+#### Pre-installing Flink vs Docker/Mesos containers
-### Pre-installing Flink vs Docker/Mesos containers
+You may install Flink on all of your Mesos Master and Agent nodes. You can
also pull the binaries
+from the Flink website during deployment and apply your custom configuration
before launching the
+application master. A more convenient and easier-to-maintain approach is to
use Docker containers
+to manage the Flink binaries and configuration.
Review comment:
I'd say to go with the docker image approach.
##########
File path: docs/deployment/resource-providers/mesos.md
##########
@@ -26,55 +26,48 @@ under the License.
* This will be replaced by the TOC
{:toc}
-## Background
+## Getting Started
Review comment:
Agreed, I would move the detailed description of the `Mesos Scheduler`
and the `Artfiact Server` to after the getting started guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
