Modified: mesos/site/source/community/user-groups.html.md URL: http://svn.apache.org/viewvc/mesos/site/source/community/user-groups.html.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/community/user-groups.html.md (original) +++ mesos/site/source/community/user-groups.html.md Thu Oct 29 17:55:21 2015 @@ -21,6 +21,9 @@ Mesos User Groups (MUGs) are responsible ### Germany * [Cologne](http://www.meetup.com/Mesos-User-Group-Cologne/) +### Poland +* [Warsaw](http://www.meetup.com/Warsaw-Mesos-User-Group/) + ### Netherlands * [Amsterdam](http://www.meetup.com/Mesos-Amsterdam/)
Modified: mesos/site/source/documentation/latest/committers.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/committers.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/committers.md (original) +++ mesos/site/source/documentation/latest/committers.md Thu Oct 29 17:55:21 2015 @@ -33,6 +33,13 @@ We'd like to thank the following committ <td>[email protected]</td> </tr> <tr> + <td>-5</td> + <td>Kapil Arya</td> + <td>Mesosphere / Northeastern University</td> + <td></td> + <td>[email protected]</td> + </tr> + <tr> <td>-8</td> <td>Adam B</td> <td>Mesosphere</td> @@ -241,7 +248,7 @@ When sending reviews, it can be valuable </tr> <tr> <td>Modules / Hooks</td> - <td>Benjamin Hindman, Niklas Nielsen</td> + <td>Kapil Arya, Benjamin Hindman, Niklas Nielsen</td> </tr> <tr> <td>Oversubscription</td> Modified: mesos/site/source/documentation/latest/configuration.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/configuration.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/configuration.md (original) +++ mesos/site/source/documentation/latest/configuration.md Thu Oct 29 17:55:21 2015 @@ -1022,7 +1022,7 @@ file:///path/to/file (where file contain </tr> <tr> <td> - --docker_sock=VALUE + --docker_socket=VALUE </td> <td> The UNIX socket path to be mounted into the docker executor container to @@ -1389,15 +1389,6 @@ file:///path/to/file (where file contain </tr> <tr> <td> - --resource_monitoring_interval=VALUE - </td> - <td> - Periodic time interval for monitoring executor - resource usage (e.g., 10secs, 1min, etc) (default: 1secs) - </td> - </tr> - <tr> - <td> --resources=VALUE </td> <td> @@ -1447,9 +1438,13 @@ file:///path/to/file (where file contain --[no-]switch_user </td> <td> - Whether to run tasks as the user who - submitted them rather than the user running - the slave (requires setuid permission) (default: true) + If set to `true`, the agent will attempt to run tasks as + the `user` who launched them (as defined in `FrameworkInfo`) + (this requires `setuid` permission and that the given `user` + exists on the agent). + If the user does not exist, an error occurs and the task will fail. + If set to `false`, tasks will be run as the same user as the Mesos + agent process. (default: true) </td> </tr> <tr> @@ -1657,6 +1652,26 @@ file:///path/to/file (where file contain </tr> <tr> <td> + --enable-libevent + </td> + <td> + use <a href="https://github.com/libevent/libevent";>libevent</a> + instead of libev for the libprocess event loop [default=no]. + Note that the libevent version 2+ development package is required + </td> + </tr> + <tr> + <td> + --enable-ssl + </td> + <td> + enable <a href="/documentation/latest/ssl">SSL</a> for libprocess + communication [default=no]. + Note that --enable-libevent is currently required for SSL functionality + </td> + </tr> + <tr> + <td> --disable-libtool-lock </td> <td> @@ -1870,12 +1885,38 @@ file:///path/to/file (where file contain </tr> <tr> <td> + --with-libevent=[=DIR] + </td> + <td> + specify where to locate the libevent library + </td> + </tr> + <tr> + <td> + --with-ssl=[=DIR] + </td> + <td> + specify where to locate the ssl library + </td> + </tr> + <tr> + <td> --with-network-isolator </td> <td> builds the network isolator </td> </tr> + <tr> + <td> + --with-nl=[=DIR] + </td> + <td> + specify where to locate the + <a href="https://www.infradead.org/~tgr/libnl/";>libnl3</a> library + (required for the network isolator) + </td> + </tr> </table> ### Some influential environment variables for configure script: Modified: mesos/site/source/documentation/latest/containerizer.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/containerizer.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/containerizer.md (original) +++ mesos/site/source/documentation/latest/containerizer.md Thu Oct 29 17:55:21 2015 @@ -20,7 +20,7 @@ filesystem. The modifications are specified in the ContainerInfo included in the ExecutorInfo, either by a framework or by using the ---default\_container\_info slave flag. +`--default_container_info` slave flag. ContainerInfo specifies Volumes which map parts of the shared filesystem (host\_path) into the container's view of the filesystem @@ -34,8 +34,8 @@ must exist) in the shared filesystem. The primary use-case for this isolator is to selectively make parts of the shared filesystem private to each container. For example, a -private "/tmp" directory can be achieved with host\_path="tmp" and -container\_path="/tmp" which will create a directory "tmp" inside the +private "/tmp" directory can be achieved with `host_path="tmp"` and +`container_path="/tmp"` which will create a directory "tmp" inside the executor's work directory (mode 1777) and simultaneously mount it as /tmp inside the container. This is transparent to processes running inside the container. Containers will not be able to see the host's @@ -45,9 +45,11 @@ inside the container. Containers will no The Pid Namespace isolator can be used to isolate each container in a separate pid namespace with two main benefits: + 1. Visibility: Processes running in the container (executor and descendants) are unable to see or signal processes outside the namespace. + 2. Clean termination: Termination of the leading process in a pid namespace will result in the kernel terminating all other processes in the namespace. Modified: mesos/site/source/documentation/latest/docker-containerizer.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/docker-containerizer.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/docker-containerizer.md (original) +++ mesos/site/source/documentation/latest/docker-containerizer.md Thu Oct 29 17:55:21 2015 @@ -14,7 +14,7 @@ The following sections will describe the To run the slave to enable the Docker Containerizer, you must launch the slave with "docker" as one of the containerizers option. -Example: mesos-slave --containerizers=docker,mesos +Example: `mesos-slave --containerizers=docker,mesos` Each slave that has the Docker containerizer should have Docker CLI client installed (version >= 1.0.0). @@ -38,17 +38,19 @@ Note that the Docker image is expected t ## What does the Docker Containerizer do? -The Docker Containerizer is translating Task/Executor Launch and Destroy calls to Docker CLI commands. +The Docker Containerizer is translating Task/Executor `Launch` and `Destroy` calls to Docker CLI commands. Currently the Docker Containerizer when launching as task will do the following: -1, Fetch all the files specified in the CommandInfo into the sandbox. -2, Pull the docker image from the remote repository. -3, Run the docker image with the Docker executor, and map the sandbox directory into the Docker container and set the directory mapping to the MESOS_SANDBOX environment variable. - The executor will also stream the container logs into stdout/stderr files in the sandbox. +1. Fetch all the files specified in the CommandInfo into the sandbox. + +2. Pull the docker image from the remote repository. + +3. Run the docker image with the Docker executor, and map the sandbox directory into the Docker container and set the directory mapping to the MESOS_SANDBOX environment variable. The executor will also stream the container logs into stdout/stderr files in the sandbox. + 4. On container exit or containerizer destroy, stop and remove the docker container. -The Docker Containerizer launches all containers with the "mesos-" prefix plus the slave id (ie: mesos-slave1-abcdefghji), and also assumes all containers with the "mesos-" prefix is managed by the slave and is free to stop or kill the containers. +The Docker Containerizer launches all containers with the `mesos-` prefix plus the slave id (ie: `mesos-slave1-abcdefghji`), and also assumes all containers with the `mesos-` prefix is managed by the slave and is free to stop or kill the containers. When launching the docker image as an Executor, the only difference is that it skips launching a command executor but just reaps on the docker container executor pid. @@ -58,20 +60,20 @@ The containerizer also supports optional ## Private Docker repository -To run a image from a private repository, one can include the uri pointing to a .dockercfg that contains login information. The .dockercfg file will be pulled into the sandbox the Docker Containerizer +To run a image from a private repository, one can include the uri pointing to a `.dockercfg` that contains login information. The `.dockercfg` file will be pulled into the sandbox the Docker Containerizer set the HOME environment variable pointing to the sandbox so docker cli will automatically pick up the config file. ## CommandInfo to run Docker images A docker image currently supports having an entrypoint and/or a default command. -To run a docker image with the default command (ie: docker run image), the CommandInfo's value must not be set. If the value is set then it will override the default command. +To run a docker image with the default command (ie: `docker run image`), the CommandInfo's value must not be set. If the value is set then it will override the default command. To run a docker image with an entrypoint defined, the CommandInfo's shell option must be set to false. -If shell option is set to true the Docker Containerizer will run the user's command wrapped with /bin/sh -c which will also become parameters to the image entrypoint. +If shell option is set to true the Docker Containerizer will run the user's command wrapped with `/bin/sh -c` which will also become parameters to the image entrypoint. ## Recover Docker containers on slave recovery The Docker containerizer supports recovering Docker containers when the slave restarts, which supports both when the slave is running in a Docker container or not. -With the docker_mesos_image flag enabled, the Docker containerizer assumes the containerizer is running in a container itself and modifies the mechanism it recovers and launches docker containers accordingly. +With the `--docker_mesos_image` flag enabled, the Docker containerizer assumes the containerizer is running in a container itself and modifies the mechanism it recovers and launches docker containers accordingly. Modified: mesos/site/source/documentation/latest/fetcher.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/fetcher.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/fetcher.md (original) +++ mesos/site/source/documentation/latest/fetcher.md Thu Oct 29 17:55:21 2015 @@ -41,7 +41,7 @@ detailed fetch action descriptions. Frameworks launch tasks by calling the scheduler driver method `launchTasks()`, passing `CommandInfo` protobuf structures as arguments. This type of structure specifies (among other things) a command and a list of URIs that need to be -"fetched" into the sandbox directory on the the slave node as a precondition for +"fetched" into the sandbox directory on the slave node as a precondition for task execution. Hence, when the slave receives a request go launch a task, it calls upon its fetcher, first, to provision the specified resources into the sandbox directory. If fetching fails, the task is not started and the reported Modified: mesos/site/source/documentation/latest/oversubscription.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/oversubscription.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/oversubscription.md (original) +++ mesos/site/source/documentation/latest/oversubscription.md Thu Oct 29 17:55:21 2015 @@ -269,16 +269,6 @@ between these corrections is controlled </td> </tr> - <tr> - <td> - --resource_monitoring_interval=VALUE - </td> - <td> - Periodic time interval for monitoring executor resource usage (e.g., -10secs, 1min, etc) (default: 1secs) - </td> - </tr> - </table> The `fixed` resource estimator is enabled as follows: Modified: mesos/site/source/documentation/latest/ssl.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/ssl.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/ssl.md (original) +++ mesos/site/source/documentation/latest/ssl.md Thu Oct 29 17:55:21 2015 @@ -2,6 +2,12 @@ layout: documentation --- +# SSL in Mesos + +By default, all the messages that flow through the Mesos cluster are unencrypted, making it possible for anyone with access to the cluster to intercept and potentially control arbitrary tasks. + +SSL/TLS support was added to libprocess in Mesos 0.23.0, which encypts the low-level communication that Mesos uses for network communication between Mesos components. Additionally, HTTPS support was added to the Mesos WebUI. + # Configuration There is currently only one implementation of the [libprocess socket interface](https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/socket.hpp) that supports SSL. This implementation uses [libevent](https://github.com/libevent/libevent). Specifically it relies on the `libevent-openssl` library that wraps `openssl`. @@ -15,16 +21,26 @@ After building Mesos 0.23.0 from source, Once you have successfully built and installed your new binaries, here are the environment variables that are applicable to the `Master`, `Slave`, `Framework Scheduler/Executor`, or any `libprocess process`: #### SSL_ENABLED=(false|0,true|1) [default=false|0] -Turn on or off SSL. When it is turned off it is the equivalent of default mesos with libevent as the backing for events. All sockets default to the non-SSL implementation. When it is turned on, the default configuration for sockets is SSL. This means outgoing connections will use SSL, and incoming connections will be expected to speak SSL as well. None of the below flags are relevant if SSL is not enabled. +Turn on or off SSL. When it is turned off it is the equivalent of default mesos with libevent as the backing for events. All sockets default to the non-SSL implementation. When it is turned on, the default configuration for sockets is SSL. This means outgoing connections will use SSL, and incoming connections will be expected to speak SSL as well. None of the below flags are relevant if SSL is not enabled. If SSL is enabled, `SSL_CERT_FILE` and `SSL_KEY_FILE` must be supplied. #### SSL_SUPPORT_DOWNGRADE=(false|0,true|1) [default=false|0] Control whether or not non-SSL connections can be established. If this is enabled __on the accepting side__, then the accepting side will downgrade to a non-SSL socket if the connecting side is attempting to communicate via non-SSL. (e.g. HTTP). See [Upgrading Your Cluster](#Upgrading) for more details. +#### SSL_KEY_FILE=(path to key) +The location of the private key used by OpenSSL. + +~~~ +// For example, to generate a key with OpenSSL: +openssl genrsa -des3 -f4 -passout pass:some_password -out key.pem 4096 +~~~ + #### SSL_CERT_FILE=(path to certificate) The location of the certificate that will be presented. -#### SSL_KEY_FILE=(path to key) -The location of the private key used by OpenSSL. +~~~ +// For example, to generate a certificate with OpenSSL: +openssl req -new -x509 -passin pass:some_password -days 365 -key key.pem -out cert.pem +~~~ #### SSL_VERIFY_CERT=(false|0,true|1) [default=false|0] Control whether certificates are verified when presented. If this is false, even when a certificate is presented, it will not be verified. When `SSL_REQUIRE_CERT` is true, `SSL_VERIFY_CERT` is overridden and all certificates will be verified _and_ required. @@ -88,5 +104,7 @@ __NOTE:__ Any tools you may use that com # <a name="WebUI"></a>WebUI The default Mesos WebUI uses relative links. Some of these links transition between endpoints served by the master and slaves. The WebUI currently does not have enough information to change the 'http' vs 'https' links based on whether the target endpoint is currently being served by an SSL-enabled binary. This may cause certain links in the WebUI to be broken when a cluster is in a transition state between SSL and non-SSL. Any tools that hit these endpoints will still be able to access them as long as they hit the endpoint using the right protocol, or the `SSL_SUPPORT_DOWNGRADE` option is set to true. +__NOTE:__ Frameworks with their own WebUI will need to add HTTPS support separately. + ### Certificates Most browsers have built in protection that guard transitions between pages served using different certificates. For this reason you may choose to serve both the master and slave endpoints using a common certificate that covers multiple hostnames. If you do not do this, certain links, such as those to slave sandboxes, may seem broken as the browser treats the transition between differing certificates transition as unsafe. Modified: mesos/site/source/documentation/latest/submitting-a-patch.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/submitting-a-patch.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/submitting-a-patch.md (original) +++ mesos/site/source/documentation/latest/submitting-a-patch.md Thu Oct 29 17:55:21 2015 @@ -43,7 +43,8 @@ layout: documentation 4. Now all of the files generated by the build process will be contained in the build directory you created, instead of being spread throughout the src directory, which is a bit messier. This is both cleaner, and makes it easy to clean up if you want to get rid of the files generated by `configure` and `make`. I.e. You can reset your build process without risking changes you made in the src directory, by simply deleting the build directory, and creating a new one. 3. Make sure that all of the unit tests pass, including the new test cases you have added: `make check`. - 1. To execute a single unit test (helpful when trying to debug a test case failure), use something like: `make check GTEST_FILTER="HTTPTest.Delete"`. + 1. To build all tests without executing them, use something like: `make tests`. + 2. To execute a single unit test (helpful when trying to debug a test case failure), use something like: `make check GTEST_FILTER="HTTPTest.Delete"`. 4. Make sure to pull in any changes that have been committed to master branch. Using Git, do this via something like: 1. `git checkout master` Modified: mesos/site/source/documentation/latest/testing-patterns.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/testing-patterns.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/testing-patterns.md (original) +++ mesos/site/source/documentation/latest/testing-patterns.md Thu Oct 29 17:55:21 2015 @@ -6,6 +6,69 @@ layout: documentation A collection of common testing patterns used in Mesos tests. If you have found a good way to test a certain condition that you think may be useful for other cases, please document it here together with motivation and background. +## Expediting events with `Clock` +Some events in Mesos are separated by certain timeouts, for example framework registration attempts. Simple waiting for such events to fire leads to blocking the test thread for the duration of the associated timeout. This increases the duration of `make check` for no good reason. + +If an event is triggered by an act of processing a message from an actor's mailbox, it can be expedited with the help of libprocess' `Clock` routines. Delayed messages are maintained in sorted order by their due time and are dispatched â i.e. pushed into destination mailboxes â when this time comes. An important bit here is that time is driven by the internal libprocess clock. We can shift this clock into the future by calling `Clock::advance(<duration>)`, rendering certain front messages in the collection due now. These messages are dispatched instantly, effectively overriding the associated event's timeout. + +**NOTE**: Without calling `Clock::settle()` there is no guarantee a dispatched message has been already processed. + +Below is an example of this pattern. To avoid master backlogging, Mesos frameworks usually wait for some time (backoff) before retrying registration. In the test below we simulate the loss of a registration request, but avoid blocking the test for the backoff duration. + +~~~{.cpp} +TEST_F(FaultToleranceTest, FrameworkReliableRegistration) +{ + Try<PID<Master>> master = StartMaster(); + ASSERT_SOME(master); + + Try<PID<Slave>> slave = StartSlave(); + ASSERT_SOME(slave); + + // As a side effect of driver instantiation, registration backoff will be set + // to a default: internal::scheduler::REGISTRATION_BACKOFF_FACTOR. + MockScheduler sched; + MesosSchedulerDriver driver( + &sched, DEFAULT_FRAMEWORK_INFO, master.get(), DEFAULT_CREDENTIAL); + + Future<Nothing> registered; + EXPECT_CALL(sched, registered(&driver, _, _)) + .WillOnce(FutureSatisfy(®istered)); + + EXPECT_CALL(sched, resourceOffers(&driver, _)) + .WillRepeatedly(Return()); + + EXPECT_CALL(sched, offerRescinded(&driver, _)) + .Times(AtMost(1)); + + Future<AuthenticateMessage> authenticateMessage = + FUTURE_PROTOBUF(AuthenticateMessage(), _, master.get()); + + // Drop the first framework registered message, allow subsequent messages. + Future<FrameworkRegisteredMessage> frameworkRegisteredMessage = + DROP_PROTOBUF(FrameworkRegisteredMessage(), master.get(), _); + + driver.start(); + + // Ensure authentication occurs. + AWAIT_READY(authenticateMessage); + + AWAIT_READY(frameworkRegisteredMessage); + + // Trigger the registration retry instantly to avoid blocking the test. + Clock::pause(); + Clock::advance(internal::scheduler::REGISTRATION_BACKOFF_FACTOR); + + AWAIT_READY(registered); // Ensures registered message is received. + + driver.stop(); + driver.join(); + + Shutdown(); + + Clock::resume(); +} +~~~ + ## Using `Clock` magic to ensure an event is processed Scheduling a sequence of events in an asynchronous environment is not easy: a function call usually initiates an action and returns immediately, while the action runs in background. A simple, obvious, and bad solution is to use `os::sleep()` to wait for action completion. The time the action needs to finish may vary on different machines, while increasing sleep duration increases the test execution time and slows down `make check`. One of the right ways to do it is to wait for an action to finish and proceed right after. This is possible using libprocess' `Clock` routines. Modified: mesos/site/source/documentation/latest/upgrades.md URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/upgrades.md?rev=1711312&r1=1711311&r2=1711312&view=diff ============================================================================== --- mesos/site/source/documentation/latest/upgrades.md (original) +++ mesos/site/source/documentation/latest/upgrades.md Thu Oct 29 17:55:21 2015 @@ -6,6 +6,14 @@ layout: documentation This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes. +## Upgrading from 0.25.x to 0.26.x ## + +**NOTE** The names of some TaskStatus::Reason enums have been changed. But the tag numbers remain unchanged, so it is backwards compatible. Frameworks using the new version might need to do some compile time adjustments: + +* REASON_MEM_LIMIT -> REASON_CONTAINER_LIMITATION_MEMORY +* REASON_EXECUTOR_PREEMPTED -> REASON_CONTAINER_PREEMPTED + + ## Upgrading from 0.24.x to 0.25.x **NOTE** The following endpoints will be deprecated in favor of new endpoints. Both versions will be available in 0.25 but the deprecated endpoints will be removed in a subsequent release. @@ -27,6 +35,7 @@ For both master and slave: * /files/download.json becomes /files/download * /files/read.json becomes /files/read +**NOTE** The C++/Java/Python scheduler bindings have been updated. In particular, the driver can make a suppressOffers() call to stop receiving offers (until reviveOffers() is called). In order to upgrade a running cluster:
