This is an automated email from the ASF dual-hosted git repository. qianzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 3e1e0b37d6a30a2c98d1227b4ac754b1d26686f3 Author: Qian Zhang <[email protected]> AuthorDate: Wed Sep 9 10:26:52 2020 +0800 Added doc for the `volume/csi` isolator. Review: https://reviews.apache.org/r/72845 --- docs/isolators/csi-volume.md | 339 +++++++++++++++++++++++++++++++++++++++++++ docs/mesos-containerizer.md | 1 + 2 files changed, 340 insertions(+) diff --git a/docs/isolators/csi-volume.md b/docs/isolators/csi-volume.md new file mode 100644 index 0000000..1b1a635 --- /dev/null +++ b/docs/isolators/csi-volume.md @@ -0,0 +1,339 @@ +--- +title: Apache Mesos - CSI Volume Support in Mesos Containerizer +layout: documentation +--- + +# Pre-provisioned CSI Volume Support in Mesos Containerizer + +Mesos 1.11.0 adds pre-provisioned CSI volume support to the +[MesosContainerizer](../mesos-containerizer.md) (a.k.a., the universal +containerizer) by introducing the new `volume/csi` isolator. + +This document describes the motivation and the configuration steps for enabling +the `volume/csi` isolator, and required framework changes. + +## Table of Contents +- [Motivation](#motivation) +- [How does it work?](#how-does-it-work) +- [Configuration](#configuration) + - [Pre-conditions](#pre-conditions) + - [Configuring CSI Volume Isolator](#configure-csi-volume-isolator) + - [Enabling frameworks to use CSI volumes](#enable-frameworks) + - [Volume Protobuf](#volume-protobuf) + - [Example](#example) + +## <a name="motivation"></a>Motivation + +[Container Storage Interface](https://github.com/container-storage-interface/spec) +(CSI) is a specification that defines a common set of APIs for all interactions +between the storage vendors and the container orchestration platforms. Building +CSI support allows Mesos to make use of the quickly-growing CSI ecosystem. + +We already have a [solution](../csi.md) to support CSI introduced in the Mesos +1.5.0 release, but that solution has a limitation: it requires CSI plugins to +implement the [ListVolumes](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#listvolumes) +and [GetCapacity](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#getcapacity) +APIs so that the external storage can be modeled as Mesos raw disk resources and +then offered to frameworks. However there are a lot of 3rd party CSI plugins the +do not implement those two APIs. + +Mesos 1.11.0 provides a more generic way to support 3rd party CSI plugins so +that Mesos can work with broader external storage ecosystem and we will benefit +from continued development of the community CSI plugins. + +## <a name="how-does-it-work"></a>How does it work? + +The `volume/csi` isolator interacts with CSI plugins via the plugin's gRPC +endpoint. + +When a new task with CSI volumes is launched, the `volume/csi` isolator will +call the CSI plugin to publish the specified CSI volumes onto the agent host +and then mount them onto the task container. When the task terminates, the +`volume/csi` isolator will call the CSI plugin to unpublish the specified CSI +volumes. + +Currently the `volume/csi` isolator will only call the CSI plugin's [node service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#node-service-rpc) but not [controller service](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controller-service-rpc), that means: + +- We only support pre-provisioned CSI volume but not dynamic CSI volumes + provisioning, so operators need to create the CSI volumes explicitly and + provide the volume info (e.g. volume ID, context, etc.) to frameworks so + that frameworks can use the volumes in their tasks. + +- We do not support the CSI volumes that require the controller service to + publish to a node ([ControllerPublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#controllerpublishvolume)) prior to the node service publishing on the node + ([NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume)). + +## <a name="configuration"></a>Configuration + +To use the `volume/csi` isolator, there are certain actions required by +operators and framework developers. In this section we list the steps +required by the operator to configure the `volume/csi` isolator and the steps +required by framework developers to specify CSI volumes in their tasks. + +### <a name="pre-conditions"></a>Pre-conditions + +- Explicitly create the CSI volumes that are going to be accessed by Mesos + tasks. For some CSI plugins (e.g. [NFS](https://github.com/kubernetes-csi/csi-driver-nfs)), + they do not implement the [CreateVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#createvolume) + API, so operators do not need to create the volume explicitly in this case. + +### <a name="configure-csi-volume-isolator"></a>Configuring the CSI Volume Isolator + +In order to configure the `volume/csi` isolator, the operator needs to +configure the `--isolation` and `--csi_plugin_config_dir` flags at agent +startup as follows: + +```{.console} + sudo mesos-agent \ + --master=<master-IP:master-port> \ + --work_dir=/var/lib/mesos \ + --isolation=filesystem/linux,volume/csi \ + --csi_plugin_config_dir=<directory that contains CSI plugin configuration files> +``` + +The `volume/csi` isolator must be specified in the `--isolation` flag at agent +startup; the `volume/csi` isolator has a dependency on the `filesystem/linux` +isolator. + +The operator needs to put the CSI plugin configuration files under the directory +specified via the agent flag `--csi_plugin_config_dir`. Each file in this +directory should contain a JSON object representing a `CSIPluginInfo` object +which can be either a managed CSI plugin (i.e. the plugin launched by Mesos as +a standalone container) or an unmanaged CSI plugin (i.e. the plugin launched +outside of Mesos). + +```{.proto} +message CSIPluginInfo { + required string type = 1; + optional string name = 2 [default = "default"]; + + // A list of container configurations to run managed CSI plugin. + repeated CSIPluginContainerInfo containers = 3; + + // The service endpoints of the unmanaged CSI plugin. + repeated CSIPluginEndpoint endpoints = 4; + + optional string target_path_root = 5; + optional bool target_path_exists = 6; +} + +message CSIPluginContainerInfo { + enum Service { + UNKNOWN = 0; + CONTROLLER_SERVICE = 1; + NODE_SERVICE = 2; + } + + repeated Service services = 1; + optional CommandInfo command = 2; + repeated Resource resources = 3; + optional ContainerInfo container = 4; +} + +message CSIPluginEndpoint { + required CSIPluginContainerInfo.Service csi_service = 1; + required string endpoint = 2; +} +``` + +Example of managed CSI plugin: +```{.json} +{ + "type": "org.apache.mesos.csi.managed-plugin", + "containers": [ + { + "services": [ + "NODE_SERVICE" + ], + "command": { + "value": "<path-to-managed-plugin> --endpoint=$CSI_ENDPOINT" + }, + "resources": [ + {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}}, + {"name": "mem", "type": "SCALAR", "scalar": {"value": 1024}} + ] + } + ] +} +``` + +Example of unmanaged CSI plugin: +```{.json} +{ + "type": "org.apache.mesos.csi.unmanaged-plugin", + "endpoints": [ + { + "csi_service": "NODE_SERVICE", + "endpoint": "/var/lib/unmanaged-plugin/csi.sock" + } + ], + "target_path_root": "/mnt/unmanaged-plugin" +} +``` + +### <a name="enable-frameworks"></a>Enabling frameworks to use CSI volumes + +#### <a name="volume-protobuf"></a>Volume Protobuf + +The `Volume` protobuf message has been updated to support CSI volumes. + +```{.proto} +message Volume { + ... + required Mode mode = 3; + required string container_path = 1; + + message Source { + enum Type { + UNKNOWN = 0; + ... + CSI_VOLUME = 5; + } + + message CSIVolume { + required string plugin_name = 1; + + message VolumeCapability { + message BlockVolume { + } + + message MountVolume { + optional string fs_type = 1; + repeated string mount_flags = 2; + } + + message AccessMode { + enum Mode { + UNKNOWN = 0; + SINGLE_NODE_WRITER = 1; + SINGLE_NODE_READER_ONLY = 2; + MULTI_NODE_READER_ONLY = 3; + MULTI_NODE_SINGLE_WRITER = 4; + MULTI_NODE_MULTI_WRITER = 5; + } + + required Mode mode = 1; + } + + oneof access_type { + BlockVolume block = 1; + MountVolume mount = 2; + } + + required AccessMode access_mode = 3; + } + + // Specifies the parameters used to stage/publish a pre-provisioned volume + // on an agent host. + message StaticProvisioning { + required string volume_id = 1; + required VolumeCapability volume_capability = 2; + optional bool readonly = 3; + map<string, Secret> node_stage_secrets = 4; + map<string, Secret> node_publish_secrets = 5; + map<string, string> volume_context = 6; + } + + optional StaticProvisioning static_provisioning = 2; + } + + optional Type type = 1; + ... + optional CSIVolume csi_volume = 6; + } + + optional Source source = 5; +} +``` + +When requesting a CSI volume for a container, the framework developer needs to +set `Volume` for the container, which includes `mode`, `container_path` and +`source` fields. + +The `source` field specifies where the volume comes from. Framework developers +need to set the `type` field to `CSI_VOLUME` and specify the `csi_volume` field. + +The `csi_volume` field specifies the information of the CSI volume. Framework +developers need to set the `plugin_name` field to the `type` field of one of the +CSI plugin configuration files in the directory specified via the agent flag +`--csi_plugin_config_dir`, and specify the `static_provisioning` field according +to the information of the pre-provisioned volume. The fields in `static_provisioning` +map directly onto the fields in the CSI calls [NodeStageVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodestagevolume) +and [NodePublishVolume](https://github.com/container-storage-interface/spec/blob/v1.3.0/spec.md#nodepublishvolume), +please find more detailed descriptions about those fields in the CSI spec. + +How to specify `container_path`: + +1. If you are launching a task without a container image and `container_path` + is an absolute path, you need to make sure the absolute path exists on your + host root file system as the container shares the host root file system; + otherwise, the task will fail. + +2. For other cases like launching a task without a container image and with a + relative `container_path`, or launching a task with a container image and an + absolute or relative `container_path`, the `volume/csi` isolator will help + create the `container_path` as the mount point. + +The following table summarizes the above rules for `container_path`: + +<table class="table table-striped"> + <tr> + <th></th> + <th>Task with rootfs</th> + <th>Task without rootfs</th> + </tr> + <tr> + <td>Absolute container_path</td> + <td>No need to exist</td> + <td>Must exist</td> + </tr> + <tr> + <td>Relative container_path</td> + <td>No need to exist</td> + <td>No need to exist</td> + </tr> +</table> + +#### <a name="example"></a>Example + +Launch a task with a CSI volume managed by NFS CSI plugin: + + ```{.json} + TaskInfo { + ... + "command" : { + "value": "echo test > volume/file" + }, + "container" : { + "type": "MESOS", + "volumes" : [ + { + "container_path" : "volume", + "mode" : "RW", + "source": { + "type": "CSI_VOLUME", + "csi_volume": { + "plugin_name": "nfs.csi.k8s.io", + "static_provisioning": { + "volume_id": "foo", + "volume_capability": { + "mount": {}, + "access_mode": { + "mode": "MULTI_NODE_MULTI_WRITER" + } + }, + "volume_context": { + "server": "192.168.1.100", + "share": "/mnt/data" + } + } + } + } + } + ] + } + } + ``` + +**NOTE**: To make the above example work, an NFS server (`192.168.1.100`) needs to +be setup to export the directory `/mnt/data`. diff --git a/docs/mesos-containerizer.md b/docs/mesos-containerizer.md index 3231cb9..c632550 100644 --- a/docs/mesos-containerizer.md +++ b/docs/mesos-containerizer.md @@ -59,6 +59,7 @@ Mesos supports the following built-in isolators. - posix/cpu - posix/mem - [posix/rlimits](isolators/posix-rlimits.md) +- [volume/csi](isolators/csi-volume.md) - [volume/host_path](container-volume.md#host_path-volume-source) - volume/image - [volume/sandbox_path](container-volume.md#sandbox_path-volume-source)
