This is an automated email from the ASF dual-hosted git repository.
lidongdai pushed a commit to branch 1.3.6-prepare
in repository https://gitbox.apache.org/repos/asf/incubator-dolphinscheduler.git
The following commit(s) were added to refs/heads/1.3.6-prepare by this push:
new 3f47bd8 [1.3.6-prepare][Improvement][Docker&K8s] Update readme for
s3/minio and skywalking, local and shared storage, and spark 3 (#5188)
3f47bd8 is described below
commit 3f47bd842565cc30b253150a1da12c1163b60b85
Author: Shiwen Cheng <[email protected]>
AuthorDate: Wed Mar 31 20:53:50 2021 +0800
[1.3.6-prepare][Improvement][Docker&K8s] Update readme for s3/minio and
skywalking, local and shared storage, and spark 3 (#5188)
* [1.3.6-prepare][Improvement][Docker&K8s] Update readme for s3/minio and
skywalking
* [1.3.6-prepare][Improvement][Docker&K8s] Update readme for local and
shared storage
* [1.3.6-prepare][Improvement][Docker&K8s] Update readme for spark 3
---
docker/build/README.md | 180 +++++++++++++++++++++---
docker/build/README_zh_CN.md | 184 +++++++++++++++++++++----
docker/kubernetes/dolphinscheduler/README.md | 153 +++++++++++++++++++-
docker/kubernetes/dolphinscheduler/values.yaml | 10 +-
4 files changed, 472 insertions(+), 55 deletions(-)
diff --git a/docker/build/README.md b/docker/build/README.md
index 97957c6..1abbb72 100644
--- a/docker/build/README.md
+++ b/docker/build/README.md
@@ -1,3 +1,39 @@
+DolphinScheduler
+=================
+
+* [What is DolphinScheduler?](#what-is-dolphinscheduler)
+* [Prerequisites](#prerequisites)
+* [How to use this Docker image](#how-to-use-this-docker-image)
+ * [You can start a DolphinScheduler by docker\-compose
(recommended)](#you-can-start-a-dolphinscheduler-by-docker-compose-recommended)
+ * [Or via specifying the existing PostgreSQL and ZooKeeper
service](#or-via-specifying-the-existing-postgresql-and-zookeeper-service)
+ * [Or start a standalone DolphinScheduler
server](#or-start-a-standalone-dolphinscheduler-server)
+* [How to build a Docker image](#how-to-build-a-docker-image)
+* [Support Matrix](#support-matrix)
+* [Environment Variables](#environment-variables)
+ * [Database](#database)
+ * [ZooKeeper](#zookeeper)
+ * [Common](#common)
+ * [Master Server](#master-server)
+ * [Worker Server](#worker-server)
+ * [Alert Server](#alert-server)
+ * [Api Server](#api-server)
+ * [Logger Server](#logger-server)
+* [Initialization scripts](#initialization-scripts)
+* [FAQ](#faq)
+ * [How to stop DolphinScheduler by
docker\-compose?](#how-to-stop-dolphinscheduler-by-docker-compose)
+ * [How to deploy DolphinScheduler on Docker
Swarm?](#how-to-deploy-dolphinscheduler-on-docker-swarm)
+ * [How to use MySQL as the DolphinScheduler's database instead of
PostgreSQL?](#how-to-use-mysql-as-the-dolphinschedulers-database-instead-of-postgresql)
+ * [How to support MySQL datasource in Datasource
manage?](#how-to-support-mysql-datasource-in-datasource-manage)
+ * [How to support Oracle datasource in Datasource
manage?](#how-to-support-oracle-datasource-in-datasource-manage)
+ * [How to support Python 2 pip and custom
requirements\.txt?](#how-to-support-python-2-pip-and-custom-requirementstxt)
+ * [How to support Python 3?](#how-to-support-python-3)
+ * [How to support Hadoop, Spark, Flink, Hive or
DataX?](#how-to-support-hadoop-spark-flink-hive-or-datax)
+ * [How to support Spark 3?](#how-to-support-spark-3)
+ * [How to support shared storage between Master, Worker and Api
server?](#how-to-support-shared-storage-between-master-worker-and-api-server)
+ * [How to support local file resource storage instead of HDFS and
S3?](#how-to-support-local-file-resource-storage-instead-of-hdfs-and-s3)
+ * [How to support S3 resource storage like
MinIO?](#how-to-support-s3-resource-storage-like-minio)
+ * [How to configure SkyWalking?](#how-to-configure-skywalking)
+
## What is DolphinScheduler?
DolphinScheduler is a distributed and easy-to-expand visual DAG workflow
scheduling system, dedicated to solving the complex dependencies in data
processing, making the scheduling system out of the box for data processing.
@@ -16,17 +52,17 @@ Official Website: https://dolphinscheduler.apache.org
- [Docker](https://docs.docker.com/engine/) 1.13.1+
- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
-## How to use this docker image
+## How to use this Docker image
-#### You can start a dolphinscheduler by docker-compose (recommended)
+#### You can start a DolphinScheduler by docker-compose (recommended)
```
$ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
```
-The default **postgres** user `root`, postgres password `root` and database
`dolphinscheduler` are created in the `docker-compose.yml`.
+The default **PostgreSQL** username `root`, password `root` and database
`dolphinscheduler` are created in the `docker-compose.yml`.
-The default **zookeeper** is created in the `docker-compose.yml`.
+The default **ZooKeeper** is created in the `docker-compose.yml`.
Access the Web UI: http://192.168.xx.xx:12345/dolphinscheduler
@@ -34,9 +70,9 @@ The default username is `admin` and the default password is
`dolphinscheduler123
> **Tip**: For quick start in docker, you can create a tenant named `ds` and
> associate the user `admin` with the tenant `ds`
-#### Or via Environment Variables **`DATABASE_HOST`**, **`DATABASE_PORT`**,
**`ZOOKEEPER_QUORUM`**
+#### Or via specifying the existing PostgreSQL and ZooKeeper service
-You can specify **existing postgres and zookeeper service**. Example:
+You can specify existing **PostgreSQL** and **ZooKeeper** service. Example:
```
$ docker run -d --name dolphinscheduler \
@@ -49,15 +85,9 @@ apache/dolphinscheduler:latest all
Access the Web UI：http://192.168.xx.xx:12345/dolphinscheduler
-#### Or start a standalone dolphinscheduler server
-
-You can start a standalone dolphinscheduler server.
+#### Or start a standalone DolphinScheduler server
-* Create a **local volume** for resource storage, For example:
-
-```
-docker volume create dolphinscheduler-resource-local
-```
+You can start a standalone DolphinScheduler server.
* Start a **master server**, For example:
@@ -76,7 +106,6 @@ $ docker run -d --name dolphinscheduler-worker \
-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e
DATABASE_DATABASE="dolphinscheduler" \
-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--v dolphinscheduler-resource-local:/dolphinscheduler \
apache/dolphinscheduler:latest worker-server
```
@@ -87,7 +116,6 @@ $ docker run -d --name dolphinscheduler-api \
-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e
DATABASE_DATABASE="dolphinscheduler" \
-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--v dolphinscheduler-resource-local:/dolphinscheduler \
-p 12345:12345 \
apache/dolphinscheduler:latest api-server
```
@@ -103,7 +131,7 @@ apache/dolphinscheduler:latest alert-server
**Note**: You must be specify `DATABASE_HOST`, `DATABASE_PORT`,
`DATABASE_DATABASE`, `DATABASE_USERNAME`, `DATABASE_PASSWORD`,
`ZOOKEEPER_QUORUM` when start a standalone dolphinscheduler server.
-## How to build a docker image
+## How to build a Docker image
You can build a docker image in A Unix-like operating system, You can also
build it in Windows operating system.
@@ -162,7 +190,9 @@ Please read `./docker/build/hooks/build`
`./docker/build/hooks/build.bat` script
## Environment Variables
-The DolphinScheduler Docker container is configured through environment
variables, and the default value will be used if an environment variable is not
set.
+The Docker container is configured through environment variables, and the
default value will be used if an environment variable is not set
+
+Especially, it can be configured through the environment variable
configuration file `config.env.sh` in Docker Compose and Docker Swarm
### Database
@@ -515,7 +545,7 @@ done
## FAQ
-### How to stop dolphinscheduler by docker-compose?
+### How to stop DolphinScheduler by docker-compose?
Stop containers:
@@ -529,7 +559,7 @@ Stop containers and remove containers, networks and volumes:
docker-compose down -v
```
-### How to deploy dolphinscheduler on Docker Swarm?
+### How to deploy DolphinScheduler on Docker Swarm?
Assuming that the Docker Swarm cluster has been created (If there is no Docker
Swarm cluster, please refer to
[create-swarm](https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/))
@@ -574,7 +604,7 @@ docker build -t apache/dolphinscheduler:mysql-driver .
6. Add `dolphinscheduler-mysql` service in `docker-compose.yml` (**Optional**,
you can directly use a external MySQL database)
-7. Modify DATABASE environments in `config.env.sh`
+7. Modify DATABASE environment variables in `config.env.sh`
```
DATABASE_TYPE=mysql
@@ -769,4 +799,112 @@ Spark on YARN (Deploy Mode is `cluster` or `client`)
requires Hadoop support. Si
Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` exists
+### How to support Spark 3?
+
+In fact, the way to submit applications with `spark-submit` is the same,
regardless of Spark 1, 2 or 3. In other words, the semantics of `SPARK_HOME2`
is the second `SPARK_HOME` instead of `SPARK2`'s `HOME`, so just set
`SPARK_HOME2=/path/to/spark3`
+
+Take Spark 3.1.1 as an example:
+
+1. Download the Spark 3.1.1 release binary `spark-3.1.1-bin-hadoop2.7.tgz`
+
+2. Run a dolphinscheduler (See **How to use this docker image**)
+
+3. Copy the Spark 3.1.1 release binary into Docker container
+
+```bash
+docker cp spark-3.1.1-bin-hadoop2.7.tgz dolphinscheduler-worker:/opt/soft
+```
+
+4. Attach the container and ensure that `SPARK_HOME2` exists
+
+```bash
+docker exec -it dolphinscheduler-worker bash
+cd /opt/soft
+tar zxf spark-3.1.1-bin-hadoop2.7.tgz
+rm -f spark-3.1.1-bin-hadoop2.7.tgz
+ln -s spark-3.1.1-bin-hadoop2.7 spark2 # or just mv
+$SPARK_HOME2/bin/spark-submit --version
+```
+
+The last command will print Spark version if everything goes well
+
+5. Verify Spark under a Shell task
+
+```
+$SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi
$SPARK_HOME2/examples/jars/spark-examples_2.12-3.1.1.jar
+```
+
+Check whether the task log contains the output like `Pi is roughly 3.146015`
+
+### How to support shared storage between Master, Worker and Api server?
+
+For example, Master, Worker and Api server may use Hadoop at the same time
+
+1. Modify the volume `dolphinscheduler-shared-local` to support nfs in
`docker-compose.yml`
+
+> If you want to deploy dolphinscheduler on Docker Swarm, you need modify
`docker-stack.yml`
+
+```yaml
+volumes:
+ dolphinscheduler-shared-local:
+ driver_opts:
+ type: "nfs"
+ o: "addr=10.40.0.199,nolock,soft,rw"
+ device: ":/path/to/shared/dir"
+```
+
+2. Put the Hadoop into the nfs
+
+3. Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` are correct
+
+### How to support local file resource storage instead of HDFS and S3?
+
+1. Modify the following environment variables in `config.env.sh`:
+
+```
+RESOURCE_STORAGE_TYPE=HDFS
+FS_DEFAULT_FS=file:///
+```
+
+2. Modify the volume `dolphinscheduler-resource-local` to support nfs in
`docker-compose.yml`
+
+> If you want to deploy dolphinscheduler on Docker Swarm, you need modify
`docker-stack.yml`
+
+```yaml
+volumes:
+ dolphinscheduler-resource-local:
+ driver_opts:
+ type: "nfs"
+ o: "addr=10.40.0.199,nolock,soft,rw"
+ device: ":/path/to/resource/dir"
+```
+
+### How to support S3 resource storage like MinIO?
+
+Take MinIO as an example: Modify the following environment variables in
`config.env.sh`
+
+```
+RESOURCE_STORAGE_TYPE=S3
+RESOURCE_UPLOAD_PATH=/dolphinscheduler
+FS_DEFAULT_FS=s3a://BUCKET_NAME
+FS_S3A_ENDPOINT=http://MINIO_IP:9000
+FS_S3A_ACCESS_KEY=MINIO_ACCESS_KEY
+FS_S3A_SECRET_KEY=MINIO_SECRET_KEY
+```
+
+`BUCKET_NAME`, `MINIO_IP`, `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` need to
be modified to actual values
+
+> **Note**: `MINIO_IP` can only use IP instead of domain name, because
DolphinScheduler currently doesn't support S3 path style access
+
+### How to configure SkyWalking?
+
+Modify SKYWALKING environment variables in `config.env.sh`:
+
+```
+SKYWALKING_ENABLE=true
+SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
+SW_GRPC_LOG_SERVER_HOST=127.0.0.1
+SW_GRPC_LOG_SERVER_PORT=11800
+```
+
For more information please refer to the
[incubator-dolphinscheduler](https://github.com/apache/incubator-dolphinscheduler.git)
documentation.
diff --git a/docker/build/README_zh_CN.md b/docker/build/README_zh_CN.md
index 309e462..31290f7 100644
--- a/docker/build/README_zh_CN.md
+++ b/docker/build/README_zh_CN.md
@@ -1,4 +1,40 @@
-## DolphinScheduler是什么?
+DolphinScheduler
+=================
+
+* [DolphinScheduler 是什么?](#dolphinscheduler-是什么)
+* [先决条件](#先决条件)
+* [如何使用 Docker 镜像](#如何使用-docker-镜像)
+ * [以 docker\-compose 的方式启动 DolphinScheduler
(推荐)](#以-docker-compose-的方式启动-dolphinscheduler-推荐)
+ * [或者通过指定已存在的 PostgreSQL 和 ZooKeeper
服务](#或者通过指定已存在的-postgresql-和-zookeeper-服务)
+ * [或者运行 DolphinScheduler 中的部分服务](#或者运行-dolphinscheduler-中的部分服务)
+* [如何构建一个 Docker 镜像](#如何构建一个-docker-镜像)
+* [支持矩阵](#支持矩阵)
+* [环境变量](#环境变量)
+ * [数据库](#数据库)
+ * [ZooKeeper](#zookeeper)
+ * [通用](#通用)
+ * [Master Server](#master-server)
+ * [Worker Server](#worker-server)
+ * [Alert Server](#alert-server)
+ * [Api Server](#api-server)
+ * [Logger Server](#logger-server)
+* [初始化脚本](#初始化脚本)
+* [FAQ](#faq)
+ * [如何通过 docker\-compose 停止
DolphinScheduler？](#如何通过-docker-compose-停止-dolphinscheduler)
+ * [如何在 Docker Swarm 上部署
DolphinScheduler？](#如何在-docker-swarm-上部署-dolphinscheduler)
+ * [如何用 MySQL 替代 PostgreSQL 作为 DolphinScheduler
的数据库？](#如何用-mysql-替代-postgresql-作为-dolphinscheduler-的数据库)
+ * [如何在数据源中心支持 MySQL 数据源？](#如何在数据源中心支持-mysql-数据源)
+ * [如何在数据源中心支持 Oracle 数据源？](#如何在数据源中心支持-oracle-数据源)
+ * [如何支持 Python 2 pip 以及自定义
requirements\.txt？](#如何支持-python-2-pip-以及自定义-requirementstxt)
+ * [如何支持 Python 3？](#如何支持-python-3)
+ * [如何支持 Hadoop, Spark, Flink, Hive 或
DataX？](#如何支持-hadoop-spark-flink-hive-或-datax)
+ * [如何支持 Spark 3？](#如何支持-spark-3)
+ * [如何在 Master、Worker 和 Api 服务之间支持共享存储？](#如何在-masterworker-和-api-服务之间支持共享存储)
+ * [如何支持本地文件存储而非 HDFS 和 S3？](#如何支持本地文件存储而非-hdfs-和-s3)
+ * [如何支持 S3 资源存储，例如 MinIO？](#如何支持-s3-资源存储例如-minio)
+ * [如何配置 SkyWalking？](#如何配置-skywalking)
+
+## DolphinScheduler 是什么?
一个分布式易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中`开箱即用`。
@@ -16,17 +52,17 @@ Official Website: https://dolphinscheduler.apache.org
- [Docker](https://docs.docker.com/engine/) 1.13.1+
- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
-## 如何使用docker镜像
+## 如何使用 Docker 镜像
-#### 以 docker-compose 的方式启动dolphinscheduler(推荐)
+#### 以 docker-compose 的方式启动 DolphinScheduler (推荐)
```
$ docker-compose -f ./docker/docker-swarm/docker-compose.yml up -d
```
-在`docker-compose.yml`文件中，默认的创建`Postgres`的用户、密码和数据库，默认值分别为：`root`、`root`、`dolphinscheduler`。
+在`docker-compose.yml`文件中，默认创建的**PostgreSQL**的用户、密码和数据库，默认值分别为：`root`、`root`、`dolphinscheduler`。
-同时，默认的`Zookeeper`也会在`docker-compose.yml`文件中被创建。
+同时，默认的**ZooKeeper**也会在`docker-compose.yml`文件中被创建。
访问前端页面：http://192.168.xx.xx:12345/dolphinscheduler
@@ -34,9 +70,9 @@ $ docker-compose -f ./docker/docker-swarm/docker-compose.yml
up -d
> **提示**: 为了在docker中快速开始，你可以创建一个名为`ds`的租户，并将这个租户`ds`关联到用户`admin`
-#### 或者通过环境变量 **`DATABASE_HOST`**, **`DATABASE_PORT`**, **`ZOOKEEPER_QUORUM`**
+#### 或者通过指定已存在的 PostgreSQL 和 ZooKeeper 服务
-你可以指定已经存在的 **`Postgres`** 和 **`Zookeeper`** 服务. 如下:
+你可以指定已存在的 **PostgreSQL** 和 **ZooKeeper** 服务. 如下:
```
$ docker run -d --name dolphinscheduler \
@@ -49,15 +85,9 @@ apache/dolphinscheduler:latest all
访问前端页面：http://192.168.xx.xx:12345/dolphinscheduler
-#### 或者运行dolphinscheduler中的部分服务
+#### 或者运行 DolphinScheduler 中的部分服务
-你能够运行dolphinscheduler中的部分服务。
-
-* 创建一个 **本地卷** 用于资源存储，如下:
-
-```
-docker volume create dolphinscheduler-resource-local
-```
+你能够运行 DolphinScheduler 中的部分服务。
* 启动一个 **master server**, 如下:
@@ -76,7 +106,6 @@ $ docker run -d --name dolphinscheduler-worker \
-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e
DATABASE_DATABASE="dolphinscheduler" \
-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--v dolphinscheduler-resource-local:/dolphinscheduler \
apache/dolphinscheduler:latest worker-server
```
@@ -87,7 +116,6 @@ $ docker run -d --name dolphinscheduler-api \
-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e
DATABASE_DATABASE="dolphinscheduler" \
-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
-e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--v dolphinscheduler-resource-local:/dolphinscheduler \
-p 12345:12345 \
apache/dolphinscheduler:latest api-server
```
@@ -103,7 +131,7 @@ apache/dolphinscheduler:latest alert-server
**注意**: 当你运行dolphinscheduler中的部分服务时，你必须指定这些环境变量 `DATABASE_HOST`,
`DATABASE_PORT`, `DATABASE_DATABASE`, `DATABASE_USERNAME`, `DATABASE_PASSWORD`,
`ZOOKEEPER_QUORUM`。
-## 如何构建一个docker镜像
+## 如何构建一个 Docker 镜像
你能够在类Unix系统和Windows系统中构建一个docker镜像。
@@ -120,7 +148,7 @@ Windows系统, 如下:
C:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
```
-如果你不理解这些脚本 `./docker/build/hooks/build`
`./docker/build/hooks/build.bat`，请阅读里面的内容。
+如果你不理解这些脚本 `./docker/build/hooks/build`
`./docker/build/hooks/build.bat`，请阅读里面的内容
## 支持矩阵
@@ -162,7 +190,9 @@ C:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat
## 环境变量
-DolphinScheduler Docker 容器通过环境变量进行配置，缺省时将会使用默认值
+Docker 容器通过环境变量进行配置，缺省时将会使用默认值
+
+特别地，在 Docker Compose 和 Docker Swarm 中，可以通过环境变量配置文件 `config.env.sh` 进行配置
### 数据库
@@ -515,7 +545,7 @@ done
## FAQ
-### 如何通过 docker-compose 停止 dolphinscheduler？
+### 如何通过 docker-compose 停止 DolphinScheduler？
停止所有容器:
@@ -529,7 +559,7 @@ docker-compose stop
docker-compose down -v
```
-### 如何在 Docker Swarm 上部署 dolphinscheduler？
+### 如何在 Docker Swarm 上部署 DolphinScheduler？
假设 Docker Swarm 集群已经部署（如果还没有创建 Docker Swarm 集群，请参考
[create-swarm](https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/)）
@@ -649,7 +679,7 @@ docker build -t apache/dolphinscheduler:oracle-driver .
6. 在数据源中心添加一个 Oracle 数据源
-### 如何支持 Python 2 pip 以及自定义 requirements.txt?
+### 如何支持 Python 2 pip 以及自定义 requirements.txt？
1. 创建一个新的 `Dockerfile`，用于安装 pip:
@@ -769,4 +799,112 @@ Spark on YARN (部署方式为 `cluster` 或 `client`) 需要 Hadoop 支持. 类
确保 `$HADOOP_HOME` 和 `$HADOOP_CONF_DIR` 存在
+### 如何支持 Spark 3？
+
+事实上，使用 `spark-submit` 提交应用的方式是相同的, 无论是 Spark 1, 2 或 3. 换句话说，`SPARK_HOME2`
的语义是第二个 `SPARK_HOME`, 而非 `SPARK2` 的 `HOME`, 因此只需设置
`SPARK_HOME2=/path/to/spark3` 即可
+
+以 Spark 3.1.1 为例:
+
+1. 下载 Spark 3.1.1 发布的二进制包 `spark-3.1.1-bin-hadoop2.7.tgz`
+
+2. 运行 dolphinscheduler (详见**如何使用docker镜像**)
+
+3. 复制 Spark 3.1.1 二进制包到 Docker 容器中
+
+```bash
+docker cp spark-3.1.1-bin-hadoop2.7.tgz dolphinscheduler-worker:/opt/soft
+```
+
+4. 登录到容器并确保 `SPARK_HOME2` 存在
+
+```bash
+docker exec -it dolphinscheduler-worker bash
+cd /opt/soft
+tar zxf spark-3.1.1-bin-hadoop2.7.tgz
+rm -f spark-3.1.1-bin-hadoop2.7.tgz
+ln -s spark-3.1.1-bin-hadoop2.7 spark2 # or just mv
+$SPARK_HOME2/bin/spark-submit --version
+```
+
+如果一切执行正常，最后一条命令将会打印 Spark 版本信息
+
+5. 在一个 Shell 任务下验证 Spark
+
+```
+$SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi
$SPARK_HOME2/examples/jars/spark-examples_2.12-3.1.1.jar
+```
+
+检查任务日志是否包含输出 `Pi is roughly 3.146015`
+
+### 如何在 Master、Worker 和 Api 服务之间支持共享存储？
+
+例如, Master、Worker 和 Api 服务可能同时使用 Hadoop
+
+1. 修改 `docker-compose.yml` 文件中的 `dolphinscheduler-shared-local` 存储卷，以支持 nfs
+
+> 如果你想在 Docker Swarm 上部署 dolphinscheduler，你需要修改 `docker-stack.yml`
+
+```yaml
+volumes:
+ dolphinscheduler-shared-local:
+ driver_opts:
+ type: "nfs"
+ o: "addr=10.40.0.199,nolock,soft,rw"
+ device: ":/path/to/shared/dir"
+```
+
+2. 将 Hadoop 放到 nfs
+
+3. 确保 `$HADOOP_HOME` 和 `$HADOOP_CONF_DIR` 正确
+
+### 如何支持本地文件存储而非 HDFS 和 S3？
+
+1. 修改 `config.env.sh` 文件中下面的环境变量:
+
+```
+RESOURCE_STORAGE_TYPE=HDFS
+FS_DEFAULT_FS=file:///
+```
+
+2. 修改 `docker-compose.yml` 文件中的 `dolphinscheduler-shared-local` 存储卷，以支持 nfs
+
+> 如果你想在 Docker Swarm 上部署 dolphinscheduler，你需要修改 `docker-stack.yml`
+
+```yaml
+volumes:
+ dolphinscheduler-resource-local:
+ driver_opts:
+ type: "nfs"
+ o: "addr=10.40.0.199,nolock,soft,rw"
+ device: ":/path/to/resource/dir"
+```
+
+### 如何支持 S3 资源存储，例如 MinIO？
+
+以 MinIO 为例: 修改 `config.env.sh` 文件中下面的环境变量
+
+```
+RESOURCE_STORAGE_TYPE=S3
+RESOURCE_UPLOAD_PATH=/dolphinscheduler
+FS_DEFAULT_FS=s3a://BUCKET_NAME
+FS_S3A_ENDPOINT=http://MINIO_IP:9000
+FS_S3A_ACCESS_KEY=MINIO_ACCESS_KEY
+FS_S3A_SECRET_KEY=MINIO_SECRET_KEY
+```
+
+`BUCKET_NAME`, `MINIO_IP`, `MINIO_ACCESS_KEY` 和 `MINIO_SECRET_KEY` 需要被修改为实际值
+
+> **注意**: `MINIO_IP` 只能使用 IP 而非域名, 因为 DolphinScheduler 尚不支持 S3 路径风格访问 (S3 path
style access)
+
+### 如何配置 SkyWalking？
+
+修改 `config.env.sh` 文件中的 SKYWALKING 环境变量
+
+```
+SKYWALKING_ENABLE=true
+SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
+SW_GRPC_LOG_SERVER_HOST=127.0.0.1
+SW_GRPC_LOG_SERVER_PORT=11800
+```
+
更多信息请查看
[incubator-dolphinscheduler](https://github.com/apache/incubator-dolphinscheduler.git)
文档.
diff --git a/docker/kubernetes/dolphinscheduler/README.md
b/docker/kubernetes/dolphinscheduler/README.md
index e558753..bd3afab 100644
--- a/docker/kubernetes/dolphinscheduler/README.md
+++ b/docker/kubernetes/dolphinscheduler/README.md
@@ -1,8 +1,30 @@
-# DolphinScheduler
+DolphinScheduler
+=================
+
+* [Introduction](#introduction)
+* [Prerequisites](#prerequisites)
+* [Installing the Chart](#installing-the-chart)
+* [Access DolphinScheduler UI](#access-dolphinscheduler-ui)
+* [Uninstalling the Chart](#uninstalling-the-chart)
+* [Support Matrix](#support-matrix)
+* [Configuration](#configuration)
+* [FAQ](#faq)
+ * [How to use MySQL as the DolphinScheduler's database instead of
PostgreSQL?](#how-to-use-mysql-as-the-dolphinschedulers-database-instead-of-postgresql)
+ * [How to support MySQL datasource in Datasource
manage?](#how-to-support-mysql-datasource-in-datasource-manage)
+ * [How to support Oracle datasource in Datasource
manage?](#how-to-support-oracle-datasource-in-datasource-manage)
+ * [How to support Python 2 pip and custom
requirements\.txt?](#how-to-support-python-2-pip-and-custom-requirementstxt)
+ * [How to support Python 3?](#how-to-support-python-3)
+ * [How to support Hadoop, Spark, Flink, Hive or
DataX?](#how-to-support-hadoop-spark-flink-hive-or-datax)
+ * [How to support Spark 3?](#how-to-support-spark-3)
+ * [How to support shared storage between Master, Worker and Api
server?](#how-to-support-shared-storage-between-master-worker-and-api-server)
+ * [How to support local file resource storage instead of HDFS and
S3?](#how-to-support-local-file-resource-storage-instead-of-hdfs-and-s3)
+ * [How to support S3 resource storage like
MinIO?](#how-to-support-s3-resource-storage-like-minio)
+ * [How to configure SkyWalking?](#how-to-configure-skywalking)
+
+## Introduction
[DolphinScheduler](https://dolphinscheduler.apache.org) is a distributed and
easy-to-expand visual DAG workflow scheduling system, dedicated to solving the
complex dependencies in data processing, making the scheduling system out of
the box for data processing.
-## Introduction
This chart bootstraps a
[DolphinScheduler](https://dolphinscheduler.apache.org) distributed deployment
on a [Kubernetes](http://kubernetes.io) cluster using the
[Helm](https://helm.sh) package manager.
## Prerequisites
@@ -377,11 +399,11 @@ docker build -t apache/dolphinscheduler:mysql-driver .
5. Modify image `repository` and update `tag` to `mysql-driver` in
`values.yaml`
-6. Modify postgresql `enabled` to `false`
+6. Modify postgresql `enabled` to `false` in `values.yaml`
-7. Modify externalDatabase (especially modify `host`, `username` and
`password`):
+7. Modify externalDatabase (especially modify `host`, `username` and
`password`) in `values.yaml`:
-```
+```yaml
externalDatabase:
type: "mysql"
driver: "com.mysql.jdbc.Driver"
@@ -577,5 +599,124 @@ Spark on YARN (Deploy Mode is `cluster` or `client`)
requires Hadoop support. Si
Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` exists
-For more information please refer to the
[incubator-dolphinscheduler](https://github.com/apache/incubator-dolphinscheduler.git)
documentation.
+### How to support Spark 3?
+
+In fact, the way to submit applications with `spark-submit` is the same,
regardless of Spark 1, 2 or 3. In other words, the semantics of `SPARK_HOME2`
is the second `SPARK_HOME` instead of `SPARK2`'s `HOME`, so just set
`SPARK_HOME2=/path/to/spark3`
+
+Take Spark 3.1.1 as an example:
+
+1. Download the Spark 3.1.1 release binary `spark-3.1.1-bin-hadoop2.7.tgz`
+
+2. Ensure that `common.sharedStoragePersistence.enabled` is turned on
+
+3. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)
+
+4. Copy the Spark 3.1.1 release binary into Docker container
+
+```bash
+kubectl cp spark-3.1.1-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft
+kubectl cp -n test spark-3.1.1-bin-hadoop2.7.tgz
dolphinscheduler-worker-0:/opt/soft # with test namespace
+```
+
+5. Attach the container and ensure that `SPARK_HOME2` exists
+
+```bash
+kubectl exec -it dolphinscheduler-worker-0 bash
+kubectl exec -n test -it dolphinscheduler-worker-0 bash # with test namespace
+cd /opt/soft
+tar zxf spark-3.1.1-bin-hadoop2.7.tgz
+rm -f spark-3.1.1-bin-hadoop2.7.tgz
+ln -s spark-3.1.1-bin-hadoop2.7 spark2 # or just mv
+$SPARK_HOME2/bin/spark-submit --version
+```
+
+The last command will print Spark version if everything goes well
+
+6. Verify Spark under a Shell task
+
+```
+$SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi
$SPARK_HOME2/examples/jars/spark-examples_2.12-3.1.1.jar
+```
+
+Check whether the task log contains the output like `Pi is roughly 3.146015`
+
+### How to support shared storage between Master, Worker and Api server?
+
+For example, Master, Worker and Api server may use Hadoop at the same time
+
+1. Modify the following configurations in `values.yaml`
+
+```yaml
+common:
+ sharedStoragePersistence:
+ enabled: false
+ mountPath: "/opt/soft"
+ accessModes:
+ - "ReadWriteMany"
+ storageClassName: "-"
+ storage: "20Gi"
+```
+
+`storageClassName` and `storage` need to be modified to actual values
+
+> **Note**: `storageClassName` must support the access mode: `ReadWriteMany`
+
+2. Put the Hadoop into the nfs
+
+3. Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` are correct
+
+### How to support local file resource storage instead of HDFS and S3?
+
+Modify the following configurations in `values.yaml`
+```yaml
+common:
+ configmap:
+ RESOURCE_STORAGE_TYPE: "HDFS"
+ RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
+ FS_DEFAULT_FS: "file:///"
+ fsFileResourcePersistence:
+ enabled: true
+ accessModes:
+ - "ReadWriteMany"
+ storageClassName: "-"
+ storage: "20Gi"
+```
+
+`storageClassName` and `storage` need to be modified to actual values
+
+> **Note**: `storageClassName` must support the access mode: `ReadWriteMany`
+
+### How to support S3 resource storage like MinIO?
+
+Take MinIO as an example: Modify the following configurations in `values.yaml`
+
+```yaml
+common:
+ configmap:
+ RESOURCE_STORAGE_TYPE: "S3"
+ RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
+ FS_DEFAULT_FS: "s3a://BUCKET_NAME"
+ FS_S3A_ENDPOINT: "http://MINIO_IP:9000";
+ FS_S3A_ACCESS_KEY: "MINIO_ACCESS_KEY"
+ FS_S3A_SECRET_KEY: "MINIO_SECRET_KEY"
+```
+
+`BUCKET_NAME`, `MINIO_IP`, `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` need to
be modified to actual values
+
+> **Note**: `MINIO_IP` can only use IP instead of domain name, because
DolphinScheduler currently doesn't support S3 path style access
+
+### How to configure SkyWalking?
+
+Modify SKYWALKING configurations in `values.yaml`:
+
+```yaml
+common:
+ configmap:
+ SKYWALKING_ENABLE: "true"
+ SW_AGENT_COLLECTOR_BACKEND_SERVICES: "127.0.0.1:11800"
+ SW_GRPC_LOG_SERVER_HOST: "127.0.0.1"
+ SW_GRPC_LOG_SERVER_PORT: "11800"
+```
+
+For more information please refer to the
[incubator-dolphinscheduler](https://github.com/apache/incubator-dolphinscheduler.git)
documentation.
diff --git a/docker/kubernetes/dolphinscheduler/values.yaml
b/docker/kubernetes/dolphinscheduler/values.yaml
index b307bc1..565adaf 100644
--- a/docker/kubernetes/dolphinscheduler/values.yaml
+++ b/docker/kubernetes/dolphinscheduler/values.yaml
@@ -67,7 +67,7 @@ externalZookeeper:
zookeeperRoot: "/dolphinscheduler"
common:
- ## ConfigMap
+ ## Configmap
configmap:
DOLPHINSCHEDULER_OPTS: ""
DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
@@ -146,7 +146,7 @@ master:
# requests:
# memory: "2Gi"
# cpu: "500m"
- ## ConfigMap
+ ## Configmap
configmap:
MASTER_SERVER_OPTS: "-Xms1g -Xmx1g -Xmn512m"
MASTER_EXEC_THREADS: "100"
@@ -215,7 +215,7 @@ worker:
# requests:
# memory: "2Gi"
# cpu: "500m"
- ## ConfigMap
+ ## Configmap
configmap:
LOGGER_SERVER_OPTS: "-Xms512m -Xmx512m -Xmn256m"
WORKER_SERVER_OPTS: "-Xms1g -Xmx1g -Xmn512m"
@@ -295,7 +295,7 @@ alert:
# requests:
# memory: "1Gi"
# cpu: "500m"
- ## ConfigMap
+ ## Configmap
configmap:
ALERT_SERVER_OPTS: "-Xms512m -Xmx512m -Xmn256m"
XLS_FILE_PATH: "/tmp/xls"
@@ -371,7 +371,7 @@ api:
# requests:
# memory: "1Gi"
# cpu: "500m"
- ## ConfigMap
+ ## Configmap
configmap:
API_SERVER_OPTS: "-Xms512m -Xmx512m -Xmn256m"
## Periodic probe of container liveness. Container will be restarted if the
probe fails. Cannot be updated.
