This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-11691-c191297c955945b62a6eb2cb7de581bfc5784646 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 08a38f1b8c5353228d3e1ba11e2eebb20619844f Author: jinniiii233 <[email protected]> AuthorDate: Fri Sep 18 15:02:26 2026 +0000 [Docs] Document DolphinScheduler integration with Docker Compose deployments (#11691) --- docs/en/tools/dolphinscheduler-integration.md | 90 +++++++++++++++++++++++++++ docs/sidebars.js | 3 +- docs/zh/tools/dolphinscheduler-integration.md | 90 +++++++++++++++++++++++++++ 3 files changed, 182 insertions(+), 1 deletion(-) diff --git a/docs/en/tools/dolphinscheduler-integration.md b/docs/en/tools/dolphinscheduler-integration.md new file mode 100644 index 0000000000..bcacf3895d --- /dev/null +++ b/docs/en/tools/dolphinscheduler-integration.md @@ -0,0 +1,90 @@ +# Integrating SeaTunnel with DolphinScheduler via Docker Compose + +A common question when deploying SeaTunnel through Docker Compose is: which `SEATUNNEL_HOME` path should DolphinScheduler be configured with so it can correctly submit SeaTunnel jobs? This document explains the reasoning and provides a minimal, ready-to-use example. + +## 1. What `SEATUNNEL_HOME` Means for Scheduler Integration + +DolphinScheduler does not call SeaTunnel over a network API. Instead, it directly executes SeaTunnel's launch script on the same machine (or container) it runs on, for example: + +``` +${SEATUNNEL_HOME}/bin/seatunnel.sh --config <job_config_path> +``` + +This means `SEATUNNEL_HOME` must point to a path **that the DolphinScheduler process itself can actually reach**, and that path must contain a complete SeaTunnel installation (`bin/`, `config/`, `lib/`, `connectors/`, etc.). If the path is empty or missing, DolphinScheduler will fail at job execution time with a "script not found" or "command not found" error. + +## 2. Host Paths vs. Container Paths + +This is the most common source of confusion. There are two distinct "points of view": + +- **Host path**: where the SeaTunnel installation actually lives on your machine or server's disk, e.g. `/home/user/seatunnel` or `./seatunnel`. +- **Container path**: what DolphinScheduler's container sees internally, e.g. `/opt/seatunnel`. Whether files exist at that path depends entirely on whether you mounted a host directory into it via `volumes`. + +**Key rule**: `SEATUNNEL_HOME` always describes the path as seen **from the environment DolphinScheduler's process runs in**, not from your local machine. If DolphinScheduler runs inside a container, `SEATUNNEL_HOME` must be a container-internal path (e.g. `/opt/seatunnel`), and that container path must be mounted to the host directory where SeaTunnel is actually installed. + +## 3. Deciding the Path When DolphinScheduler Runs on the Host + +If DolphinScheduler is not containerized and runs directly as a process on a physical or virtual machine: + +- `SEATUNNEL_HOME` should be set to the real host-side installation path, e.g. `/opt/module/seatunnel`. +- There is no volume-mounting concern here — DolphinScheduler's process and the SeaTunnel installation share the same filesystem, so the path is exactly what it appears to be. + +## 4. Deciding the Path When DolphinScheduler Also Runs in Docker or Kubernetes + +This is the case most people hit with Docker Compose. Two things are required: + +1. Mount the host's SeaTunnel installation directory into a path inside the DolphinScheduler container via `volumes`. +2. Set `SEATUNNEL_HOME` to the **mounted container-side path**, not the host path. + +The same logic applies in Kubernetes: "mounting" becomes `volumeMounts` combined with a `PersistentVolume` (or `hostPath`), but the principle is identical — `SEATUNNEL_HOME` should be set to the path as seen inside the Pod. + +## 5. Minimal Working Example (Using the Common Container Path `/opt/seatunnel`) + +```yaml +version: '3.8' + +services: + dolphinscheduler: + image: apache/dolphinscheduler-standalone-server:3.2.1 + container_name: dolphinscheduler + hostname: dolphinscheduler + ports: + - "12345:12345" + environment: + - SEATUNNEL_HOME=/opt/seatunnel + volumes: + # host ./seatunnel -> container /opt/seatunnel + # the host directory must contain a full SeaTunnel installation beforehand + - ./seatunnel:/opt/seatunnel:ro + - ./dolphinscheduler/logs:/opt/dolphinscheduler/logs + networks: + - ds-network + +networks: + ds-network: + driver: bridge +``` + +Prepare the host-side SeaTunnel installation directory (the official DolphinScheduler image does not bundle SeaTunnel itself — it must be downloaded and extracted manually): + +```bash +export version="3.0.0" +mkdir -p seatunnel dolphinscheduler/logs +wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"; +tar -zxvf "apache-seatunnel-${version}-bin.tar.gz" -C seatunnel --strip-components=1 +``` +:::caution Warning +Since 2.2.0-beta, connector plugins are no longer bundled by default. You must install them **before** mounting the directory read-only and starting the container, otherwise every submitted job will fail with a missing-connector error. +::: + +```bash +sh seatunnel/bin/install-plugin.sh 3.0.0 +``` + +After this, `./seatunnel` should contain `bin/`, `config/`, `lib/`, and similar subdirectories, and the same content will be visible inside the container at `/opt/seatunnel` once it starts. + +## 6. Required Volume Mounts and Network Assumptions + +- **Required mount**: the host's SeaTunnel installation directory → the container path referenced by `SEATUNNEL_HOME`. A read-only mount (`:ro`) is sufficient, since DolphinScheduler only needs to execute scripts, not write to the installation directory. +- **Recommended mounts**: DolphinScheduler's log directory and any directory holding job configuration files, so logs and configs remain accessible outside the container for debugging and version control. +- **Network assumption**: if SeaTunnel jobs need to reach data sources (databases, message queues, etc.) that are also deployed via Docker Compose, they must be on the same custom network as DolphinScheduler (e.g. `ds-network` above); otherwise containers cannot resolve each other by service name and you'll need to fall back to host IPs or additional network configuration. +- If SeaTunnel jobs need to reach services on the host machine that are outside the Docker network, be aware of Docker's default network isolation — you may need `host.docker.internal` (on Docker Desktop) or explicit host-network configuration. diff --git a/docs/sidebars.js b/docs/sidebars.js index 6b03c8ce66..20b6f38a1a 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -386,7 +386,8 @@ const sidebars = { "tools/overview", "tools/seatunnel-skill", "tools/seatunnel-mcp", - "tools/x2seatunnel" + "tools/x2seatunnel", + "tools/dolphinscheduler-integration" ] }, { diff --git a/docs/zh/tools/dolphinscheduler-integration.md b/docs/zh/tools/dolphinscheduler-integration.md new file mode 100644 index 0000000000..4c3cb58a7d --- /dev/null +++ b/docs/zh/tools/dolphinscheduler-integration.md @@ -0,0 +1,90 @@ +# 使用 Docker Compose 将 SeaTunnel 与 DolphinScheduler 集成 + +当 SeaTunnel 通过 Docker Compose 部署时，一个常见的问题是：DolphinScheduler 应该配置哪个 `SEATUNNEL_HOME` 路径才能正确提交作业？本文档说明这个问题背后的原理，并给出一个可直接使用的最小示例。 + +## 1. `SEATUNNEL_HOME` 对调度器集成意味着什么 + +DolphinScheduler 并不是通过网络接口远程调用 SeaTunnel，而是在自己所在的机器（或容器）上，直接执行 SeaTunnel 提供的启动脚本，例如： + +``` +${SEATUNNEL_HOME}/bin/seatunnel.sh --config <job_config_path> +``` + +因此 `SEATUNNEL_HOME` 必须指向 **DolphinScheduler 进程实际能够访问到的一个路径**，这个路径下必须包含完整的 SeaTunnel 安装目录（`bin/`、`config/`、`lib/`、`connectors/` 等）。如果这个路径下是空的或者不存在，DolphinScheduler 执行任务时会直接报"找不到脚本"或"命令不存在"的错误。 + +## 2. 宿主机路径 与 容器内路径 的区别 + +这是最容易搞混的地方，需要分清两个"视角"： + +- **宿主机路径（Host Path）**：SeaTunnel 安装包实际存放在你电脑或服务器磁盘上的位置，例如 `/home/user/seatunnel` 或 `./seatunnel`。 +- **容器内路径（Container Path）**：DolphinScheduler 容器内部看到的路径，例如 `/opt/seatunnel`。这个路径是否存在文件，取决于你有没有通过 `volumes` 把宿主机目录挂载进去。 + +**关键点**：`SEATUNNEL_HOME` 这个环境变量，配置的永远是 **DolphinScheduler 进程自己所在环境看到的路径**，而不是你电脑上的路径。如果 DolphinScheduler 跑在容器里，`SEATUNNEL_HOME` 就必须写容器内路径（如 `/opt/seatunnel`），并确保这个容器路径通过 volume 挂载对应到了宿主机上真正装了 SeaTunnel 的目录。 + +## 3. DolphinScheduler 运行在宿主机上时，如何确定路径 + +如果你没有用容器运行 DolphinScheduler，而是直接在物理机 / 虚拟机上以进程方式运行它，那么： + +- `SEATUNNEL_HOME` 直接填宿主机上 SeaTunnel 的真实安装路径，例如 `/opt/module/seatunnel`。 +- 不涉及任何 volume 挂载问题，DolphinScheduler 进程和 SeaTunnel 安装包在同一套文件系统里，路径所见即所得。 + +## 4. DolphinScheduler 也运行在 Docker（或 Kubernetes）中时，如何确定路径 + +这是 Docker Compose 场景下最常遇到的情况。此时必须做两件事： + +1. 把宿主机上的 SeaTunnel 安装目录，通过 `volumes` 挂载到 DolphinScheduler 容器内的某个路径。 +2. 把 `SEATUNNEL_HOME` 设置为**挂载后的容器内路径**，而不是宿主机路径。 + +在 Kubernetes 环境下同理，只是"挂载"变成了 `volumeMounts` + `PersistentVolume`（或 `hostPath`），思路完全一致：Pod 内部看到的路径才是 `SEATUNNEL_HOME` 应该填的值。 + +## 5. 最小可用示例（使用常见容器路径 `/opt/seatunnel`） + +```yaml +version: '3.8' + +services: + dolphinscheduler: + image: apache/dolphinscheduler-standalone-server:3.2.1 + container_name: dolphinscheduler + hostname: dolphinscheduler + ports: + - "12345:12345" + environment: + - SEATUNNEL_HOME=/opt/seatunnel + volumes: + # 宿主机 ./seatunnel 目录 -> 容器内 /opt/seatunnel + # 宿主机这个目录下需要预先放好完整的 SeaTunnel 安装包内容 + - ./seatunnel:/opt/seatunnel:ro + - ./dolphinscheduler/logs:/opt/dolphinscheduler/logs + networks: + - ds-network + +networks: + ds-network: + driver: bridge +``` + +准备宿主机上的 SeaTunnel 安装目录（Docker 官方镜像不自带 SeaTunnel 程序本体，需要手动下载解压）： + +```bash +export version="3.0.0" +mkdir -p seatunnel dolphinscheduler/logs +wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"; +tar -zxvf "apache-seatunnel-${version}-bin.tar.gz" -C seatunnel --strip-components=1 +``` +:::caution 警告 +从 2.2.0-beta 版本开始，连接器插件默认不再随安装包一起打包。你必须在把目录以只读方式挂载并启动容器**之前**安装好这些插件，否则所有提交的作业都会因为缺少连接器而失败。 +::: + +```bash +sh seatunnel/bin/install-plugin.sh 3.0.0 +``` + +完成后，`./seatunnel` 目录下应包含 `bin/`、`config/`、`lib/` 等子目录，容器启动后即可在 `/opt/seatunnel` 下看到同样的内容。 + +## 6. 所需的 Volume 挂载与网络假设 + +- **必须挂载**：宿主机的 SeaTunnel 安装目录 → 容器内 `SEATUNNEL_HOME` 指向的路径（只读挂载 `:ro` 即可，因为 DolphinScheduler 只需要执行脚本，不需要写入安装目录）。 +- **建议挂载**：DolphinScheduler 的日志目录、以及存放作业配置文件（job config）的目录，方便在容器外查看任务配置和日志，也便于版本管理和排查问题。 +- **网络假设**：如果 SeaTunnel 需要连接的数据源（数据库、消息队列等）也用 Docker Compose 部署，需确保它们和 DolphinScheduler 处于同一个自定义网络（如上例中的 `ds-network`），否则容器间无法通过服务名互相访问，必须改用宿主机 IP 或额外的网络配置。 +- 如果 SeaTunnel 任务需要访问宿主机上的其他服务（不在 Docker 网络内），需要注意 Docker 默认的网络隔离，可能需要使用 `host.docker.internal`（Docker Desktop 环境）或显式配置宿主机网络访问。
