This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12459-5ff1fe6dd989657ab6291387d313c97891de398a in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 8622bbdc883704faecb14e94bd73313291647438 Author: Jast <[email protected]> AuthorDate: Sat Nov 7 06:39:43 2026 +0000 [Docs] Fix documentation inaccuracies verified against source code (#12459) Co-authored-by: David Zollo <[email protected]> Co-authored-by: Claude Sonnet 5.5 <[email protected]> Co-authored-by: jast <[email protected]> --- docs/en/connectors/sink/Databend.md | 2 +- docs/en/connectors/sink/Hudi.md | 2 +- docs/en/connectors/sink/Jdbc.md | 4 +- docs/en/connectors/sink/Snowflake.md | 8 +- docs/en/connectors/source/Snowflake.md | 11 +- docs/en/engines/zeta/checkpoint-storage.md | 4 +- docs/en/engines/zeta/hybrid-cluster-deployment.md | 2 +- docs/en/engines/zeta/log-analysis-with-ai.md | 7 +- docs/en/engines/zeta/logging.md | 2 +- docs/en/engines/zeta/resource-isolation.md | 2 +- docs/en/engines/zeta/state-storage-and-recovery.md | 177 +++++++++------------ docs/en/engines/zeta/tuning-guide.md | 18 ++- .../submit-job-to-remote-zeta-cluster.md | 14 +- docs/sidebars.js | 3 + docs/zh/connectors/sink/Databend.md | 2 +- docs/zh/connectors/sink/Hudi.md | 2 +- docs/zh/connectors/sink/Jdbc.md | 4 +- docs/zh/connectors/sink/Snowflake.md | 8 +- docs/zh/connectors/source/Snowflake.md | 11 +- docs/zh/engines/zeta/checkpoint-storage.md | 2 +- docs/zh/engines/zeta/hybrid-cluster-deployment.md | 2 +- docs/zh/engines/zeta/log-analysis-with-ai.md | 5 +- docs/zh/engines/zeta/logging.md | 2 +- docs/zh/engines/zeta/resource-isolation.md | 2 +- docs/zh/engines/zeta/rest-api-job-lifecycle.md | 2 +- docs/zh/engines/zeta/state-storage-and-recovery.md | 159 ++++++++---------- docs/zh/engines/zeta/tuning-guide.md | 16 +- .../submit-job-to-remote-zeta-cluster.md | 10 +- 28 files changed, 219 insertions(+), 264 deletions(-) diff --git a/docs/en/connectors/sink/Databend.md b/docs/en/connectors/sink/Databend.md index ea27ed9c6b2..22ecf150932 100644 --- a/docs/en/connectors/sink/Databend.md +++ b/docs/en/connectors/sink/Databend.md @@ -49,7 +49,7 @@ They can be downloaded via install-plugin.sh or from the Maven central repositor | url | String | Yes | - | Databend JDBC connection URL. It must start with `jdbc:databend://` | | username | String | Yes | - | Databend database username | | password | String | Yes | - | Databend database password | -| database | String | No | - | Databend database name (together with `table`) that locates the target table to write | +| database | String | No | - | Databend database name (together with `table`) that locates the target table to write. No default is applied when unset | | table | String | No | - | Databend table name | | batch_size | Integer | No | 1000 | Number of records for batch writing | | auto_commit | Boolean | No | true | Whether to auto-commit transactions | diff --git a/docs/en/connectors/sink/Hudi.md b/docs/en/connectors/sink/Hudi.md index 5a08174fc06..e6449c3acf9 100644 --- a/docs/en/connectors/sink/Hudi.md +++ b/docs/en/connectors/sink/Hudi.md @@ -114,7 +114,7 @@ Note: When this configuration corresponds to a single table, you can flatten the ### op_type [enum] -`op_type` The operation type of hudi table. The value is `insert` or `upsert` or `bulk_insert`. +`op_type` The operation type of hudi table. The value is `INSERT`, `UPSERT`, or `BULK_INSERT`. ### batch_interval_ms [Int] diff --git a/docs/en/connectors/sink/Jdbc.md b/docs/en/connectors/sink/Jdbc.md index a451d04c326..1ad92352f71 100644 --- a/docs/en/connectors/sink/Jdbc.md +++ b/docs/en/connectors/sink/Jdbc.md @@ -299,8 +299,8 @@ If one dialect not supported by SeaTunnel, it will use the default dialect `Gene | SqlServer | Tablestore | Teradata | | Vertica | OceanBase | XUGU | | IRIS | Inceptor | Highgo | -| DSQL | | | -| YashanDB | CockroachDB | | +| OpenGauss | Presto | Dsql | +| DuckDB | YashanDB | CockroachDB | ### database [string] The target database or catalog used in generated SQL mode. This option is required when `generate_sink_sql = true` and must not be combined with `query`. diff --git a/docs/en/connectors/sink/Snowflake.md b/docs/en/connectors/sink/Snowflake.md index fa736b8a7f8..a98c45055a7 100644 --- a/docs/en/connectors/sink/Snowflake.md +++ b/docs/en/connectors/sink/Snowflake.md @@ -43,14 +43,16 @@ Write data to Snowflake through JDBC. The sink supports batch and streaming jobs | INT<br/>INTEGER | INT | | BIGINT | LONG | | DECIMAL<br/>NUMERIC<br/>NUMBER<br/> | DECIMAL(p, s) | -| DECIMAL(p, s) (with `p > 38`) | DECIMAL(38, 18) | +| DECIMAL(p, s) (with `p > 38`) | DECIMAL(38, s - (p - 38)) (minimum scale 0) | | REAL<br/>FLOAT4 | FLOAT | | DOUBLE<br/>DOUBLE PRECISION<br/>FLOAT8<br/>FLOAT | DOUBLE | | CHAR<br/>CHARACTER<br/>VARCHAR<br/>STRING<br/>TEXT<br/>VARIANT<br/>OBJECT | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_LTZ<br/>TIMESTAMP_NTZ<br/>TIMESTAMP_TZ | TIMESTAMP | -| BINARY<br/>VARBINARY<br/>GEOGRAPHY<br/>GEOMETRY | BYTES | +| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_NTZ | TIMESTAMP | +| TIMESTAMP_LTZ<br/>TIMESTAMP_TZ | TIMESTAMP_TZ | +| BINARY<br/>VARBINARY | BYTES | +| GEOGRAPHY<br/>GEOMETRY | STRING | ## Sink Options diff --git a/docs/en/connectors/source/Snowflake.md b/docs/en/connectors/source/Snowflake.md index b9b12d2ba5b..07357381268 100644 --- a/docs/en/connectors/source/Snowflake.md +++ b/docs/en/connectors/source/Snowflake.md @@ -45,16 +45,15 @@ Read data from Snowflake through JDBC. SeaTunnel uses the official Snowflake JDB | INT<br/>INTEGER | INT | | BIGINT | LONG | | DECIMAL<br/>NUMERIC<br/>NUMBER<br/> | DECIMAL(p, s) | -| DECIMAL(p, s) (with `p > 38`) | DECIMAL(38, 18) | | REAL<br/>FLOAT4 | FLOAT | | DOUBLE<br/>DOUBLE PRECISION<br/>FLOAT8<br/>FLOAT | DOUBLE | | CHAR<br/>CHARACTER<br/>VARCHAR<br/>STRING<br/>TEXT<br/>VARIANT<br/>OBJECT | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_LTZ<br/>TIMESTAMP_NTZ<br/>TIMESTAMP_TZ | TIMESTAMP | +| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_NTZ | TIMESTAMP | +| TIMESTAMP_LTZ<br/>TIMESTAMP_TZ | TIMESTAMP_TZ | | BINARY<br/>VARBINARY | BYTES | -| GEOGRAPHY (WKB or EWKB)<br/>GEOMETRY (WKB or EWKB) | BYTES | -| GEOGRAPHY (GeoJSON, WKT or EWKT)<br/>GEOMETRY (GeoJSON, WKB or EWKB) | STRING | +| GEOGRAPHY<br/>GEOMETRY | STRING | ## Source Options @@ -64,7 +63,7 @@ Read data from Snowflake through JDBC. SeaTunnel uses the official Snowflake JDB | driver | String | Yes | - | JDBC driver class name. Use `net.snowflake.client.jdbc.SnowflakeDriver` for Snowflake. | | username | String | No | - | Username for the Snowflake account. | | password | String | No | - | Password for the Snowflake account. | -| query | String | Yes | - | SELECT statement used to read data. The column list of the SELECT defines the output schema; select only the columns you need. | +| query | String | No | - | SELECT statement used to read data. Required when neither `table_path` nor `table_list` is configured. The column list of the SELECT defines the output schema; select only the columns you need. | | connection_check_timeout_sec | Int | No | 30 | Seconds to wait for the connection check before failing. | | partition_column | String | No | - | Column used to split data for parallel reading. Supports numeric columns and string columns (with `split.string_split_mode`); only one column can be configured. | | partition_lower_bound | String | No | - | Lower bound of `partition_column` for range splitting. If not set, SeaTunnel queries the minimum value. | @@ -151,7 +150,7 @@ source { - Use the `Jdbc` plugin name for Snowflake jobs and set `driver = "net.snowflake.client.jdbc.SnowflakeDriver"`. - Place the Snowflake JDBC driver jar in `$SEATUNNEL_HOME/plugins/jdbc/lib/` before running the job. - When reading in parallel, `partition_column`, `partition_lower_bound`, `partition_upper_bound`, and `partition_num` must describe the same numeric column range. -- Snowflake geospatial columns are returned as bytes or string depending on Snowflake JDBC URL parameters such as `GEOGRAPHY_OUTPUT_FORMAT`. +- Snowflake geospatial columns are mapped to the SeaTunnel `STRING` type. ## Changelog diff --git a/docs/en/engines/zeta/checkpoint-storage.md b/docs/en/engines/zeta/checkpoint-storage.md index d366382f62b..f92132336be 100644 --- a/docs/en/engines/zeta/checkpoint-storage.md +++ b/docs/en/engines/zeta/checkpoint-storage.md @@ -340,10 +340,10 @@ seatunnel: namespace: # checkpoint storage parent path, the default value is /seatunnel/checkpoint/ storage.type: hdfs fs.defaultFS: hdfs://localhost:9000 - // if you used kerberos, you can config like this: + # if you used kerberos, you can config like this: kerberosPrincipal: your-kerberos-principal kerberosKeytabFilePath: your-kerberos-keytab - // if you need hdfs-site config, you can config like this: + # if you need hdfs-site config, you can config like this: hdfs_site_path: /path/to/your/hdfs_site_path ``` diff --git a/docs/en/engines/zeta/hybrid-cluster-deployment.md b/docs/en/engines/zeta/hybrid-cluster-deployment.md index abf1489d112..87a3deb7c3d 100644 --- a/docs/en/engines/zeta/hybrid-cluster-deployment.md +++ b/docs/en/engines/zeta/hybrid-cluster-deployment.md @@ -494,7 +494,7 @@ mkdir -p $SEATUNNEL_HOME/logs ./bin/seatunnel-cluster.sh -d ``` -The logs will be written to `$SEATUNNEL_HOME/logs/seatunnel-engine-server.log` +The logs will be written to `$SEATUNNEL_HOME/logs/` — for the default master-and-worker deployment this is `seatunnel-engine-server.log` ## 8. Submit And Manage Jobs diff --git a/docs/en/engines/zeta/log-analysis-with-ai.md b/docs/en/engines/zeta/log-analysis-with-ai.md index 236db16a22e..662606dd1d6 100644 --- a/docs/en/engines/zeta/log-analysis-with-ai.md +++ b/docs/en/engines/zeta/log-analysis-with-ai.md @@ -55,7 +55,8 @@ When logs are mixed, use the `ST-JID` value to select one job. For example: ```shell JOB_ID=<job-id> -grep -F "[${JOB_ID}]" "$SEATUNNEL_HOME/logs/seatunnel-engine-server.log" > job.log +# seatunnel-engine-server.log (hybrid); seatunnel-engine-master.log / -worker.log (separated) +grep -hF "[${JOB_ID}]" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log > job.log ``` If per-job routing is enabled, inspect `job-<job-id>.log`. In a multi-node Zeta cluster, collect @@ -65,12 +66,12 @@ REST endpoints: ```text GET http://<master-host>:8080/logs/<job-id> GET http://<master-host>:8080/logs?format=json -GET http://<node-host>:5801/log +GET http://<node-host>:8080/log ``` The first endpoint retrieves matching logs across Zeta nodes. The last endpoint reads logs from one node. A configured context path or dynamic HTTP port changes these URLs. See -[RESTful API V2](rest-api-v2.md#get-logs-from-all-nodes) for the complete behavior. +[RESTful API V2](rest-api-v2.md#get-log-content-from-a-single-node) for the complete behavior. For Kubernetes, preserve logs from all relevant master and worker pods. Start with the failure window and include previous container logs when a pod restarted: diff --git a/docs/en/engines/zeta/logging.md b/docs/en/engines/zeta/logging.md index 3abb488629b..9410d6e0922 100644 --- a/docs/en/engines/zeta/logging.md +++ b/docs/en/engines/zeta/logging.md @@ -119,7 +119,7 @@ the [REST-API](rest-api-v2.md). ### Scheduled deletion of old logs -SeaTunnel supports scheduled deletion of old log files to prevent disk space exhaustion. You can add the following configuration in the `seatunnel.yml` file: +SeaTunnel supports scheduled deletion of old log files to prevent disk space exhaustion. You can add the following configuration in the `seatunnel.yaml` file: ```yaml seatunnel: diff --git a/docs/en/engines/zeta/resource-isolation.md b/docs/en/engines/zeta/resource-isolation.md index 27a85b52d77..781c7d156cc 100644 --- a/docs/en/engines/zeta/resource-isolation.md +++ b/docs/en/engines/zeta/resource-isolation.md @@ -47,7 +47,7 @@ SeaTunnel can add `tag` to each worker node, when you submit job you can use `ta 2. add `tag_filter` to your job config -```hacon +```hocon env { parallelism = 1 job.mode = "BATCH" diff --git a/docs/en/engines/zeta/state-storage-and-recovery.md b/docs/en/engines/zeta/state-storage-and-recovery.md index c60b69539f9..b29048b9c71 100644 --- a/docs/en/engines/zeta/state-storage-and-recovery.md +++ b/docs/en/engines/zeta/state-storage-and-recovery.md @@ -13,10 +13,8 @@ long-running streaming jobs. | Storage Category | Purpose | Default Location | |---|---|---| | Checkpoint | Fault-tolerant snapshots of pipeline operator state | `seatunnel.yaml` `checkpoint.storage` | -| Savepoint | User-triggered named checkpoint for planned stop/restart | Same path as checkpoint, different directory | -| IMap / MapStore | Distributed in-memory state (job metadata, running job data, metrics) | `hazelcast.yaml` MapStore base-dir | -| WAL (Write-Ahead Log) | Durability log for IMap persistence | Same MapStore base-dir | -| History Job Metadata | Completed/failed job records | MapStore base-dir | +| Savepoint | User-triggered named checkpoint for planned stop/restart | Same checkpoint storage, under the job's own directory | +| IMap / MapStore | Distributed in-memory state (job metadata, job state, history) | In memory; optionally persisted through the Hazelcast MapStore configured in `hazelcast.yaml` | --- @@ -48,12 +46,10 @@ For a CDC job this includes: seatunnel: engine: checkpoint: - interval: 10000 # milliseconds between checkpoints - timeout: 60000 # checkpoint completion timeout (ms) - max-concurrent: 1 # max concurrent in-flight checkpoints - tolerable-failure: 2 # allowed consecutive checkpoint failures + interval: 10000 # milliseconds between checkpoints + timeout: 60000 # checkpoint completion timeout (ms) storage: - type: hdfs # hdfs | localfile (deprecated) + type: hdfs # hdfs (also supports S3 / local file via the HDFS API) | localfile (deprecated) plugin-config: namespace: /seatunnel/checkpoint/ # must end with / # For S3: @@ -85,7 +81,7 @@ seatunnel: | Aspect | Checkpoint | Savepoint | |---|---|---| -| Trigger | Periodic / automatic | Manual (`seatunnel.sh -r <jobId> --savepoint`) | +| Trigger | Periodic / automatic | Manual (`$SEATUNNEL_HOME/bin/seatunnel.sh --savepoint <jobId>`) | | Purpose | Fault tolerance | Planned stop, upgrade, migration | | Lifecycle | Managed by engine | Managed by operator | | Retention | Auto-rotated | Kept until manually deleted | @@ -94,7 +90,7 @@ seatunnel: ```bash # Stop a running job and create a savepoint -$SEATUNNEL_HOME/bin/seatunnel.sh --stop-job <job-id> --savepoint +$SEATUNNEL_HOME/bin/seatunnel.sh --savepoint <job-id> # Or via REST API v2 curl -X POST http://<master>:8080/stop-job \ @@ -123,14 +119,9 @@ $SEATUNNEL_HOME/bin/seatunnel.sh --config job.conf --restore-with-checkpoint <jo ### Savepoint path layout -``` -<namespace>/ - savepoint/ - <job-id>/ - <savepoint-timestamp>/ - <pipeline-id>/ - <task-location>/state-data -``` +A savepoint is a checkpoint of savepoint type: it is written to the same checkpoint storage +configured above, under the directory of the job it belongs to (`<namespace>/<job-id>/`). There is +no separate savepoint root directory. ### Safe cleanup @@ -143,45 +134,45 @@ Deleting an active savepoint mid-restore causes the job to fail with a "state no ### What IMap stores -SeaTunnel Engine uses Hazelcast IMap as its distributed in-memory key-value store. The following -logical maps are persisted: +SeaTunnel Engine uses Hazelcast IMap as its distributed in-memory key-value store. The main +logical maps used by the engine are: | IMap Name | Content | |---|---| -| `running-job-state` | Current state machine status of each running job | -| `running-job-metrics` | Real-time throughput, latency, and record count metrics | -| `running-pipeline-state` | Pipeline-level state for each logical pipeline | -| `finished-job-state` | Terminal state for completed, cancelled, or failed jobs | -| `finished-job-metrics` | Final metrics snapshot after job termination | +| `engine_runningJobInfo` | Submitted job information and initialization metadata (job id, job name, etc.; metrics live in `engine_finishedJobMetrics` after completion) | +| `engine_runningJobState` | Current state machine status of running jobs and pipelines | +| `engine_stateTimestamps` | Timestamps of job/pipeline state transitions | +| `engine_finishedJobState` | Terminal state for completed, canceled, or failed jobs | +| `engine_finishedJobMetrics` | Final metrics snapshot after job termination | +| `engine_finishedJobVertexInfo` | Execution vertex information of finished jobs | ### MapStore (disk-backed persistence) -Hazelcast MapStore writes IMap entries to local disk so they survive process restarts. This is -**separate** from checkpoint storage. +By default IMap data is memory-only (replicated across nodes according to the backup count). If all +nodes stop, the data is lost unless the MapStore persistence is enabled. When enabled, the Hazelcast +MapStore writes IMap entries to an external file system (HDFS, S3, or local files through the HDFS +API) so they survive full cluster restarts. This is **separate** from checkpoint storage. -Default storage path: configured in `hazelcast.yaml`: +The persistence is configured in `hazelcast.yaml` (see +[Separated Zeta Cluster Deployment](separated-cluster-deployment.md) for details): ```yaml map: - seatunnel: + engine*: map-store: enabled: true - initial-load-mode: EAGER + initial-mode: EAGER + factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory properties: - hazelcast.fs.base-dir: /tmp/seatunnel/imap # absolute path - hazelcast.fs.write-behind-delay-seconds: 1 + type: hdfs + namespace: /tmp/seatunnel/imap # storage namespace (default /seatunnel-imap when unset) + clusterName: seatunnel-cluster + storage.type: hdfs + fs.defaultFS: hdfs://localhost:9000 ``` -MapStore directory layout: - -``` -<hazelcast.fs.base-dir>/ - maps/ - running-job-state/ - running-job-metrics/ - finished-job-state/ - finished-job-metrics/ -``` +In separated cluster mode only the Master node stores IMap data, so this configuration takes effect +on the Master node. ### Relationship between IMap, MapStore, and Checkpoint @@ -191,38 +182,33 @@ IMap / MapStore <─────────────────── ``` They are **independent**. Deleting checkpoint storage does not affect IMap, and vice versa. -A job can be restarted from a checkpoint even if the IMap MapStore is wiped, **but the job ID -and pipeline mapping must be re-submitted** because running-job-state is lost. +A job can be restarted from a checkpoint even if the IMap MapStore data is wiped, **but the job ID +and pipeline mapping must be re-submitted** because the running job state is lost. --- -## 4. Write-Ahead Log (WAL) - -Hazelcast uses a write-ahead log for MapStore durability. WAL files accumulate under: - -``` -<hazelcast.fs.base-dir>/ - wal/ - <imap-name>-<partition>.wal -``` - -### WAL growth in long-running CDC jobs - -Each CDC binlog event that updates `running-job-metrics` or `running-pipeline-state` generates a -WAL write. Over time (days or weeks) WAL files can grow into several GB if: +## 4. MapStore File Maintenance -- `write-behind-delay-seconds` is set very low (high flush frequency) -- The job processes millions of events per second +The IMap persistence is an append-only write-ahead log: `FileMapStore` delegates to `IMapFileStorage`, +which appends every state change through the WAL writer into region files under the configured +`namespace`. Entries of all jobs are appended into the same region files, so the files keep growing +while jobs keep running. **Mitigation:** -```yaml -hazelcast.fs.write-behind-delay-seconds: 5 # increase flush interval -hazelcast.fs.compaction-threshold: 1000 # trigger compaction after N entries +- Let finished jobs expire: `history-job-expire-minutes` removes finished-job records from the + IMaps, which stops them from being persisted again. +- Monitor the namespace directory on the master node: + +```bash +du -sh /tmp/seatunnel/imap/ # replace with the namespace you configured in hazelcast.yaml (local file storage, i.e. fs.defaultFS starts with file:///) +hadoop fs -du -s -h <fs-uri>/<namespace> # for HDFS or object-store backed MapStore ``` -WAL files for **finished jobs** can be safely compacted or removed after the corresponding IMap -entries have been flushed to MapStore files. Do not delete WAL files for running jobs. +Do not delete MapStore files selectively: there are no job-scoped files to pick out, and removing +files by hand can corrupt IMap replay on the next master start. Size the namespace volume instead, +and wipe the whole namespace only while the cluster is stopped and when losing all running-job +IMap state is acceptable. --- @@ -243,11 +229,10 @@ seatunnel: | Keep the job removed after a cluster restart (IMap persistence enabled) | Yes (the expiration time is persisted with the entry, so an expired job is not loaded again and a job that has not expired yet keeps its remaining time) | | Delete MapStore persistence files for that job | **No** (the expired records stay in the IMap storage files) | | Delete checkpoint storage directories | **No** | -| Delete savepoint directories | **No** | -| Delete WAL files | **No** (only indirectly via compaction) | +| Delete savepoint data | **No** | -**Key takeaway**: `history-job-expire-minutes` only cleans up the job metadata stored in -`finished-job-state`. Checkpoint and savepoint directories on HDFS / S3 / local are **never** +**Key takeaway**: `history-job-expire-minutes` only cleans up the job metadata stored in the +finished-job IMaps. Checkpoint and savepoint directories on HDFS / S3 / local are **never** touched by this setting. You must manage them separately. --- @@ -264,7 +249,7 @@ touched by this setting. You must manage them separately. | PostgreSQL CDC (logical replication) | 1–50 KB per table | | Oracle CDC (LogMiner) | 50 KB – 2 MB | -Formula: `checkpoint_size ≈ N_tables × avg_splits × state_per_split × max_concurrent_checkpoints` +Formula: `checkpoint_size ≈ N_tables × avg_splits × state_per_split` (only one checkpoint is in flight at a time, so no concurrency factor applies) ### Storage sizing rule of thumb @@ -274,26 +259,20 @@ Retain **at least 3 checkpoints** per job at all times. Allocate: storage_needed = checkpoint_size × 3 × safety_factor(1.5) ``` -### IMap / WAL sizing +### IMap storage sizing Each running CDC job stores approximately: -- 50–200 bytes per pipeline in `running-job-state` -- 1–2 KB per pipeline per metric flush in `running-job-metrics` - -For a cluster running 100 CDC jobs: - -``` -imap_memory ≈ 100 × 200B ≈ 20 KB (state) -imap_memory ≈ 100 × 1KB × flush_rate ≈ manageable -``` +- 50–200 bytes per pipeline of job state +- 1–2 KB per pipeline per metrics flush -WAL disk: plan 2–5 GB per node for a busy CDC cluster. Mount a dedicated disk partition and monitor -with: +For a cluster running 100 CDC jobs this is in the order of a few MB of IMap data. The MapStore +namespace disk usage grows with the state transition history; plan a few GB per long-running +cluster and monitor it with: ```bash -du -sh $SEATUNNEL_HOME/imap/wal/ -watch -n 60 'du -sh /tmp/seatunnel/imap/' +du -sh /tmp/seatunnel/imap/ # local file storage (fs.defaultFS starts with file:///) +hadoop fs -du -s -h <fs-uri>/<namespace> # for HDFS or object-store backed MapStore ``` --- @@ -331,32 +310,27 @@ state is rotated out. --- -### IMap / WAL directory growing +### MapStore namespace directory growing -**Symptom**: `/tmp/seatunnel/imap/` fills the disk on master/worker nodes. +**Symptom**: The configured MapStore `namespace` directory fills the disk on the master node. **Diagnosis**: ```bash -du -sh /tmp/seatunnel/imap/wal/ -du -sh /tmp/seatunnel/imap/maps/ +du -sh /tmp/seatunnel/imap/ # replace with the namespace you configured in hazelcast.yaml (local file storage, i.e. fs.defaultFS starts with file:///) +hadoop fs -du -s -h <fs-uri>/<namespace> # for HDFS or object-store backed MapStore ``` **Root cause candidates**: -- `running-job-metrics` updates at every checkpoint for every running job -- `write-behind-delay-seconds` is too low (default 1 s) -- WAL compaction is not happening frequently enough +- Finished jobs are not expiring (`history-job-expire-minutes` is not set or too large) +- Too many running jobs continuously updating their state and metrics **Fix**: -```yaml -# hazelcast.yaml -hazelcast.fs.write-behind-delay-seconds: 10 -hazelcast.fs.compaction-threshold: 500 -``` - -After restarting the engine, old WAL files from finished jobs will be compacted during startup. +Enable/tune `history-job-expire-minutes` in `seatunnel.yaml`. MapStore region files cannot be +trimmed per job; reclaiming their space requires stopping the cluster and wiping the whole +namespace, which discards the IMap state of all jobs. --- @@ -380,12 +354,11 @@ After restarting the engine, old WAL files from finished jobs will be compacted Before deleting any state directory: -- [ ] Confirm the job is in `FINISHED`, `CANCELLED`, or `FAILED` terminal state +- [ ] Confirm the job is in `FINISHED`, `CANCELED`, or `FAILED` terminal state - [ ] Confirm you will not restore from checkpoint or savepoint - [ ] Confirm the job ID is not referenced in any monitoring or alerting rule - [ ] For checkpoint storage: delete `<namespace>/<job-id>/` recursively -- [ ] For savepoint: delete `<namespace>/savepoint/<job-id>/` recursively -- [ ] For MapStore/WAL: restart the engine after manual deletion to trigger reload +- [ ] For MapStore data: only a full namespace wipe (cluster stopped) is safe; selective per-job deletion is not supported --- diff --git a/docs/en/engines/zeta/tuning-guide.md b/docs/en/engines/zeta/tuning-guide.md index c49b2b5ee9b..ef96aab52aa 100644 --- a/docs/en/engines/zeta/tuning-guide.md +++ b/docs/en/engines/zeta/tuning-guide.md @@ -20,11 +20,14 @@ If the SeaTunnel Engine cluster responds slowly or hangs, it may be due to insuf ##### Troubleshooting Process 1. Check JVM heap memory usage in real time - Use the `jcmd` command to check JVM heap memory usage, where `<pid>` is the PID of the SeaTunnel Engine process. + Use the `jcmd` command, where `<pid>` is the PID of the SeaTunnel Engine process. On JDK 9+ use `jcmd <pid> GC.heap_info`; on JDK 8 use `jmap -heap <pid>` (removed in JDK 9+). ```bash + # JDK 9+ + jcmd <pid> GC.heap_info + # JDK 8 jmap -heap <pid> ``` - Example output: + Example output from a JDK 8 host (`jmap -heap` style): ```shell Attaching to process ID 2111950, please wait... Debugger attached successfully. @@ -169,7 +172,8 @@ Use the following decision tree to narrow down the root cause of slow operations ```bash # Check the slow operation log frequency and timing -grep "SlowOperationDetector" $SEATUNNEL_HOME/logs/seatunnel-server.log | tail -50 +grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -50 +# seatunnel-engine-server.log on the default hybrid deployment; -master.log / -worker.log when roles are separated ``` Correlate the timestamps with: @@ -190,7 +194,7 @@ free -h **REST submission latency:** - Symptom: Slow operations appear when jobs are submitted via REST API, and the submitting client experiences long response times. -- Check: `grep "submitJob" $SEATUNNEL_HOME/logs/seatunnel-server.log` — look for elapsed time. +- Check: `grep "submitJob" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log` — look for elapsed time. - Common cause: Master node is overloaded with concurrent submissions, or the job configuration is very large (many connectors/transforms). - Mitigation: Rate-limit concurrent submissions, increase master node resources, or use `hazelcast.operation.generic.thread.count` tuning. @@ -208,7 +212,7 @@ free -h **Checkpoint storage latency:** - Symptom: Slow operations align with checkpoint intervals, and checkpoint duration exceeds the configured timeout. -- Check: Enable DEBUG logging for `org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator`, then `grep "pending checkpoint completed" $SEATUNNEL_HOME/logs/seatunnel-server.log | grep -oP 'cost: \d+ms'` to see checkpoint durations. +- Check: Enable DEBUG logging for `org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator`, then `grep "pending checkpoint completed" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | grep -oP 'cost: \d+ms'` to see checkpoint durations. - If using S3: Run `aws s3api head-object --bucket <bucket> --key <checkpoint-path>` to measure latency, or check CloudWatch S3 metrics (`FirstByteLatency`, `TotalRequestLatency`). - Common cause: High network latency to S3/HDFS, small files causing many round trips, or S3 throttling. - Mitigation: See [Section 6](#6-s3-checkpointstate-storage-latency). @@ -307,7 +311,7 @@ SeaTunnel outputs health monitor logs periodically (every 60 seconds by default) ```bash # Extract slow operation warnings with their durations -grep "SlowOperationDetector" $SEATUNNEL_HOME/logs/seatunnel-server.log | tail -20 +grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -20 ``` #### 4.3 Node Resource Metrics @@ -373,7 +377,7 @@ curl -w "DNS: %{time_namelookup}s, Connect: %{time_connect}s, TTFB: %{time_start **Check checkpoint write performance:** ```bash # Monitor checkpoint duration from logs (requires DEBUG logging for CheckpointCoordinator) -grep "pending checkpoint completed" $SEATUNNEL_HOME/logs/seatunnel-server.log | \ +grep "pending checkpoint completed" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | \ grep -oP 'cost: \d+ms' | sort -t: -k2 -nr | head -20 ``` diff --git a/docs/en/getting-started/submit-job-to-remote-zeta-cluster.md b/docs/en/getting-started/submit-job-to-remote-zeta-cluster.md index b62b4224361..f327f72a902 100644 --- a/docs/en/getting-started/submit-job-to-remote-zeta-cluster.md +++ b/docs/en/getting-started/submit-job-to-remote-zeta-cluster.md @@ -24,18 +24,19 @@ port-forward` or direct NodePort/LoadBalancer access, and EKS/Helm deployments. ### 2.1 `--master` flag (Zeta) -All SeaTunnel Zeta client commands accept a `--master` argument that tells the client which cluster -to connect to: +All SeaTunnel Zeta client commands accept a `--master` (or `-m`) argument that tells the client +whether the job runs locally or on a cluster. It only accepts `local` or `cluster` — it does not +take a cluster address: ```bash bin/seatunnel.sh \ --config job.conf \ - --master seatunnel://192.168.1.100:5801 + -m cluster ``` -The default Zeta cluster-internal port is **5801**. This is *different* from the REST API port -(8080). The `--master` flag is used when running the job via the SeaTunnel client binary, which -communicates with the cluster over the Hazelcast member protocol. +The client picks up the target cluster from `config/hazelcast-client.yaml` (`cluster-name` and +`cluster-members`). The default Zeta cluster-internal port is **5801**. This is *different* from +the REST API port (8080). The client communicates with the cluster over the Hazelcast protocol. ### 2.2 Using the REST API (recommended for automation) @@ -382,7 +383,6 @@ helm upgrade seatunnel . \ |---|---|---|---| | **5801** | TCP | Hazelcast cluster | Member-to-member communication; do NOT expose publicly | | **8080** | TCP | REST API | Expose only via authenticated gateway in production | -| **8090** | TCP | Web UI | Optional; only needed for the dashboard | In Kubernetes, set up a `NetworkPolicy` to restrict port 5801 to the SeaTunnel namespace only: diff --git a/docs/sidebars.js b/docs/sidebars.js index a56d3d5606d..402589ca866 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -197,6 +197,7 @@ const sidebars = { "getting-started/recipes/kafka-to-iceberg", "getting-started/recipes/postgresql-cdc-to-iceberg", "getting-started/recipes/http-to-jdbc", + "getting-started/recipes/jdbc-to-jdbc", "getting-started/recipes/file-to-starrocks", "getting-started/recipes/multi-table-cdc" ] @@ -307,6 +308,8 @@ const sidebars = { "engines/zeta/user-command", "engines/zeta/logging", "engines/zeta/log-analysis-with-ai", + "engines/zeta/stain-trace", + "engines/zeta/stain-trace-quickstart", "engines/zeta/telemetry", "engines/zeta/busyness-and-backpressure", "engines/zeta/live-metrics-chart", diff --git a/docs/zh/connectors/sink/Databend.md b/docs/zh/connectors/sink/Databend.md index fc179ffb986..f391ec23178 100644 --- a/docs/zh/connectors/sink/Databend.md +++ b/docs/zh/connectors/sink/Databend.md @@ -48,7 +48,7 @@ Databend sink 内部通过 stage attachment 实现数据的批量导入。 | url | String | 是 | - | Databend JDBC 连接 URL,必须以 `jdbc:databend://` 开头 | | username | String | 是 | - | Databend 数据库用户名 | | password | String | 是 | - | Databend 数据库密码 | -| database | String | 否 | - | 与 `table` 一起用于定位写入目标表的 Databend 数据库名称 | +| database | String | 否 | - | 与 `table` 一起用于定位写入目标表的 Databend 数据库名称。未设置时不会自动回退 | | table | String | 否 | - | Databend 表名称 | | batch_size | Integer | 否 | 1000 | 批量写入的记录数 | | auto_commit | Boolean | 否 | true | 是否自动提交事务 | diff --git a/docs/zh/connectors/sink/Hudi.md b/docs/zh/connectors/sink/Hudi.md index 811a8f51065..3109518be62 100644 --- a/docs/zh/connectors/sink/Hudi.md +++ b/docs/zh/connectors/sink/Hudi.md @@ -113,7 +113,7 @@ SeaTunnel Hudi sink 会写入 Hudi 数据文件和 `.hoodie` 元数据,但不 ### op_type [enum] -`op_type` Hudi 表的操作类型。值可以是 `insert`、`upsert` 或 `bulk_insert`。 +`op_type` Hudi 表的操作类型。值可以是 `INSERT`、`UPSERT` 或 `BULK_INSERT`。 ### batch_interval_ms [Int] diff --git a/docs/zh/connectors/sink/Jdbc.md b/docs/zh/connectors/sink/Jdbc.md index 6440fb9fcf7..6af0b46243c 100644 --- a/docs/zh/connectors/sink/Jdbc.md +++ b/docs/zh/connectors/sink/Jdbc.md @@ -298,8 +298,8 @@ Postgres 9.5及以下版本,请设置为 `postgresLow` 来支持 CDC | SqlServer | Tablestore | Teradata | | Vertica | OceanBase | XUGU | | IRIS | Inceptor | Highgo | -| DSQL | | | -| YashanDB | CockroachDB | | +| OpenGauss | Presto | Dsql | +| DuckDB | YashanDB | CockroachDB | ### database [string] diff --git a/docs/zh/connectors/sink/Snowflake.md b/docs/zh/connectors/sink/Snowflake.md index 24fd79e5697..4fe53c28957 100644 --- a/docs/zh/connectors/sink/Snowflake.md +++ b/docs/zh/connectors/sink/Snowflake.md @@ -43,14 +43,16 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | INT<br/>INTEGER | INT | | BIGINT | LONG | | DECIMAL<br/>NUMERIC<br/>NUMBER<br/> | DECIMAL(p, s) | -| DECIMAL(p, s)(p > 38 时) | DECIMAL(38, 18) | +| DECIMAL(p, s)(p > 38 时) | DECIMAL(38, s - (p - 38))(最小为 0) | | REAL<br/>FLOAT4 | FLOAT | | DOUBLE<br/>DOUBLE PRECISION<br/>FLOAT8<br/>FLOAT | DOUBLE | | CHAR<br/>CHARACTER<br/>VARCHAR<br/>STRING<br/>TEXT<br/>VARIANT<br/>OBJECT | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_LTZ<br/>TIMESTAMP_NTZ<br/>TIMESTAMP_TZ | TIMESTAMP | -| BINARY<br/>VARBINARY<br/>GEOGRAPHY<br/>GEOMETRY | BYTES | +| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_NTZ | TIMESTAMP | +| TIMESTAMP_LTZ<br/>TIMESTAMP_TZ | TIMESTAMP_TZ | +| BINARY<br/>VARBINARY | BYTES | +| GEOGRAPHY<br/>GEOMETRY | STRING | ## 选项 diff --git a/docs/zh/connectors/source/Snowflake.md b/docs/zh/connectors/source/Snowflake.md index 82bb055975c..00492f12b0e 100644 --- a/docs/zh/connectors/source/Snowflake.md +++ b/docs/zh/connectors/source/Snowflake.md @@ -45,16 +45,15 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | INT<br/>INTEGER | INT | | BIGINT | LONG | | DECIMAL<br/>NUMERIC<br/>NUMBER<br/> | DECIMAL(p, s) | -| DECIMAL(p, s)(p > 38 时) | DECIMAL(38, 18) | | REAL<br/>FLOAT4 | FLOAT | | DOUBLE<br/>DOUBLE PRECISION<br/>FLOAT8<br/>FLOAT | DOUBLE | | CHAR<br/>CHARACTER<br/>VARCHAR<br/>STRING<br/>TEXT<br/>VARIANT<br/>OBJECT | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_LTZ<br/>TIMESTAMP_NTZ<br/>TIMESTAMP_TZ | TIMESTAMP | +| DATETIME<br/>TIMESTAMP<br/>TIMESTAMP_NTZ | TIMESTAMP | +| TIMESTAMP_LTZ<br/>TIMESTAMP_TZ | TIMESTAMP_TZ | | BINARY<br/>VARBINARY | BYTES | -| GEOGRAPHY (WKB 或 EWKB)<br/>GEOMETRY (WKB 或 EWKB) | BYTES | -| GEOGRAPHY (GeoJSON, WKT 或 EWKT)<br/>GEOMETRY (GeoJSON, WKB 或 EWKB) | STRING | +| GEOGRAPHY<br/>GEOMETRY | STRING | ## 源选项 @@ -64,7 +63,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | driver | String | 是 | - | JDBC 驱动类名,Snowflake 使用 `net.snowflake.client.jdbc.SnowflakeDriver`。 | | username | String | 否 | - | Snowflake 账户用户名。 | | password | String | 否 | - | Snowflake 账户密码。 | -| query | String | 是 | - | 读取数据的 SELECT 语句。SELECT 的列列表决定输出 schema,只选择需要的列即可。 | +| query | String | 否 | - | 读取数据的 SELECT 语句。未配置 `table_path` 或 `table_list` 时必填。SELECT 的列列表决定输出 schema,只选择需要的列即可。 | | connection_check_timeout_sec | Int | 否 | 30 | 连接校验超时时间(秒),超过该时间未完成则失败。 | | partition_column | String | 否 | - | 用于并行拆分读取的列。支持数值列和字符串列(配合 `split.string_split_mode` 使用);只能配置一列。 | | partition_lower_bound | String | 否 | - | `partition_column` 的下界,用于范围拆分;不设置时 SeaTunnel 查询最小值。 | @@ -151,7 +150,7 @@ source { - Snowflake 任务使用 `Jdbc` 插件名,并设置 `driver = "net.snowflake.client.jdbc.SnowflakeDriver"`。 - 运行任务前请把 Snowflake JDBC 驱动 jar 放到 `$SEATUNNEL_HOME/plugins/jdbc/lib/`。 - 并行读取时,`partition_column`、`partition_lower_bound`、`partition_upper_bound` 和 `partition_num` 必须描述同一数值列的范围。 -- Snowflake 地理空间列的字节/字符串返回形式由 Snowflake JDBC URL 参数(如 `GEOGRAPHY_OUTPUT_FORMAT`)决定。 +- Snowflake 地理空间列映射为 SeaTunnel `STRING` 类型。 ## 变更日志 diff --git a/docs/zh/engines/zeta/checkpoint-storage.md b/docs/zh/engines/zeta/checkpoint-storage.md index b1f58ade27b..6bee47ac96d 100644 --- a/docs/zh/engines/zeta/checkpoint-storage.md +++ b/docs/zh/engines/zeta/checkpoint-storage.md @@ -309,7 +309,7 @@ seatunnel: namespace: # 检查点存储父路径,默认值为/seatunnel/checkpoint/ storage.type: hdfs fs.defaultFS: hdfs://localhost:9000 - // 如果您使用kerberos,您可以这样配置: + # 如果您使用kerberos,您可以这样配置: kerberosPrincipal: your-kerberos-principal kerberosKeytabFilePath: your-kerberos-keytab ``` diff --git a/docs/zh/engines/zeta/hybrid-cluster-deployment.md b/docs/zh/engines/zeta/hybrid-cluster-deployment.md index 8cac9840a85..b59101405fa 100644 --- a/docs/zh/engines/zeta/hybrid-cluster-deployment.md +++ b/docs/zh/engines/zeta/hybrid-cluster-deployment.md @@ -444,7 +444,7 @@ mkdir -p $SEATUNNEL_HOME/logs ./bin/seatunnel-cluster.sh -d ``` -日志将写入 `$SEATUNNEL_HOME/logs/seatunnel-engine-server.log` +日志将写入 `$SEATUNNEL_HOME/logs/` 目录——默认的 master-and-worker 部署写入 `seatunnel-engine-server.log` ## 8. 提交作业和管理作业 diff --git a/docs/zh/engines/zeta/log-analysis-with-ai.md b/docs/zh/engines/zeta/log-analysis-with-ai.md index 68df847aa83..1e3f548a80c 100644 --- a/docs/zh/engines/zeta/log-analysis-with-ai.md +++ b/docs/zh/engines/zeta/log-analysis-with-ai.md @@ -46,7 +46,8 @@ SeaTunnel 默认将进程日志写入 `$SEATUNNEL_HOME/logs`。集群启动脚 ```shell JOB_ID=<job-id> -grep -F "[${JOB_ID}]" "$SEATUNNEL_HOME/logs/seatunnel-engine-server.log" > job.log +# seatunnel-engine-server.log(hybrid);seatunnel-engine-master.log / -worker.log(分离部署) +grep -hF "[${JOB_ID}]" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log > job.log ``` 如果启用了按作业路由,请检查 `job-<job-id>.log`。在多节点 Zeta 集群中,需要同时收集 Master 和执行该作业的 Worker 日志。当前 Master 还提供以下 REST 接口: @@ -54,7 +55,7 @@ grep -F "[${JOB_ID}]" "$SEATUNNEL_HOME/logs/seatunnel-engine-server.log" > job.l ```text GET http://<master-host>:8080/logs/<job-id> GET http://<master-host>:8080/logs?format=json -GET http://<node-host>:5801/log +GET http://<node-host>:8080/log ``` 第一个接口从所有 Zeta 节点获取匹配的日志,最后一个接口读取单个节点的日志。如果配置了 context path 或动态 HTTP 端口,接口地址也会变化。完整行为请参阅 [RESTful API V2](rest-api-v2.md#获取所有节点日志内容)。 diff --git a/docs/zh/engines/zeta/logging.md b/docs/zh/engines/zeta/logging.md index d1cfcdce225..fa4b6273b96 100644 --- a/docs/zh/engines/zeta/logging.md +++ b/docs/zh/engines/zeta/logging.md @@ -116,7 +116,7 @@ SeaTunnel 提供了一个 API,用于查询日志。 ### 定时删除旧日志 -SeaTunnel 支持定时删除旧日志文件,以避免磁盘空间不足。您可以在 `seatunnel.yml` 文件中添加以下配置: +SeaTunnel 支持定时删除旧日志文件,以避免磁盘空间不足。您可以在 `seatunnel.yaml` 文件中添加以下配置: ```yaml seatunnel: diff --git a/docs/zh/engines/zeta/resource-isolation.md b/docs/zh/engines/zeta/resource-isolation.md index b009fe605aa..91bddc6ba0c 100644 --- a/docs/zh/engines/zeta/resource-isolation.md +++ b/docs/zh/engines/zeta/resource-isolation.md @@ -47,7 +47,7 @@ SeaTunnel支持对每个实例添加`tag`, 然后在提交任务时可以在配 2. 在任务的配置中添加`tag_filter`来选择你需要运行该任务的节点 -```hacon +```hocon env { parallelism = 1 job.mode = "BATCH" diff --git a/docs/zh/engines/zeta/rest-api-job-lifecycle.md b/docs/zh/engines/zeta/rest-api-job-lifecycle.md index ca8f4e45aac..4a9ae38afed 100644 --- a/docs/zh/engines/zeta/rest-api-job-lifecycle.md +++ b/docs/zh/engines/zeta/rest-api-job-lifecycle.md @@ -318,7 +318,7 @@ curl -X POST "http://<master>:8080/submit-job?restoreMode=SAVEPOINT&restoreSourc }' ``` -### 6.3 从指定 Savepoint 路径恢复 +### 6.3 从显式 Savepoint 路径恢复 REST API 不支持指定具体的 Savepoint 路径进行恢复。不存在 `restore.mode` 或 `savepoint.path` 之类的 env 选项;恢复行为只由请求参数 `restoreMode` 和 `restoreSourceJobId` 控制,并且 diff --git a/docs/zh/engines/zeta/state-storage-and-recovery.md b/docs/zh/engines/zeta/state-storage-and-recovery.md index 98af1592d88..3a1b079a66d 100644 --- a/docs/zh/engines/zeta/state-storage-and-recovery.md +++ b/docs/zh/engines/zeta/state-storage-and-recovery.md @@ -12,10 +12,8 @@ SeaTunnel Engine(Zeta)在作业执行过程中会持久化多类状态数据 | 存储类别 | 用途 | 默认位置 | |---|---|---| | Checkpoint(检查点)| 流水线算子状态的容错快照 | `seatunnel.yaml` `checkpoint.storage` 配置路径 | -| Savepoint(保存点)| 用户手动触发的命名检查点,用于计划性停止或重启 | 与 Checkpoint 同路径,目录名不同 | -| IMap / MapStore | 分布式内存状态(作业元数据、运行中作业数据、指标)| `hazelcast.yaml` MapStore base-dir | -| WAL(预写日志)| IMap 持久化的耐久性日志 | 与 MapStore base-dir 相同 | -| 历史作业元数据 | 已完成 / 失败作业记录 | MapStore base-dir | +| Savepoint(保存点)| 用户手动触发的命名检查点,用于计划性停止或重启 | 与 Checkpoint 同一存储,位于所属作业自身的目录下 | +| IMap / MapStore | 分布式内存状态(作业元数据、作业状态、历史记录)| 默认仅存于内存;可通过 `hazelcast.yaml` 中的 Hazelcast MapStore 配置持久化 | --- @@ -48,10 +46,8 @@ seatunnel: checkpoint: interval: 10000 # 两次 Checkpoint 间隔(毫秒) timeout: 60000 # Checkpoint 完成超时(毫秒) - max-concurrent: 1 # 最大并发 Checkpoint 数 - tolerable-failure: 2 # 允许连续失败次数 storage: - type: hdfs # hdfs | localfile(已废弃) + type: hdfs # hdfs(通过 HDFS API 同时支持 S3、本地文件)| localfile(已废弃) plugin-config: namespace: /seatunnel/checkpoint/ # 必须以 / 结尾 # S3 示例: @@ -79,7 +75,7 @@ seatunnel: | 维度 | Checkpoint | Savepoint | |---|---|---| -| 触发方式 | 周期性 / 自动 | 手动(`seatunnel.sh -r <jobId> --savepoint`)| +| 触发方式 | 周期性 / 自动 | 手动(`$SEATUNNEL_HOME/bin/seatunnel.sh --savepoint <jobId>`)| | 用途 | 容错恢复 | 计划性停止、升级、迁移 | | 生命周期 | 引擎管理 | 操作人员管理 | | 保留策略 | 自动轮转 | 手动删除前永久保留 | @@ -88,7 +84,7 @@ seatunnel: ```bash # 停止运行中的作业并创建 Savepoint -$SEATUNNEL_HOME/bin/seatunnel.sh --stop-job <job-id> --savepoint +$SEATUNNEL_HOME/bin/seatunnel.sh --savepoint <job-id> # 或通过 REST API v2 curl -X POST http://<master>:8080/stop-job \ @@ -117,14 +113,8 @@ $SEATUNNEL_HOME/bin/seatunnel.sh --config job.conf --restore-with-checkpoint <jo ### Savepoint 路径结构 -``` -<namespace>/ - savepoint/ - <job-id>/ - <savepoint-timestamp>/ - <pipeline-id>/ - <task-location>/state-data -``` +Savepoint 是保存点类型的 Checkpoint:它写入上文配置的同一个 Checkpoint 存储中,位于所属作业的目录下 +(`<namespace>/<job-id>/`)。不存在独立的 savepoint 根目录。 ### 安全清理 @@ -136,43 +126,41 @@ $SEATUNNEL_HOME/bin/seatunnel.sh --config job.conf --restore-with-checkpoint <jo ### IMap 存储的内容 -SeaTunnel Engine 使用 Hazelcast IMap 作为分布式内存键值存储。以下逻辑映射会被持久化: +SeaTunnel Engine 使用 Hazelcast IMap 作为分布式内存键值存储。引擎使用的主要逻辑映射包括: | IMap 名称 | 内容 | |---|---| -| `running-job-state` | 每个运行中作业的状态机当前状态 | -| `running-job-metrics` | 实时吞吐量、延迟和记录数指标 | -| `running-pipeline-state` | 每条逻辑流水线的 Pipeline 级状态 | -| `finished-job-state` | 已完成、已取消或失败作业的终态 | -| `finished-job-metrics` | 作业终止后的最终指标快照 | +| `engine_runningJobInfo` | 已提交作业的信息与初始化元数据(作业 ID、作业名称等;指标在作业完成后存于 `engine_finishedJobMetrics`)| +| `engine_runningJobState` | 运行中作业和流水线的状态机当前状态 | +| `engine_stateTimestamps` | 作业 / 流水线状态流转的时间戳 | +| `engine_finishedJobState` | 已完成、已取消或失败作业的终态 | +| `engine_finishedJobMetrics` | 作业终止后的最终指标快照 | +| `engine_finishedJobVertexInfo` | 已完成作业的执行节点信息 | ### MapStore(磁盘持久化) -Hazelcast MapStore 将 IMap 条目写入本地磁盘,使其在进程重启后可以恢复。这与 Checkpoint 存储**相互独立**。 +默认情况下 IMap 数据仅存储在内存中(并按备份份数在节点间复制)。如果所有节点停止,数据将丢失, +除非启用了 MapStore 持久化。启用后,Hazelcast MapStore 会将 IMap 条目写入外部文件系统(HDFS、S3, +或通过 HDFS API 访问的本地文件),使数据在整个集群重启后仍然可用。这与 Checkpoint 存储**相互独立**。 -默认存储路径在 `hazelcast.yaml` 中配置: +持久化在 `hazelcast.yaml` 中配置(详见[Zeta 集群分离式部署](separated-cluster-deployment.md)): ```yaml map: - seatunnel: + engine*: map-store: enabled: true - initial-load-mode: EAGER + initial-mode: EAGER + factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory properties: - hazelcast.fs.base-dir: /tmp/seatunnel/imap # 绝对路径 - hazelcast.fs.write-behind-delay-seconds: 1 + type: hdfs + namespace: /tmp/seatunnel/imap # 存储 namespace(未设置时默认 /seatunnel-imap) + clusterName: seatunnel-cluster + storage.type: hdfs + fs.defaultFS: hdfs://localhost:9000 ``` -MapStore 目录结构: - -``` -<hazelcast.fs.base-dir>/ - maps/ - running-job-state/ - running-job-metrics/ - finished-job-state/ - finished-job-metrics/ -``` +在分离式集群模式下,只有 Master 节点存储 IMap 数据,因此该配置仅在 Master 节点生效。 ### IMap、MapStore 与 Checkpoint 的关系 @@ -182,38 +170,31 @@ IMap / MapStore <──────────────────── ``` 二者**完全独立**。删除 Checkpoint 存储不影响 IMap;反之亦然。 -作业可以从 Checkpoint 恢复,即使 IMap MapStore 已被清除——但作业 ID 和流水线映射需要**重新提交**, -因为 `running-job-state` 已丢失。 +作业可以从 Checkpoint 恢复,即使 IMap MapStore 数据已被清除——但作业 ID 和流水线映射需要**重新提交**, +因为运行作业的状态已丢失。 --- -## 4. WAL(预写日志) - -Hazelcast 使用 WAL 保证 MapStore 的耐久性。WAL 文件累积在: - -``` -<hazelcast.fs.base-dir>/ - wal/ - <imap-name>-<partition>.wal -``` - -### 长期运行 CDC 作业的 WAL 增长 - -每次 CDC binlog 事件更新 `running-job-metrics` 或 `running-pipeline-state` 时,都会产生一次 WAL 写入。 -随着时间推移(数天或数周),如果出现以下情况,WAL 文件可能增长至数 GB: +## 4. MapStore 文件维护 -- `write-behind-delay-seconds` 设置过低(刷新频率过高) -- 作业每秒处理数百万事件 +IMap 持久化是一种追加写(append-only)的预写日志:`FileMapStore` 委托 `IMapFileStorage`,通过 WAL +写入器把每次状态变更追加到配置的 `namespace` 目录下的 region 文件中。所有作业的记录都追加到同一批 +region 文件里,因此只要作业持续运行,这些文件就会不断增长。 **缓解措施:** -```yaml -hazelcast.fs.write-behind-delay-seconds: 5 # 增大刷新间隔 -hazelcast.fs.compaction-threshold: 1000 # N 条写入后触发压缩 +- 让已完成作业按期过期:`history-job-expire-minutes` 会将已完成作业的记录从 IMap 中移除, + 使其不再被持久化。 +- 在 Master 节点上监控 namespace 目录: + +```bash +du -sh /tmp/seatunnel/imap/ # 替换为你在 hazelcast.yaml 中配置的 namespace(本地文件存储,即 fs.defaultFS 以 file:/// 开头时) +hadoop fs -du -s -h <fs-uri>/<namespace> # HDFS 或对象存储后端使用该命令查看 ``` -**已完成作业**的 WAL 文件在对应 IMap 条目已刷入 MapStore 文件后可以安全压缩或删除。 -**不要**删除运行中作业的 WAL 文件。 +**不要**按作业选择性地删除 MapStore 文件:并不存在按作业划分的文件,手工删除文件可能损坏下次 +Master 启动时的 IMap 重放。正确做法是为 namespace 卷规划足够的容量;只有当集群停止、且可以接受 +丢失所有运行中作业的 IMap 状态时,才整体清空 namespace。 --- @@ -234,10 +215,9 @@ seatunnel: | 集群重启后作业仍保持已移除(启用 IMap 持久化时) | 是(过期时间随条目一起持久化,已过期的作业不会被重新加载,尚未过期的作业保留剩余时间) | | 删除该作业的 MapStore 持久化文件 | **否**(过期记录仍保留在 IMap 存储文件中) | | 删除 Checkpoint 存储目录 | **否** | -| 删除 Savepoint 目录 | **否** | -| 删除 WAL 文件 | **否**(仅通过压缩间接处理)| +| 删除 Savepoint 数据 | **否** | -**核心结论**:`history-job-expire-minutes` 仅清理 `finished-job-state` 中的作业元数据。 +**核心结论**:`history-job-expire-minutes` 仅清理已完成作业 IMap 中的作业元数据。 HDFS / S3 / 本地的 Checkpoint 和 Savepoint 目录**不受此配置影响**,必须独立管理。 --- @@ -254,7 +234,7 @@ HDFS / S3 / 本地的 Checkpoint 和 Savepoint 目录**不受此配置影响** | PostgreSQL CDC(逻辑复制)| 每张表 1–50 KB | | Oracle CDC(LogMiner)| 50 KB – 2 MB | -公式:`checkpoint_size ≈ 表数量 × 平均 Split 数 × 每 Split 状态大小 × 最大并发 Checkpoint 数` +公式:`checkpoint_size ≈ 表数量 × 平均 Split 数 × 每 Split 状态大小`(同一时刻只有一个 Checkpoint 在途,因此无需乘并发系数) ### 存储容量经验值 @@ -264,25 +244,19 @@ HDFS / S3 / 本地的 Checkpoint 和 Savepoint 目录**不受此配置影响** 所需存储 = checkpoint_size × 3 × 安全系数(1.5) ``` -### IMap / WAL 容量估算 +### IMap 存储容量估算 每个运行中的 CDC 作业大约占用: -- `running-job-state`:每条流水线约 50–200 字节 -- `running-job-metrics`:每次指标刷新每条流水线约 1–2 KB - -对于运行 100 个 CDC 作业的集群: - -``` -imap 内存 ≈ 100 × 200B ≈ 20 KB(状态) -WAL 磁盘:建议每节点规划 2–5 GB -``` +- 作业状态:每条流水线约 50–200 字节 +- 指标:每次指标刷新每条流水线约 1–2 KB -使用以下命令持续监控: +对于运行 100 个 CDC 作业的集群,IMap 数据量在几 MB 量级。MapStore namespace 的磁盘占用随状态 +流转历史增长,长期运行的集群建议规划数 GB 空间,并使用以下命令持续监控: ```bash -du -sh $SEATUNNEL_HOME/imap/wal/ -watch -n 60 'du -sh /tmp/seatunnel/imap/' +du -sh /tmp/seatunnel/imap/ # 替换为你配置的 MapStore namespace(本地文件存储场景) +hadoop fs -du -s -h <fs-uri>/<namespace> # HDFS 或对象存储后端使用该命令查看 ``` --- @@ -319,32 +293,26 @@ ls -lh /tmp/seatunnel/checkpoint/ --- -### IMap / WAL 目录增长 +### MapStore namespace 目录增长 -**现象**:`/tmp/seatunnel/imap/` 占满 Master / Worker 节点磁盘。 +**现象**:配置的 MapStore `namespace` 目录占满 Master 节点磁盘。 **诊断**: ```bash -du -sh /tmp/seatunnel/imap/wal/ -du -sh /tmp/seatunnel/imap/maps/ +du -sh /tmp/seatunnel/imap/ # 替换为你在 hazelcast.yaml 中配置的 namespace(本地文件存储,即 fs.defaultFS 以 file:/// 开头时) +hadoop fs -du -s -h <fs-uri>/<namespace> # HDFS 或对象存储后端使用该命令查看 ``` **可能根因**: -- `running-job-metrics` 在每次 Checkpoint 时为每个运行中作业更新 -- `write-behind-delay-seconds` 过低(默认 1 秒) -- WAL 压缩触发频率不足 +- 已完成作业未过期(`history-job-expire-minutes` 未设置或过大) +- 运行中作业过多,状态和指标持续更新 **修复**: -```yaml -# hazelcast.yaml -hazelcast.fs.write-behind-delay-seconds: 10 -hazelcast.fs.compaction-threshold: 500 -``` - -重启引擎后,已完成作业的旧 WAL 文件将在启动时被压缩。 +在 `seatunnel.yaml` 中启用 / 调整 `history-job-expire-minutes`。MapStore 的 region 文件无法按作业 +裁剪;要回收其空间需要停止集群并整体清空 namespace,这会丢弃所有作业的 IMap 状态。 --- @@ -367,12 +335,11 @@ hazelcast.fs.compaction-threshold: 500 删除任何状态目录前,请确认: -- [ ] 作业已处于 `FINISHED`、`CANCELLED` 或 `FAILED` 终态 +- [ ] 作业已处于 `FINISHED`、`CANCELED` 或 `FAILED` 终态 - [ ] 确认不会从该 Checkpoint 或 Savepoint 恢复 - [ ] 确认该作业 ID 未被任何监控或告警规则引用 - [ ] Checkpoint 存储:递归删除 `<namespace>/<job-id>/` -- [ ] Savepoint:递归删除 `<namespace>/savepoint/<job-id>/` -- [ ] MapStore / WAL:手动删除后重启引擎以触发重载 +- [ ] MapStore 数据:仅支持整体清空 namespace(先停止集群),不支持按作业选择性删除 --- diff --git a/docs/zh/engines/zeta/tuning-guide.md b/docs/zh/engines/zeta/tuning-guide.md index 5e2eb104a72..d4f4287d647 100644 --- a/docs/zh/engines/zeta/tuning-guide.md +++ b/docs/zh/engines/zeta/tuning-guide.md @@ -20,11 +20,14 @@ SeaTunnel Engine 是基于 [JVM](https://zh.wikipedia.org/wiki/Java%E8%99%9A%E6% ##### 排查流程 1. 检查 JVM 堆内存实时占用 - 使用 `jcmd` 命令查看 JVM 堆内存使用情况, 其中 `<pid>` 是 SeaTunnel Engine 进程的 PID。 + 使用 `jcmd` 命令查看 JVM 堆内存使用情况, 其中 `<pid>` 是 SeaTunnel Engine 进程的 PID。JDK 9+ 使用 `jcmd <pid> GC.heap_info`;JDK 8 使用 `jmap -heap <pid>`(该参数在 JDK 9+ 已移除)。 ```bash + # JDK 9+ + jcmd <pid> GC.heap_info + # JDK 8 jmap -heap <pid> ``` - 输出结果示例: + 输出结果示例(JDK 8 主机上 `jmap -heap` 风格的输出): ```shell Attaching to process ID 2111950, please wait... Debugger attached successfully. @@ -169,7 +172,8 @@ Hazelcast 的 `SlowOperationDetector` 监控分区线程上的操作执行时间 ```bash # 检查慢操作日志的频率和时间 -grep "SlowOperationDetector" $SEATUNNEL_HOME/logs/seatunnel-server.log | tail -50 +grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -50 +# 默认的 master-and-worker 部署写入 seatunnel-engine-server.log;master/worker 分离部署分别写入 -master.log / -worker.log ``` 将时间戳与以下事件关联: @@ -190,7 +194,7 @@ free -h **REST 提交延迟:** - 症状:通过 REST API 提交作业时出现慢操作,且提交客户端响应时间较长。 -- 检查:`grep "submitJob" $SEATUNNEL_HOME/logs/seatunnel-server.log` —— 关注耗时。 +- 检查:`grep "submitJob" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log` —— 关注耗时。 - 常见原因:Master 节点并发提交过载,或作业配置非常庞大(连接器/Transform 数量多)。 - 缓解:限制并发提交速率、增加 Master 节点资源、或调整 `hazelcast.operation.generic.thread.count`。 @@ -208,7 +212,7 @@ free -h **Checkpoint 存储延迟:** - 症状:慢操作与 Checkpoint 间隔对齐,且 Checkpoint 耗时超过配置的超时。 -- 检查:为 `org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator` 启用 DEBUG 日志,然后执行 `grep "pending checkpoint completed" $SEATUNNEL_HOME/logs/seatunnel-server.log | grep -oP 'cost: \d+ms'` 查看 Checkpoint 耗时。 +- 检查:为 `org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator` 启用 DEBUG 日志,然后执行 `grep "pending checkpoint completed" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | grep -oP 'cost: \d+ms'` 查看 Checkpoint 耗时。 - 如果使用 S3:运行 `aws s3api head-object --bucket <bucket> --key <checkpoint-path>` 测量延迟,或查看 CloudWatch S3 指标(`FirstByteLatency`、`TotalRequestLatency`)。 - 常见原因:到 S3/HDFS 的网络延迟高、小文件导致多次往返、或 S3 限流。 - 缓解:参见[第 6 节](#6-s3-checkpoint状态存储延迟)。 @@ -307,7 +311,7 @@ SeaTunnel 定期输出健康监控日志(默认每 60 秒一次)。这些日 ```bash # 提取慢操作告警及其耗时 -grep "SlowOperationDetector" $SEATUNNEL_HOME/logs/seatunnel-server.log | tail -20 +grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -20 ``` #### 4.3 节点资源指标 diff --git a/docs/zh/getting-started/submit-job-to-remote-zeta-cluster.md b/docs/zh/getting-started/submit-job-to-remote-zeta-cluster.md index 659c0191dbd..5418cd29bcf 100644 --- a/docs/zh/getting-started/submit-job-to-remote-zeta-cluster.md +++ b/docs/zh/getting-started/submit-job-to-remote-zeta-cluster.md @@ -24,16 +24,17 @@ Docker 单节点、多节点集群、Kubernetes(通过 `kubectl port-forward` ### 2.1 `--master` 参数(Zeta) -所有 SeaTunnel Zeta 客户端命令均支持 `--master` 参数,用于指定集群连接地址: +所有 SeaTunnel Zeta 客户端命令均支持 `--master`(或 `-m`)参数,用于指定作业在本地运行还是提交到集群。 +该参数只接受 `local` 或 `cluster`,不接受集群地址: ```bash bin/seatunnel.sh \ --config job.conf \ - --master seatunnel://192.168.1.100:5801 + -m cluster ``` -Zeta 集群内部默认端口为 **5801**,与 REST API 端口(8080)不同。`--master` 参数用于 -通过 Hazelcast 成员协议直接连接集群。 +客户端从 `config/hazelcast-client.yaml`(`cluster-name` 与 `cluster-members`)获取目标集群地址。 +Zeta 集群内部默认端口为 **5801**,与 REST API 端口(8080)不同。客户端通过 Hazelcast 协议与集群通信。 ### 2.2 使用 REST API(推荐用于自动化场景) @@ -372,7 +373,6 @@ helm upgrade seatunnel . \ |---|---|---|---| | **5801** | TCP | Hazelcast 集群 | 成员间通信;生产环境不要对外暴露 | | **8080** | TCP | REST API | 生产环境建议通过认证网关暴露 | -| **8090** | TCP | Web UI | 可选,仅用于管理看板 | 在 Kubernetes 中,建议设置 NetworkPolicy 将 5801 端口限制在 SeaTunnel 命名空间内:
