davidzollo opened a new issue, #10979:
URL: https://github.com/apache/seatunnel/issues/10979
We need community help to improve several P0-level Apache SeaTunnel
documents.
Based on recent documentation Q&A feedback, users are repeatedly asking
about production operations rather than basic usage. The most frequent gaps are
around long-running CDC jobs, Zeta state storage, checkpoint/savepoint
behavior, REST API job lifecycle, Docker/Kubernetes/EKS job submission, and the
capability boundary of multi-table transform/join.
This issue tracks all P0 documentation improvements in one place. No
sub-issues are used. Please claim work directly in this issue before starting
implementation.
## Documentation items open for claim
| Priority | Area | Document Item | Suggested Location | Contributor |
Status | PR |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| P0 | Zeta Engine | Zeta State Storage / IMap / WAL / Checkpoint /
Savepoint Guide | `docs/en/engines/zeta/checkpoint-storage.md`,
`docs/zh/engines/zeta/checkpoint-storage.md`, or new
`state-storage-and-recovery.md` | | Todo | |
| P0 | CDC | CDC Production Cookbook: Full + Incremental Synchronization |
New `docs/en/connectors/cdc-production-cookbook.md`,
`docs/zh/connectors/cdc-production-cookbook.md`; link from CDC connector pages
| | Todo | |
| P0 | REST API | REST API Job Lifecycle Cookbook | Extend
`docs/en/engines/zeta/rest-api-v2.md`, `docs/zh/engines/zeta/rest-api-v2.md`,
or new `rest-api-job-lifecycle.md` | | Todo | |
| P0 | Deployment | Docker / Kubernetes / EKS Job Submission Guide | Extend
Docker/Kubernetes/Helm docs or add `submit-job-to-remote-zeta-cluster.md` | |
Todo | |
| P0 | Transform | Multi-table Transform / Join / EtLT Capability Boundary |
Extend transform docs or add `multi-table-transform-and-join-boundary.md` | |
Todo | |
## Task list
- [ ] Claim a documentation item before starting implementation.
- [ ] Update the `Contributor` and `Status` columns after a documentation
item is claimed.
- [ ] Add the PR link after work starts.
- [ ] Update both `docs/en/...` and `docs/zh/...` where possible.
- [ ] Update `docs/sidebars.js` if new pages are added.
- [ ] Add cross-links from existing related documents to the new or updated
guides.
## P0 documentation checklist
- [ ] **P0-1. Zeta State Storage / IMap / WAL / Checkpoint / Savepoint
Guide**
- [ ] Extend `docs/en/engines/zeta/checkpoint-storage.md`.
- [ ] Extend `docs/zh/engines/zeta/checkpoint-storage.md`.
- [ ] Add `docs/en/engines/zeta/state-storage-and-recovery.md` if the
content becomes too large.
- [ ] Add `docs/zh/engines/zeta/state-storage-and-recovery.md` if the
content becomes too large.
- [ ] Update `docs/sidebars.js` if a new page is added.
- [ ] Explain checkpoint, savepoint, IMap, MapStore, WAL, running job
state, and running job metrics.
- [ ] Explain the relationship between Hazelcast distributed state and
SeaTunnel Engine state persistence.
- [ ] Explain common storage paths and directory meanings.
- [ ] Explain what `history-job-expire-minutes` cleans and what it does
not clean.
- [ ] Provide safe cleanup rules.
- [ ] Provide capacity planning guidance for long-running CDC jobs.
- [ ] Provide troubleshooting examples for large checkpoint/IMap/WAL
directories.
- [ ] **P0-2. CDC Production Cookbook: Full + Incremental Synchronization**
- [ ] Add `docs/en/connectors/cdc-production-cookbook.md`.
- [ ] Add `docs/zh/connectors/cdc-production-cookbook.md`.
- [ ] Update `docs/sidebars.js` if a new page is added.
- [ ] Add cross-links from `docs/en/connectors/source/MySQL-CDC.md` and
`docs/zh/connectors/source/MySQL-CDC.md`.
- [ ] Add cross-links from `docs/en/connectors/source/PostgreSQL-CDC.md`
and `docs/zh/connectors/source/PostgreSQL-CDC.md`.
- [ ] Add cross-links from `docs/en/connectors/source/Oracle-CDC.md` and
`docs/zh/connectors/source/Oracle-CDC.md`.
- [ ] Explain the full + incremental synchronization lifecycle.
- [ ] Explain `startup.mode`, `server-id`, GTID, snapshot split,
binlog/WAL/logminer/logical replication prerequisites.
- [ ] Provide a production example for MySQL CDC -> Doris.
- [ ] Provide a production example for MySQL CDC -> StarRocks.
- [ ] Provide a production example for MySQL CDC -> Kafka.
- [ ] Provide a production example for PostgreSQL CDC ->
JDBC/Doris/StarRocks.
- [ ] Provide a production example for Oracle CDC -> Doris/StarRocks/JDBC.
- [ ] Explain checkpoint and 2PC interaction.
- [ ] Explain schema evolution and DDL support boundaries.
- [ ] Provide troubleshooting checklist for permission, network,
replication, offset, and checkpoint failures.
- [ ] **P0-3. REST API Job Lifecycle Cookbook**
- [ ] Extend `docs/en/engines/zeta/rest-api-v2.md`.
- [ ] Extend `docs/zh/engines/zeta/rest-api-v2.md`.
- [ ] Add `docs/en/engines/zeta/rest-api-job-lifecycle.md` if needed.
- [ ] Add `docs/zh/engines/zeta/rest-api-job-lifecycle.md` if needed.
- [ ] Update `docs/sidebars.js` if a new page is added.
- [ ] Provide a job submission example.
- [ ] Provide a job status query example.
- [ ] Provide a logs query example.
- [ ] Explain stop/cancel/savepoint semantics.
- [ ] Provide job recovery/restart examples.
- [ ] Add authentication and authorization notes.
- [ ] Add REST API performance considerations.
- [ ] Provide a multi-transform JSON submission example.
- [ ] Add common errors and troubleshooting.
- [ ] **P0-4. Docker / Kubernetes / EKS Job Submission Guide**
- [ ] Extend `docs/en/getting-started/docker/docker.md`.
- [ ] Extend `docs/zh/getting-started/docker/docker.md`.
- [ ] Extend `docs/en/getting-started/kubernetes/kubernetes.mdx`.
- [ ] Extend `docs/zh/getting-started/kubernetes/kubernetes.mdx`.
- [ ] Extend `docs/en/getting-started/kubernetes/helm.md`.
- [ ] Extend `docs/zh/getting-started/kubernetes/helm.md`.
- [ ] Add `docs/en/getting-started/submit-job-to-remote-zeta-cluster.md`
if needed.
- [ ] Add `docs/zh/getting-started/submit-job-to-remote-zeta-cluster.md`
if needed.
- [ ] Update `docs/sidebars.js` if a new page is added.
- [ ] Explain how to submit jobs from inside a Docker container.
- [ ] Explain how to submit jobs from a local machine to a remote Zeta
cluster.
- [ ] Explain how to submit jobs to Kubernetes/EKS deployment.
- [ ] Provide `--master`, `--deploy-mode`, REST API, and port-forward
examples.
- [ ] Add Helm deployment notes.
- [ ] Add ConfigMap/job config mounting examples.
- [ ] Explain network and port requirements.
- [ ] Add common troubleshooting for connection failures.
- [ ] **P0-5. Multi-table Transform / Join / EtLT Capability Boundary**
- [ ] Extend `docs/en/transforms/transform-multi-table.md`.
- [ ] Extend `docs/zh/transforms/transform-multi-table.md`.
- [ ] Extend `docs/en/transforms/table-merge.md`.
- [ ] Extend `docs/zh/transforms/table-merge.md`.
- [ ] Extend `docs/en/transforms/sql.md`.
- [ ] Extend `docs/zh/transforms/sql.md`.
- [ ] Add `docs/en/transforms/multi-table-transform-and-join-boundary.md`
if needed.
- [ ] Add `docs/zh/transforms/multi-table-transform-and-join-boundary.md`
if needed.
- [ ] Update `docs/sidebars.js` if a new page is added.
- [ ] Explain what multi-table transform supports.
- [ ] Explain what TableMerge is and what it is not.
- [ ] Explain whether SQL Transform supports cross-source/cross-table join.
- [ ] Provide examples for per-table transform rules.
- [ ] Provide examples for adding common fields to multiple tables.
- [ ] Add an explicit capability boundary table: Supported / Not Supported
/ Recommended Alternative.
- [ ] Provide recommended EtLT patterns: CDC to raw/ODS first, then
join/aggregation/modeling in Doris/StarRocks/PostgreSQL/warehouse, orchestrated
by scheduler/workflow.
## Problem details
### P0-1. Zeta State Storage / IMap / WAL / Checkpoint / Savepoint Guide
Users often cannot distinguish between checkpoint storage, savepoint,
IMap/MapStore, WAL, running job state, and job metrics storage. This causes
confusion in production when long-running jobs generate large storage
directories in HDFS/S3/OSS/MinIO/local paths.
Typical questions include:
- What is stored in checkpoint storage?
- What is IMap/MapStore used for in Zeta?
- Why are WAL or state-related directories growing?
- What is the difference between checkpoint and savepoint?
- Which directories can be safely cleaned and which must not be deleted?
- Does `history-job-expire-minutes` clean checkpoint/state files or only
expired job metadata?
- How should long-running CDC jobs plan checkpoint/state storage capacity?
Suggested sidebar location:
```text
Engines
-> SeaTunnel Engine (Zeta)
-> checkpoint-storage
-> state-storage-and-recovery
```
### P0-2. CDC Production Cookbook: Full + Incremental Synchronization
CDC users frequently ask how to configure production-grade full +
incremental synchronization, especially for MySQL/PostgreSQL/Oracle CDC to
Doris, StarRocks, Kafka, JDBC targets, etc. The current connector pages provide
parameters, but users still need an end-to-end production cookbook.
Typical questions include:
- What does `startup.mode = initial` really mean?
- Does CDC support batch mode?
- Why is `server-id` required for MySQL CDC?
- Is GTID required?
- How does `checkpoint.interval` affect CDC latency and 2PC commit frequency?
- How should Doris/StarRocks 2PC be configured?
- How should schema evolution and DDL changes be handled?
- How to observe CDC delay and troubleshoot replication problems?
### P0-3. REST API Job Lifecycle Cookbook
The REST API documentation currently works mainly as an API reference. Users
need a practical job lifecycle guide that explains how to submit, query, stop,
cancel, recover, and inspect jobs through REST API.
Typical questions include:
- How to submit a job through REST API?
- How to stop a job and keep checkpoint/savepoint state?
- What is the difference between cancel, stop, and stop-with-savepoint?
- How to query job status and logs?
- Why does `job-info` become slow when many jobs exist?
- How to use REST API with authentication enabled?
- How to submit jobs with multiple transforms in JSON format?
Suggested sidebar location:
```text
Engines
-> SeaTunnel Engine (Zeta)
-> REST API
-> rest-api-v1
-> rest-api-v2
-> rest-api-job-lifecycle
-> security
```
### P0-4. Docker / Kubernetes / EKS Job Submission Guide
Users can find deployment documents, but many still do not know how to
submit and operate jobs after deployment, especially in Docker, Kubernetes,
Helm, and EKS scenarios.
Typical questions include:
- After deploying SeaTunnel on Kubernetes, how should I run a job?
- Can I submit a job from my local laptop to a remote Zeta cluster?
- How should `--master` and `--deploy-mode` be used?
- How should Docker client connect to master/worker nodes?
- How should Helm deployment expose REST API or client submission endpoints?
- What is the difference between local mode, cluster mode, hybrid mode, and
separated mode from an operation perspective?
Suggested sidebar location:
```text
Getting Started
-> Docker
-> Kubernetes
-> Submit Job To Remote Zeta Cluster
```
### P0-5. Multi-table Transform / Join / EtLT Capability Boundary
Many users expect multi-table CDC to support arbitrary joins and
business-entity reshaping inside the synchronization pipeline. The current
multi-table transform documentation should more explicitly describe what is
supported and what is not supported.
Typical questions include:
- Can two CDC tables be joined and written into one target table?
- Can multiple sources be joined in SQL Transform?
- Is TableMerge the same as SQL join?
- Can different tables use different transform rules?
- Can a multi-table CDC job add a common `load_time` field to all tables?
- When should users use downstream ELT/EtLT instead of in-pipeline transform?
Suggested sidebar location:
```text
Transforms
-> transform-multi-table
-> table-merge
-> multi-table-transform-and-join-boundary
```
## Acceptance criteria
- [ ] All P0 documents are added or updated in both `docs/en` and `docs/zh`
where possible.
- [ ] `docs/sidebars.js` is updated if new pages are added.
- [ ] Each new/updated document includes production-oriented examples, not
only parameter references.
- [ ] Each document includes a troubleshooting section.
- [ ] Each document clearly states capability boundaries and safe operation
rules.
- [ ] Existing related pages contain cross-links to the new guides.
- [ ] The documentation can answer the most common user questions without
requiring maintainers to repeatedly explain the same concepts in GitHub issues,
Slack, or mailing lists.
## Code of Conduct
I agree to follow this project's Code of Conduct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
