This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/seatunnel-website.git


The following commit(s) were added to refs/heads/main by this push:
     new 1900839db807 [Blog] Explain SeaTunnel performance and Tongcheng 
production practice (#568)
1900839db807 is described below

commit 1900839db8077800f61c448031f45b295eecf5a0
Author: Daniel <[email protected]>
AuthorDate: Wed Nov 18 09:42:33 2026 -0500

    [Blog] Explain SeaTunnel performance and Tongcheng production practice 
(#568)
    
    Co-authored-by: David Zollo <[email protected]>
---
 ...here-apache-seatunnel-performance-comes-from.md | 170 ++++++++++++++++++++
 ...here-apache-seatunnel-performance-comes-from.md | 173 +++++++++++++++++++++
 .../apache-seatunnel-logo.png                      | Bin 0 -> 211862 bytes
 .../end-to-end-throughput.png                      | Bin 0 -> 1453576 bytes
 .../mysql-cdc-handoff-and-recovery.png             | Bin 0 -> 1698888 bytes
 .../mysql-history-and-changes.png                  | Bin 0 -> 1365721 bytes
 .../zeta-parallel-execution.png                    | Bin 0 -> 1396128 bytes
 7 files changed, 343 insertions(+)

diff --git a/blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md 
b/blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md
new file mode 100644
index 000000000000..aef23fc8d7f2
--- /dev/null
+++ b/blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md
@@ -0,0 +1,170 @@
+---
+slug: where-apache-seatunnel-performance-comes-from
+title: "Where Apache SeaTunnel Performance Comes From: From One Record to Tens 
of Thousands of Jobs"
+description: "How Apache SeaTunnel combines efficient execution, shared CDC 
resources, parallelism, and recovery, illustrated by Tongcheng Travel's 
production architecture."
+tags: [SeaTunnel, Zeta, Performance, CDC, Data Integration]
+---
+
+Tongcheng Travel put Apache SeaTunnel at the center of its unified data 
channel.
+
+The consolidation covered tens of thousands of existing synchronization jobs 
across batch transfers and real-time integration. Application teams would keep 
their familiar submission interfaces. The platform would translate their 
configurations, submit jobs through a common service, and execute them on 
SeaTunnel's native Zeta Engine. [Tongcheng Travel's published case 
study](https://seatunnel.apache.org/blog/tongcheng-travel-unified-data-channel-with-apache-seatunnel/)
 describes that arch [...]
+
+At that scale, several aspects of SeaTunnel's design become especially useful. 
A large table needs to finish its initial load promptly. Many low-volume tables 
need their changes captured continuously. Destinations have different write 
capacities, and long-running jobs need to recover after interruptions. The 
platform must bring these workloads into one execution and operations model 
while putting its resources to work moving data.
+
+SeaTunnel addresses these needs at several levels: a shorter execution path 
for each record, shared work across multi-table synchronization, lower per-task 
execution overhead, and parallelism and recovery mechanisms that support 
sustained data delivery.
+
+Understanding those mechanisms is more useful than reading a throughput figure 
in isolation. It shows an engineering team where SeaTunnel can improve its own 
production workload, and which conditions determine the result.
+
+<!-- truncate -->
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/mysql-history-and-changes.png" 
alt="MySQL history and binlog changes flow through an Apache SeaTunnel MySQL 
CDC source, optional transforms, and a selected target connector." 
style={{display: "block", width: "100%", height: "auto"}} />
+</div>
+
+*Figure 1. Historical data and ongoing changes share a connector-based 
synchronization lifecycle. The destinations are examples, not an automatic 
broadcast to every target. Delivery semantics depend on the selected connectors 
and configuration.*
+
+## 1. A shorter path leaves more resources for useful work
+
+SeaTunnel describes a synchronization job through three logical roles: Source, 
Transform, and Sink. They define where records come from, how they are 
processed, and where they go. Zeta turns this logical description into tasks 
that can be deployed and executed.
+
+That separation also creates room for execution optimization. Before a record 
reaches its destination, it may spend a considerable amount of time simply 
moving between execution stages.
+
+Cross-node transfer typically involves serialization, buffering, network 
transmission, reception, and deserialization. For relatively small operations 
such as field mapping, type conversion, and filtering, moving the record can 
account for a meaningful share of the work. Reducing those intermediate steps 
leaves more CPU and network capacity for reading, processing, and writing data.
+
+Zeta's [execution-plan 
design](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/engines/zeta/about.md)
 explicitly aims to reduce network transfer. Where the execution plan permits 
it, adjacent tasks can be grouped into a TaskGroup and pass data locally. The 
actual grouping depends on the topology, parallelism, and any need to 
redistribute records.
+
+This preserves the logical distinction between Source, Transform, and Sink 
while making the physical execution path more compact. It does not imply that 
every job runs without network transfer or that every operator can be fused 
into one group.
+
+The benefit accumulates with data volume. Saving a small amount of work on a 
frequently traversed path matters when that path is used by every batch and 
every record. For an engineer investigating performance, this suggests a 
concrete next step: when both the source and destination have spare capacity, 
examine internal transfers, serialization, and task placement to find where the 
engine spends its resources.
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/zeta-parallel-execution.png" alt="A 
logical Source, Transform, and Sink flow is planned and scheduled into 
illustrative parallel execution instances." style={{display: "block", width: 
"100%", height: "auto"}} />
+</div>
+
+*Figure 2. The three lanes illustrate parallel execution, not a fixed thread 
layout, guaranteed scaling, or three concurrent MySQL binlog readers. 
Placement, task grouping, and parallelism follow the actual execution plan.*
+
+## 2. Moving from one table to a database changes the resource equation
+
+A large synchronization platform usually serves two different kinds of load: a 
smaller number of continuously busy jobs and many jobs whose tables change only 
intermittently.
+
+The first group needs throughput. The second tests how many synchronization 
tasks the platform can sustain. That capacity depends partly on the threads, 
connections, and runtime state each task retains even when it has little data 
to process.
+
+SeaTunnel addresses this through shared execution threads and CDC resource 
reuse.
+
+### Share execution threads across tasks
+
+Zeta's task execution service supports cooperative execution. Tasks suitable 
for shared execution enter a queue. A worker invokes a task's execution method, 
and once that call returns, the worker can serve another task. A task that has 
not finished can be scheduled again.
+
+This allows many tasks to share execution threads rather than each retaining a 
dedicated thread for its entire lifetime.
+
+Different tasks still have different execution requirements. In v3.0.0, 
[TaskExecutionService](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java)
 also provides a blocking execution path with dedicated threads. It measures 
task invocation duration to manage long-running calls in cooperative execution. 
The design makes the relationship between task count and execution-thread co 
[...]
+
+This is useful when table activity is uneven. Shared execution resources can 
serve tasks with work to do, giving a platform more control over 
thread-management overhead as its task count grows.
+
+### Share log-reading work across tables
+
+A database log contains changes from multiple tables. Building a completely 
independent synchronization job for every table can repeat log-reading, 
parsing, and related runtime work.
+
+With MySQL CDC, one Source can select multiple tables and assign them to 
incremental reading tasks. In v3.0.0, `incremental.parallelism` defaults to 
`1`, so the selected tables can share one incremental log-reading path. 
Increasing that setting groups tables across incremental tasks; it does not 
split a single table's binlog stream across readers. This reuse is within the 
configured Source; independent jobs do not automatically share a global binlog 
reader.
+
+For database-wide synchronization, this offers a practical way to organize 
related tables together, reuse common log-processing work, and route their 
records to the appropriate downstream processing paths.
+
+### Keep the initial load and ongoing changes in one lifecycle
+
+The MySQL CDC `initial` startup mode supports an initial snapshot followed by 
ongoing change capture. Snapshot work can be divided into splits, allowing the 
historical data to be read in parallel.
+
+The subtle part is that the database continues to change while those splits 
are being read.
+
+Consider the v3.0.0 path with the Source option `exactly_once=true`. Each 
snapshot split records binlog watermarks before and after reading its range. 
Changes between those watermarks are reconciled by key with the snapshot 
results. Subsequent incremental reading uses completed-split watermark 
information to identify overlapping events already covered by the snapshot 
phase.
+
+This gives snapshot chunking and the transition to streaming explicit progress 
state. It lets engineers reason about both the speed of the initial load and 
how updates made during that load enter the resulting data.
+
+The configuration boundary matters: **the MySQL CDC Source's `exactly_once` 
option defaults to `false` in v3.0.0**. That default path skips the snapshot 
backfill step described above. Source-side reconciliation, engine checkpoints, 
and sink commit semantics serve different purposes. An end-to-end consistency 
guarantee requires the appropriate combination to be configured and verified. 
Per-split reconciliation also does not constitute one globally atomic snapshot 
across the entire database.
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/mysql-cdc-handoff-and-recovery.png" 
alt="MySQL CDC v3.0.0 with exactly_once enabled uses per-split low and high 
watermarks, reconciles overlapping binlog changes, and restores split state and 
offsets from completed checkpoints." style={{display: "block", width: "100%", 
height: "auto"}} />
+</div>
+
+*Figure 3. The illustrated reconciliation path is specific to MySQL CDC v3.0.0 
with Source `exactly_once=true`. Unfinished work may replay after recovery, and 
sink delivery guarantees require separate configuration.*
+
+Together, these capabilities make SeaTunnel's value concrete: multi-table 
organization, shared log-processing work, parallel snapshot reads, and 
snapshot-to-streaming reconciliation are parts of the same integration 
framework.
+
+## 3. Match parallelism to the work the whole pipeline can complete
+
+SeaTunnel provides parallel execution. A production platform still has to 
decide how much parallelism each job should use.
+
+Tongcheng Travel's implementation illustrates that distinction. Before 
submission, its platform probes source characteristics such as table row 
counts, directory sizes, and partition or bucket counts. It combines those 
observations with available cluster resources and sink-side rate limits to 
estimate task parallelism.
+
+This estimation belongs to Tongcheng Travel's platform layer. SeaTunnel 
supplies the execution foundation; the surrounding platform uses its knowledge 
of data volume and capacity to generate suitable configurations. The case study 
should not be read as a claim that every SeaTunnel deployment automatically 
applies the same policy.
+
+A useful parallelism choice has to fit three constraints.
+
+**How much source work can run in parallel?** Splitting strategy, data 
distribution, and indexes influence read efficiency. Configuring more 
parallelism does not eliminate skew. If most of the remaining data sits in a 
few splits, the job can still wait for those slow splits to finish.
+
+**How much execution capacity is available?** Reads and transformations 
consume CPU, memory, and network bandwidth. More concurrent tasks improve 
throughput only while the cluster can support the additional work.
+
+**How quickly can the destination complete writes?** Batch sizes, connection 
counts, index maintenance, and commit behavior all affect the destination's 
capacity. Accounting for that capacity helps the entire pipeline operate at a 
balanced rate.
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/end-to-end-throughput.png" alt="Source 
capacity, execution capacity, and sink capacity jointly constrain throughput, 
with backpressure propagating upstream." style={{display: "block", width: 
"100%", height: "auto"}} />
+</div>
+
+*Figure 4. Tune the constrained stage instead of increasing parallelism 
blindly. This is a qualitative model, not a benchmark or a claim of linear 
scaling.*
+
+During tuning, observe source reads, successful destination commits, and 
end-to-end latency together. These measurements show whether additional 
resources produce more successfully delivered data or merely move the queue to 
another part of the system.
+
+Tongcheng Travel's approach also shows why a common job model helps platform 
teams. They can improve parallelism estimation centrally while application 
teams continue submitting work through existing interfaces.
+
+## 4. Recovery cost is part of production performance
+
+Once synchronization runs in production, performance includes what happens 
after an interruption.
+
+How much data must be reread? Which state must be restored? How many writes 
must be repeated? Those costs affect both completion time and how much cluster 
capacity remains available for new data.
+
+SeaTunnel uses the Pipeline as an important checkpoint and recovery boundary. 
Pipelines in the physical execution plan can have separate checkpoint 
coordination. For recoverable task-level failures, recovery can be scoped to 
the affected Pipeline instead of rolling back unrelated healthy Pipelines. 
Wider failures and exhausted retry policies can still affect the job or cluster.
+
+The actual execution topology determines this boundary. A table is not 
automatically an independent Pipeline. A Source branching to several Sinks does 
not necessarily create separate recovery units either. Understanding how the 
plan is divided into Pipelines helps a platform team relate its business 
organization to its recovery scope.
+
+Checkpoints preserve the progress information required by participating 
connectors, including read positions, split state, and write state. Recovery 
restores state from a completed checkpoint. With a Sink that supports the 
corresponding commit protocol, checkpoints can also coordinate destination 
commits and recovery.
+
+These mechanisms help sustain useful throughput by containing recovery work 
and limiting how much data needs to be reprocessed. They do not remove recovery 
cost.
+
+Checkpointing itself consumes resources. More frequent checkpoints generally 
require more state processing and storage work while providing a more recent 
recovery point. A production configuration should account for state size, 
checkpoint completion time, and the recovery time the workload can tolerate.
+
+Tongcheng Travel monitors this operational layer explicitly. Its platform 
observes checkpoint end-to-end duration, state size, and failure frequency, 
alongside master election and failover metrics. That makes it easier to 
distinguish task-processing problems from control-plane problems.
+
+The performance picture is therefore continuous: use resources efficiently 
during normal operation, constrain repeated work after failures, and use 
measurements to determine whether both goals are being met.
+
+## From engine design to an evolving data platform
+
+Tongcheng Travel's experience also shows how SeaTunnel can fit into a platform 
that has been running for years.
+
+Existing tasks are translated centrally into SeaTunnel configurations. During 
migration, the legacy job remains the production baseline while the translated 
SeaTunnel job writes to an isolated destination. Output validation gates the 
cutover. The platform team can then improve connector integration, resource 
policies, and observability around a common runtime without replacing every 
application team's submission workflow.
+
+At the record level, SeaTunnel reduces unnecessary transfer work. At the task 
level, shared execution and CDC resource reuse control overhead. At the 
platform level, a common configuration and batch-and-streaming execution model 
provide a foundation for centralized resource management, monitoring, and 
migration.
+
+These layers explain where SeaTunnel's performance potential comes from. To 
evaluate it on a real workload, trace the execution boundaries a record 
crosses, identify the work shared across tables, test whether source and 
destination capacity support the proposed parallelism, and measure how much 
work recovery repeats.
+
+The answers turn a general claim of high performance into an engineering 
assessment grounded in a specific deployment, configuration, and observed 
result.
+
+## Sources and version scope
+
+The implementation discussion in this article is scoped to Apache SeaTunnel 
**v3.0.0**, using its documentation and source code. It is not a claim about 
every release or connector, and this article does not report a new performance 
benchmark or fault-injection test.
+
+The Tongcheng Travel case study summarizes a public Meetup presentation. 
**Tens of thousands refers to existing jobs in the migration scope, not a 
measured number of concurrently running jobs.** The published case does not 
provide a complete before-and-after performance dataset, so no throughput or 
resource-saving multiplier is inferred here.
+
+- [Tongcheng Travel: a unified data channel with Apache 
SeaTunnel](https://seatunnel.apache.org/blog/tongcheng-travel-unified-data-channel-with-apache-seatunnel/)
+- [Zeta Engine design, 
v3.0.0](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/engines/zeta/about.md)
+- [TaskExecutionService, 
v3.0.0](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java)
+- [MySQL CDC configuration, 
v3.0.0](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/connectors/source/MySQL-CDC.md)
+- [Snapshot split low and high watermarks, 
v3.0.0](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/scan/MySqlSnapshotSplitReadTask.java#L137-L159)
+- [Snapshot backfill and the exactly_once branch, 
v3.0.0](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/scan/MySqlSnapshotFetchTask.java#L83-L123)
+
+- [TaskGroup fusion and Pipeline recovery 
boundaries](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/architecture/engine/dag-execution.md)
+- [Incremental parallelism 
configuration](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java#L75-L118)
+- [Table grouping and incremental split 
watermarks](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/IncrementalSplitAssigner.java#L213-L277)
+- [Checkpoint coordination per 
Pipeline](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointManager.java#L119-L168)
diff --git 
a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md
 
b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md
new file mode 100644
index 000000000000..21ec319bdaee
--- /dev/null
+++ 
b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-11-06-where-apache-seatunnel-performance-comes-from.md
@@ -0,0 +1,173 @@
+---
+slug: where-apache-seatunnel-performance-comes-from
+title: "Apache SeaTunnel 的性能从哪里来:从一条数据的传输,到数万条任务的生产实践"
+description: "从同程旅行的生产架构出发,理解 Apache SeaTunnel 的执行路径、CDC 资源复用、并行度与恢复机制。"
+tags: [SeaTunnel, Zeta, Performance, CDC, Data Integration]
+---
+
+同程旅行把 Apache SeaTunnel 放在了统一数据通道的核心。
+
+这次整合面向数万条既有同步任务,覆盖离线搬运和实时数据集成。平台团队希望业务人员继续使用熟悉的任务入口,由平台完成配置转换与统一提交,再交给 
SeaTunnel 的原生 Zeta 
引擎执行。[同程旅行的公开案例](https://seatunnel.apache.org/blog/tongcheng-travel-unified-data-channel-with-apache-seatunnel/)记录了这套架构及其落地过程。
+
+这个规模让 SeaTunnel 的几项设计变得格外有意义。
+
+一张大表需要尽快完成历史数据同步;大量低流量表需要持续捕获变化;不同目标系统有不同的写入能力;长时间运行的任务还需要从中断中恢复。平台要把这些工作放进同一套执行与运维体系,并让机器资源真正用于数据传输。
+
+SeaTunnel 
的性能设计正是沿着这些具体问题展开的:缩短数据在引擎内部的路径,复用多表同步中的公共工作,降低任务的运行开销,再通过合适的并行度与恢复机制,维持持续的数据交付。
+
+理解这些机制,比孤立地看一个吞吐数字,更能帮助工程师判断 SeaTunnel 在自己的生产环境中能发挥什么价值。
+
+<!-- truncate -->
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/mysql-history-and-changes.png" 
alt="MySQL 历史数据与 binlog 变更通过 Apache SeaTunnel 的 Source、Transform 和 Sink 
同步到目标系统。" style={{display: "block", width: "100%", height: "auto"}} />
+</div>
+
+*图 1:历史数据和持续变更通过同一条同步生命周期进入选定目标。目标类别只是示例,并非自动广播;交付语义取决于连接器与配置。*
+
+## 一、一条数据少走几次路,执行资源就能多做一些有效工作
+
+SeaTunnel 用 Source、Transform 和 Sink 描述一条数据同步链路:从哪里读,经过什么处理,写到哪里。
+
+这三个逻辑角色划分清楚,也给执行优化留下了空间。用户描述的是数据如何流动,Zeta 则负责把这份描述转换成实际部署和执行的任务。
+
+这里有一个容易被忽略的性能问题:一条记录在完成业务转换之前,可能已经为“移动自己”付出了不少开销。
+
+跨节点传输通常涉及序列化、缓冲、网络发送、接收和反序列化。对于字段映射、类型转换、过滤这类处理,数据在执行阶段之间的传递成本,可能占据相当一部分运行时间。减少这些中间工作,就能让更多
 CPU 和网络资源用于真正的读取与写入。
+
+Zeta 的执行计划优化把减少网络传输列为明确目标。在适合合并的执行条件下,相邻任务可以组织进同一个 
TaskGroup,在本地完成数据交接,降低跨节点传递带来的开销。具体组织方式取决于任务拓扑、并行度以及是否需要重新分发数据。
+
+这让 Source、Transform、Sink 的逻辑分工可以保留,同时让物理执行路径更紧凑。
+
+从工程角度看,这项设计的价值会随着数据量累积。一次传递省下的工作很小,但同样的路径会被每一批、每一条数据反复经过。把常走的路径缩短,是同步引擎提高效率的一项基础能力。
+
+这也给性能分析提供了明确切入点:当源端与目标端都有余量时,可以进一步观察引擎内部的数据传输、序列化和任务部署情况,判断资源消耗在哪一段。
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/zeta-parallel-execution.png" alt="Zeta 
将逻辑 Source、Transform、Sink 链路规划为并行执行实例,实际部署由执行计划决定。" style={{display: "block", 
width: "100%", height: "auto"}} />
+</div>
+
+*图 2:三条泳道仅示意并行执行,不代表固定线程布局、线性扩展保证,或三条并发 MySQL binlog reader。*
+
+## 二、从一张表到一整库,性能开始取决于资源复用
+
+大规模同步平台通常同时承载两类负载:少数持续繁忙的任务,以及大量流量较小、间歇产生变化的任务。
+
+前者需要吞吐,后者更考验任务密度。同样的机器能稳定承载多少条同步链路,取决于每条链路除了处理数据,还要维持多少线程、连接与运行状态。
+
+SeaTunnel 在这一层做了两项有辨识度的优化:线程共享和 CDC 资源复用。
+
+**让执行线程服务于多项任务。**Zeta 
的任务执行服务支持协作式执行。适合共享的任务进入队列,工作线程调用任务的一次执行逻辑,任务返回后,线程可以继续处理其他任务。尚未完成的任务再等待后续调度。
+
+这样,大量任务可以复用执行线程,降低为每项工作长期保留独立线程的开销。
+
+共享执行需要照顾不同任务的行为。v3.0.0 的 TaskExecutionService 
同时保留了独占线程的阻塞执行路径,并通过任务调用计时管理协作执行中的长时间占用。它把“任务数量”和“执行线程数量”之间的关系做得更灵活。
+
+对于多表实时同步,这种灵活性很实用。不同表的变更往往并不均匀,执行资源可以在有工作可做的任务之间周转。表的数量增长时,线程管理开销也就有了更好的控制空间。
+
+**多张表共享日志读取工作。**数据库日志里包含多张表的变化。如果把每张表都建成完全独立的同步链路,日志读取、解析和相关运行资源也可能被重复建立。
+
+以 MySQL CDC 为例,一个 Source 可以选择多张表,将它们分组交给增量读取任务。增量并行度 `incremental.parallelism` 
默认值为 1 时,选定的多张表可以共享这一条增量日志读取路径;提高增量并行度后,表会分组处理,并不是将同一张表的 binlog 流拆给多个 
reader。这种复用发生在该 Source 内,而不是不同作业之间自动全局共享。
+
+这项能力对整库同步尤其有价值:业务人员可以按一组相关表组织同步任务,让公共日志处理工作得到复用,再把各表记录送往对应的处理路径。
+
+**历史数据与持续变化,放在同一条同步生命周期里。**MySQL CDC 的 initial 启动模式支持历史数据同步以及后续增量捕获。快照读取可以按 
split 拆分,为历史数据阶段提供并行读取能力。
+
+真正需要处理好的,是快照读取期间数据库仍然发生的变化。
+
+以 v3.0.0 中启用 Source 的 exactly_once 选项的路径为例:每个快照 split 会记录读取前后的 binlog 
水位,读取对应范围的数据,再把水位区间内的日志变化按键合并到快照结果中。后续增量读取利用各 split 的水位信息,识别已经覆盖的记录。
+
+这套处理让快照分块与增量衔接有了明确的状态依据。分片级别的衔接并不等于整个数据库在同一时刻的原子快照。工程师可以同时考虑历史数据读取速度,以及读取期间发生的更新如何进入最终结果。
+
+这里需要区分一个具体配置边界:v3.0.0 的 MySQL CDC Source 中,`exactly_once` 
默认关闭,这条默认路径会跳过前述快照日志回填;源端选项、引擎检查点和目标端提交语义分别承担不同职责。端到端的一致性,需要把这几部分一起配置和验证。
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/mysql-cdc-handoff-and-recovery.png" 
alt="MySQL CDC v3.0.0 开启 exactly_once 后的分片水位、日志回填和检查点恢复。" style={{display: 
"block", width: "100%", height: "auto"}} />
+</div>
+
+*图 3:图示适用于 MySQL CDC v3.0.0 的 Source exactly_once=true 
路径。未完成工作在恢复后可能重放,目标端交付保证仍需单独配置。*
+
+SeaTunnel 的优势由此变得具体:它既提供多表同步的组织方式,也把资源复用、快照拆分和增量衔接放进了同一套数据集成能力中。
+
+## 三、把并行度配置好,才能把引擎能力变成有效吞吐
+
+SeaTunnel 提供了并行执行能力,生产平台还需要决定:一项任务应该使用多少并行度?
+
+同程旅行的做法给出了一个具体答案。
+
+其平台会在提交任务前探查源端特征,例如表的行数、文件目录大小,以及数据源的分区或分桶数量,再结合可用集群资源和目标端限速,估算任务并行度。
+
+这个估算发生在同程的平台层。SeaTunnel 提供执行基础,平台利用自身掌握的数据规模和资源信息,为不同任务生成更合适的执行配置。
+
+这一分工值得借鉴,因为并行度需要同时匹配三个条件。
+
+首先是**源端有多少可以并行读取的工作**。表的切分方式、数据分布和索引条件会影响实际读取效率。即使配置了较高并行度,如果工作集中在少数切分上,任务仍可能等待最后几个慢分片完成。
+
+其次是**执行集群有多少可用资源**。读取与转换需要 CPU、内存和网络。并行任务增加后,这些资源要能支撑新增工作,吞吐才有继续增长的空间。
+
+最后是**目标端能以多快的速度完成写入**。写入批次、连接数量、索引维护和提交方式,都可能影响目标系统的接收能力。把目标端的容量纳入配置,能让整条链路以更平衡的速度工作。
+
+<div style={{position: "relative", background: "#fff", paddingTop: "6%", 
marginBottom: "1rem"}}>
+  <img src="/blog/seatunnel_performance/apache-seatunnel-logo.png" alt="" 
style={{position: "absolute", top: "0", right: "1%", width: "5.5%", height: 
"auto"}} />
+  <img src="/blog/seatunnel_performance/end-to-end-throughput.png" 
alt="源端能力、执行资源与目标端写入能力共同约束吞吐,反压向上游传播。" style={{display: "block", width: "100%", 
height: "auto"}} />
+</div>
+
+*图 4:先定位受限阶段,再调整并行度。此图为定性模型,不是性能测试数据或线性扩展承诺。*
+
+因此,调优时值得同时观察源端读取、目标端成功提交和端到端延迟。它们可以帮助工程师判断:增加并行度之后,新增资源是否转化成了更多完成交付的数据。
+
+同程的实践体现了 SeaTunnel 
作为平台执行基础的一项长处:统一的任务模型让资源策略有了共同的落点。平台可以集中改进并行度估算,而业务团队继续通过原有入口提交任务。
+
+## 四、让已经完成的工作,在长时间运行中继续产生价值
+
+同步任务进入生产之后,性能还包含恢复成本。
+
+一次中断之后,要重新读取多少数据、恢复多少状态、重做多少写入,会影响任务最终何时完成,也会影响集群有多少资源能够继续处理新数据。
+
+SeaTunnel 把 Pipeline 作为检查点和故障恢复的重要边界。物理执行计划中的不同 Pipeline 
可以拥有各自的检查点协调过程;发生任务级故障时,恢复工作可以围绕受影响的 Pipeline 展开,通常无需让无关的健康 Pipeline 
一同回退。更大范围的故障或重试次数耗尽,仍可能影响整个作业或集群。
+
+这个边界由实际执行拓扑决定。一张表并不天然对应一个独立 Pipeline,一个 Source 分流到多个 
Sink,也不必然意味着多个独立恢复单元。理解任务如何被拆成 Pipeline,有助于把业务组织方式与故障恢复范围对应起来。
+
+检查点则为恢复保存进度依据,包括连接器所需的读取位置、切分状态和写入状态。恢复时,任务可以利用已完成检查点中的状态继续执行。配合具备相应提交能力的 
Sink,还能协调目标端提交与恢复过程。
+
+这些机制对持续吞吐的意义在于:让故障后的工作范围更可控,减少无关链路参与恢复,并限制需要重复处理的数据量。
+
+检查点本身也有成本。更频繁的检查点通常意味着更多状态处理与存储工作,同时提供更近的恢复位置。适合生产的配置,需要结合状态大小、完成时长和可接受的恢复时间选择。
+
+同程旅行也把这一层纳入了平台监控:检查点的端到端耗时、状态大小和失败频率,与集群主节点的选举及切换指标一起被观察。任务的数据处理情况和控制面的运行情况,因而能够被分别定位。
+
+性能优化由此形成了完整链条:正常运行时提高有效处理效率,中断发生后控制恢复代价,再用运行指标判断这些设计是否发挥了作用。
+
+## 从引擎设计,到可以持续演进的数据通道
+
+回到同程旅行的案例,SeaTunnel 的价值还体现在它如何进入一个已经运行多年的数据平台。
+
+既有任务被集中转换成 SeaTunnel 配置。迁移期间旧任务继续作为生产基线,转换后的 SeaTunnel 
任务写入隔离目标,并通过结果验证逐步切换。平台团队可以围绕同一执行基础,继续改进连接器适配、资源策略和可观测性。业务入口得以延续,运行层则获得了统一演进的空间。
+
+从底层看,SeaTunnel 关注一条记录如何减少传输开销;从任务层看,它通过线程共享和 CDC 
资源复用提高运行效率;从平台层看,统一配置与批流执行模型使资源调度、监控和迁移策略更容易集中建设。
+
+这几层能力共同解释了 SeaTunnel 的性能设计为何适合大规模数据同步。
+
+对于准备评估 SeaTunnel 
的工程团队,可以沿着同样的路径观察真实负载:数据经过哪些执行边界,多表之间共享了哪些工作,源端和目标端能否承受计划中的并行度,恢复时又需要重做多少数据。
+
+这些问题的答案,会把“高性能”落到具体的部署、配置和运行结果上。SeaTunnel 提供的,正是一套围绕这些问题持续优化的数据同步基础。
+
+## 资料与版本说明
+
+本文的技术机制以 Apache SeaTunnel v3.0.0 
文档与源码为依据,不构成对所有版本或连接器的保证,也不是一次新的性能基准测试或故障注入测试。同程旅行案例来自官方站点对公开 Meetup 
分享的整理;其中“数万条”指既有任务规模,不代表同时并发的压测结果。公开案例未披露完整的前后对照性能数据,本文未据此推导吞吐或资源节省倍数。
+
+- 
[同程旅行:统一批流数据通道的生产实践](https://seatunnel.apache.org/blog/tongcheng-travel-unified-data-channel-with-apache-seatunnel/)
+- [SeaTunnel v3.0.0:Zeta 
引擎设计](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/engines/zeta/about.md)
+- 
[TaskExecutionService:线程共享源码](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java)
+- [MySQL 
CDC:多表与快照配置](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/connectors/source/MySQL-CDC.md)
+- [快照 split 
低、高水位源码](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/scan/MySqlSnapshotSplitReadTask.java#L137-L159)
+- [快照日志回填与 exactly_once 
分支](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/scan/MySqlSnapshotFetchTask.java#L83-L123)
+
+
+- [TaskGroup 融合与 Pipeline 
恢复边界](https://github.com/apache/seatunnel/blob/v3.0.0/docs/en/architecture/engine/dag-execution.md)
+- 
[增量并行度配置](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java#L75-L118)
+- [多表分组与增量 split 
水位](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/IncrementalSplitAssigner.java#L213-L277)
+- [Pipeline 
级检查点协调](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointManager.java#L119-L168)
diff --git a/static/blog/seatunnel_performance/apache-seatunnel-logo.png 
b/static/blog/seatunnel_performance/apache-seatunnel-logo.png
new file mode 100644
index 000000000000..aafed25d366c
Binary files /dev/null and 
b/static/blog/seatunnel_performance/apache-seatunnel-logo.png differ
diff --git a/static/blog/seatunnel_performance/end-to-end-throughput.png 
b/static/blog/seatunnel_performance/end-to-end-throughput.png
new file mode 100644
index 000000000000..b61652556959
Binary files /dev/null and 
b/static/blog/seatunnel_performance/end-to-end-throughput.png differ
diff --git 
a/static/blog/seatunnel_performance/mysql-cdc-handoff-and-recovery.png 
b/static/blog/seatunnel_performance/mysql-cdc-handoff-and-recovery.png
new file mode 100644
index 000000000000..bddbdfe5db1a
Binary files /dev/null and 
b/static/blog/seatunnel_performance/mysql-cdc-handoff-and-recovery.png differ
diff --git a/static/blog/seatunnel_performance/mysql-history-and-changes.png 
b/static/blog/seatunnel_performance/mysql-history-and-changes.png
new file mode 100644
index 000000000000..e82a6593909a
Binary files /dev/null and 
b/static/blog/seatunnel_performance/mysql-history-and-changes.png differ
diff --git a/static/blog/seatunnel_performance/zeta-parallel-execution.png 
b/static/blog/seatunnel_performance/zeta-parallel-execution.png
new file mode 100644
index 000000000000..0d3e3099ef16
Binary files /dev/null and 
b/static/blog/seatunnel_performance/zeta-parallel-execution.png differ

Reply via email to