This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/seatunnel-website.git
The following commit(s) were added to refs/heads/main by this push:
new 651d4ccf1508 [Blog] Add Tongcheng Travel unified data channel case
study (#487)
651d4ccf1508 is described below
commit 651d4ccf15080f8015e5634d5d22792357378627
Author: Daniel <[email protected]>
AuthorDate: Sat Jul 18 22:07:02 2026 +0800
[Blog] Add Tongcheng Travel unified data channel case study (#487)
---
...l-unified-data-channel-with-apache-seatunnel.md | 170 +++++++++++++++++++++
...l-unified-data-channel-with-apache-seatunnel.md | 170 +++++++++++++++++++++
2 files changed, 340 insertions(+)
diff --git
a/blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
b/blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
new file mode 100644
index 000000000000..fe6b4777cd6c
--- /dev/null
+++
b/blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
@@ -0,0 +1,170 @@
+---
+slug: tongcheng-travel-unified-data-channel-with-apache-seatunnel
+title: "From Four Pipelines to One: How Tongcheng Travel Built a Unified Data
Channel with Apache SeaTunnel"
+tags: [Meetup, SeaTunnel, Zeta, Tongcheng Travel, Data Integration, AI]
+authors:
+ - name: Daniel
+ title: Apache SeaTunnel Contributor
+ url: https://github.com/DanielLeens
+ image_url: https://github.com/DanielLeens.png
+---
+
+Tongcheng Travel's data channel evolved over several years into four parallel
systems for offline transfer, real-time integration, Sqoop jobs, and SeaTunnel
jobs. Each system solved a problem at a particular stage, but their overlapping
capabilities, different execution engines, and separate operational models
eventually became a barrier to platform-wide governance.
+
+At an Apache SeaTunnel Meetup, Xiaochen Zhou, who works on the data platform
at Tongcheng Travel, explained how the company consolidated those systems into
a unified batch and streaming data channel based on the Apache SeaTunnel Zeta
Engine. The project had three non-negotiable goals: keep the migration
transparent to application teams, prove data consistency before switching
traffic, and improve execution efficiency and operational stability.
+
+This article summarizes the architecture, migration safeguards, AI-assisted
task generation, validation design, and future direction presented in that
session.
+
+<!-- truncate -->
+
+## Why four data channels had to become one
+
+Before the consolidation, Tongcheng Travel operated four data transfer paths:
+
+- An offline data transfer service based on Flink 1.6
+- A real-time data integration service based on Flink
+- Legacy Sqoop jobs running on MapReduce
+- SeaTunnel Zeta jobs already used by part of the business
+
+The systems covered similar source and sink combinations, but they differed in
task syntax, runtime behavior, resource models, and operational tooling. Legacy
MapReduce jobs were comparatively heavy, the old Flink runtime was difficult to
maintain, and large database extraction workloads could also put pressure on
online databases.
+
+The target architecture was therefore not another independent service. It was
one platform entry point and one execution foundation for batch and streaming
data movement.
+
+```mermaid
+flowchart LR
+ subgraph Legacy[Legacy task interfaces]
+ Offline[Offline transfer]
+ Realtime[Real-time integration]
+ Sqoop[Sqoop scripts]
+ Existing[Existing SeaTunnel jobs]
+ end
+
+ Submit[Unified submission service]
+ Translate[Task translation and validation]
+ Zeta[Apache SeaTunnel Zeta Engine]
+ Targets[Databases, lakes, warehouses, and messaging systems]
+
+ Offline --> Submit
+ Realtime --> Submit
+ Sqoop --> Submit
+ Existing --> Submit
+ Submit --> Translate
+ Translate --> Zeta
+ Zeta --> Targets
+```
+
+*Simplified flow based on the public Meetup presentation.*
+
+Tongcheng Travel established three principles for the migration:
+
+1. **No application-side rewrite.** Existing teams should continue submitting
their familiar scripts while the platform translates them underneath.
+2. **Consistency before cutover.** New and old engines must run in parallel
until their outputs pass validation.
+3. **Better efficiency and stability.** The new foundation should reduce the
burden of maintaining several runtimes while taking advantage of Zeta's
data-integration-oriented execution model.
+
+## Translating existing tasks without changing user behavior
+
+The largest migration challenge was not writing a new SeaTunnel job. It was
moving tens of thousands of existing Sqoop and Flink SQL tasks without
requiring every application team to rewrite its jobs.
+
+Tongcheng Travel introduced a skill layer that recognizes the syntax and
configuration of each legacy task type. A Sqoop skill, a Flink SQL skill, and a
SeaTunnel skill map the original task into a standard SeaTunnel configuration.
A unified submission service then validates and submits that generated
configuration.
+
+Migration uses a dual-run process:
+
+```mermaid
+flowchart TD
+ Trigger[Scheduler triggers an existing task]
+ Baseline[Run the legacy task as the production baseline]
+ Convert[Translate the task into SeaTunnel configuration]
+ Candidate[Run the SeaTunnel task against an isolated target]
+ Compare[Validate outputs from both engines]
+ Passed{Validation passed?}
+ Switch[Route future executions to SeaTunnel]
+ Diagnose[Keep the original route and investigate differences]
+
+ Trigger --> Baseline
+ Trigger --> Convert
+ Convert --> Candidate
+ Baseline --> Compare
+ Candidate --> Compare
+ Compare --> Passed
+ Passed -->|Yes| Switch
+ Passed -->|No| Diagnose
+```
+
+*Simplified migration flow based on the public Meetup presentation.*
+
+The legacy task remains the production baseline during migration, while the
translated SeaTunnel task writes to an isolated location. Only after validation
succeeds does the platform change the execution route. This keeps the migration
invisible to application developers and provides a rollback boundary for every
task.
+
+## Building new tasks from natural language
+
+The platform also explores a different entry point for new workloads:
describing a data movement requirement in natural language.
+
+A request such as "synchronize yesterday's core product sales data to the
warehouse" is incomplete by itself. The system first combines the request with
metadata and historical behavior to identify the likely source table, target
system, partitions, and execution parameters. The SeaTunnel skill then
assembles the corresponding source, transform, and sink configuration.
+
+For SQL generation, Tongcheng Travel uses multiple candidate-generation
strategies rather than trusting one model response:
+
+- A reasoning generator creates SQL directly from the intent and schema.
+- In-context learning retrieves successful historical SQL as examples.
+- A divide-and-conquer generator decomposes complex SQL into smaller common
table expressions.
+- A selector evaluates syntax, abstract syntax tree complexity, and estimated
execution cost before choosing the final SQL.
+
+For transformation tasks, common requirements are routed to native SeaTunnel
transforms such as Filter, Replace, and Split. When a requirement cannot be
expressed by a normal transform combination, the platform can generate code or
a script and load the compiled result as a task plugin. An interactive data
preview remains part of the workflow so that users can confirm the result
before execution.
+
+This design treats the model as a task authoring assistant, while metadata,
validation, preview, and the SeaTunnel runtime provide the engineering
guardrails.
+
+## Proving data consistency during migration
+
+Running both engines is useful only when their outputs can be compared
reliably. Tongcheng Travel uses different validation strategies for file
outputs and table outputs.
+
+### File output validation
+
+Loading every record from large files into memory would be slow and could
cause out-of-memory failures. Instead, the validation process normalizes each
row, calculates a hash, and builds an external index containing the hash, file
identifier, and line number.
+
+The comparison has two levels:
+
+1. Bloom filters quickly identify records that are definitely absent from the
other output.
+2. Externally sorted indexes are compared with two pointers. When equal hashes
are found, their occurrence counts are also compared so duplicate rows are not
hidden.
+
+If a difference is detected, the file identifier and line number provide a
direct path back to the original record. This avoids an expensive in-memory
full comparison while preserving the ability to diagnose a mismatch.
+
+### Table output validation
+
+For relational databases and warehouses, the platform uses the target system's
SQL engine to calculate per-column characteristics. One example presented in
the session applies `murmur_hash3_32` after normalizing null values, then
aggregates those hashes for comparison.
+
+This method is intended to detect migration differences efficiently.
+
+> **Editorial note:** For critical data, hash comparison should remain one
layer of a broader validation strategy that also includes row counts, null
distributions, key business aggregates, and targeted record checks.
+
+## Improving connectors, parallelism, and observability
+
+After the execution path was unified, Tongcheng Travel continued improving the
runtime around three areas.
+
+### Connector reliability
+
+The work described in the session covered Paimon type support and predicate
pushdown, high availability for StarRocks and Doris frontends, Kafka and
RocketMQ stream stability, HBase range reads, and HDFS ViewFs compatibility.
These improvements reflect a practical lesson from platform consolidation: the
unified engine must handle the long tail of production source and sink
behavior, not only the common happy path.
+
+### Adaptive parallelism
+
+Before task submission, the platform probes source characteristics such as row
count, directory size, Kafka partition count, or Paimon bucket count. It
combines those characteristics with available cluster resources and sink-side
rate limits to estimate task parallelism. The goal is to align execution
concurrency with both the physical data layout and the capacity of the
destination system.
+
+### Operational visibility
+
+The platform extends checkpoint monitoring with end-to-end duration, state
size, and failure frequency. It also monitors master election duration,
failover frequency, and active-master health. These metrics make it easier to
distinguish a task-level bottleneck from an engine or cluster-control-plane
problem.
+
+## What the architecture demonstrates
+
+Tongcheng Travel's experience is not simply a migration from one tool to
another. Its main value lies in how the migration was controlled:
+
+- Preserve existing user interfaces while replacing the runtime underneath.
+- Translate tasks centrally instead of distributing migration work across
application teams.
+- Run old and new engines together before switching production traffic.
+- Make output validation a cutover gate rather than a post-migration check.
+- Use AI to reduce task authoring effort, but keep metadata, preview,
selection, and validation in the control loop.
+- Treat connector coverage, adaptive execution, and observability as platform
capabilities.
+
+The next stage presented by Tongcheng Travel focuses on Kubernetes-native
elastic workers, centralized remote job logs, natural-language task generation,
automated root-cause analysis, and automatic parallelism tuning. Together,
these capabilities move the data channel from manually operated pipelines
toward an increasingly self-managing integration platform.
+
+## Speaker
+
+Xiaochen Zhou works on the data platform at Tongcheng Travel and presented
this architecture at an Apache SeaTunnel Meetup. This independently written
technical summary reports the publicly presented facts in a new structure; it
does not reproduce the source article's images or extended passages.
+
+- [Xiaochen Zhou on GitHub](https://github.com/xiaochen-zhou)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
new file mode 100644
index 000000000000..2bcadc90781f
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-18-tongcheng-travel-unified-data-channel-with-apache-seatunnel.md
@@ -0,0 +1,170 @@
+---
+slug: tongcheng-travel-unified-data-channel-with-apache-seatunnel
+title: "一套替代四套：同程旅行如何用 Apache SeaTunnel 建设统一数据通道"
+tags: [Meetup, SeaTunnel, Zeta, 同程旅行, 数据集成, AI]
+authors:
+ - name: Daniel
+ title: Apache SeaTunnel Contributor
+ url: https://github.com/DanielLeens
+ image_url: https://github.com/DanielLeens.png
+---
+
+同程旅行的数据通道经过多年演进，逐步形成了离线搬运、实时集成、Sqoop 和 SeaTunnel
四套系统并存的格局。每套系统都解决了特定阶段的问题，但功能重叠、执行引擎割裂、运维体系分散，也逐渐成为平台统一治理的障碍。
+
+在 Apache SeaTunnel Meetup 上，负责同程旅行数据平台相关工作的周晓晨，分享了同程旅行以 Apache SeaTunnel Zeta
引擎为统一底座，将四套系统收敛为批流一体数据通道的实践。整个项目有三个不可妥协的目标：迁移过程对业务透明、切换前证明数据一致、同时提升执行效率和运行稳定性。
+
+本文以该场公开分享的事实为依据，独立梳理其统一架构、迁移保障、AI 辅助任务生成、数据校验设计以及后续规划。
+
+<!-- truncate -->
+
+## 为什么要把四套通道统一为一套
+
+整合之前，同程旅行运行着四类数据传输链路：
+
+- 基于 Flink 1.6 的离线数据搬运服务；
+- 基于 Flink 的实时数据集成服务；
+- 运行在 MapReduce 上的存量 Sqoop 任务；
+- 已经在部分业务中使用的 SeaTunnel Zeta 任务。
+
+这些系统覆盖了相似的数据源和目标端，但任务语法、运行方式、资源模型和运维工具各不相同。存量 MapReduce 任务相对笨重，老版本 Flink
的维护成本较高，大规模数据库抽取任务还可能给在线数据库带来压力。
+
+因此，目标架构不是再建设第五套服务，而是形成统一的平台入口和批流一体执行底座。
+
+```mermaid
+flowchart LR
+ subgraph Legacy[存量任务入口]
+ Offline[离线数据搬运]
+ Realtime[实时数据集成]
+ Sqoop[Sqoop 脚本]
+ Existing[已有 SeaTunnel 任务]
+ end
+
+ Submit[统一提交服务]
+ Translate[任务转换与校验]
+ Zeta[Apache SeaTunnel Zeta 引擎]
+ Targets[数据库、数据湖、数仓和消息系统]
+
+ Offline --> Submit
+ Realtime --> Submit
+ Sqoop --> Submit
+ Existing --> Submit
+ Submit --> Translate
+ Translate --> Zeta
+ Zeta --> Targets
+```
+
+*根据公开 Meetup 分享整理的简化流程。*
+
+同程旅行为迁移设定了三项原则：
+
+1. **业务侧零改造。** 业务团队继续使用熟悉的脚本和提交方式，由平台在底层完成转换。
+2. **验证通过后再切换。** 新老引擎必须并行运行，输出校验通过后才能切换执行路由。
+3. **效率和稳定性共同提升。** 在减少多套运行时维护负担的同时，发挥 Zeta 面向数据集成场景的执行优势。
+
+## 在不改变用户习惯的情况下转换存量任务
+
+迁移最大的难点并不是新写一个 SeaTunnel 任务，而是在不要求业务逐一改造的前提下，迁移数万个存量 Sqoop 和 Flink SQL 任务。
+
+同程旅行引入了一层 Skill 能力组件，用于识别不同存量任务的语法和配置。Sqoop Skill、Flink SQL Skill 和 SeaTunnel
Skill 分别负责将原任务映射为标准的 SeaTunnel 配置，再由统一提交服务完成校验和提交。
+
+迁移采用双跑机制：
+
+```mermaid
+flowchart TD
+ Trigger[调度系统触发存量任务]
+ Baseline[运行原任务，作为生产基线]
+ Convert[转换为 SeaTunnel 配置]
+ Candidate[SeaTunnel 任务写入隔离目标]
+ Compare[比较新老引擎输出]
+ Passed{校验是否通过？}
+ Switch[后续执行路由切换至 SeaTunnel]
+ Diagnose[保留原路由并定位差异]
+
+ Trigger --> Baseline
+ Trigger --> Convert
+ Convert --> Candidate
+ Baseline --> Compare
+ Candidate --> Compare
+ Compare --> Passed
+ Passed -->|通过| Switch
+ Passed -->|未通过| Diagnose
+```
+
+*根据公开 Meetup 分享整理的简化迁移流程。*
+
+迁移期间，原任务继续承担生产基线，转换后的 SeaTunnel 任务写入隔离目录或测试目标。只有校验通过，平台才会把后续执行路由切换到
SeaTunnel。这既让业务开发人员无须感知底层变化，也为每个任务保留了清晰的回退边界。
+
+## 用自然语言生成新的数据通道任务
+
+对于新增任务，平台还探索了另一种入口：通过自然语言描述数据搬运需求。
+
+“把昨天核心商品的销售数据同步到数仓”这样的请求本身并不完整。系统首先结合元数据和历史行为识别可能的源表、目标系统、分区和执行参数，再由 SeaTunnel
Skill 组装相应的 Source、Transform 和 Sink 配置。
+
+在 SQL 生成环节，同程旅行没有直接信任单次模型输出，而是并行生成多个候选方案：
+
+- 推理生成器根据意图和 Schema 直接生成 SQL；
+- ICL 生成器检索历史上成功执行的高质量 SQL 作为示例；
+- 分治生成器把复杂 SQL 拆分成较小的 CTE 子查询；
+- 最终选择器根据语法、抽象语法树复杂度和预估执行代价选出结果。
+
+对于数据转换需求，常规场景会路由到 Filter、Replace、Split 等 SeaTunnel 原生 Transform；当普通 Transform
组合无法表达复杂清洗逻辑时，平台可以生成代码或脚本，编译后作为任务插件加载。流程中仍保留交互式数据预览，让用户在正式执行前确认结果。
+
+在这套设计里，大模型承担任务编写助手的角色，元数据、候选选择、数据预览、结果校验和 SeaTunnel 执行引擎共同构成工程护栏。
+
+## 如何证明迁移前后的数据一致
+
+只有能够可靠比较输出，双跑才真正有价值。同程旅行针对文件输出和数据表输出设计了不同的校验方法。
+
+### 文件类输出校验
+
+把海量文件全部加载到内存中进行全量比较，不仅耗时，也容易引发 OOM。为此，校验流程会对每一行数据做标准化和 Hash 计算，并在外部存储中建立包含
Hash、文件标识和行号的有序索引。
+
+比较分为两层：
+
+1. 先通过 Bloom Filter 快速识别另一份输出中确定不存在的数据；
+2. 再使用双指针对外部排序后的索引做对齐比较；遇到相同 Hash 时继续比较出现次数，避免重复行被掩盖。
+
+发现差异后，可以根据文件标识和行号直接回查原始记录。这种方式避免了高成本的内存全量比较，同时保留了对差异数据的精确定位能力。
+
+### 数据表输出校验
+
+对于关系型数据库和数据仓库，平台利用目标系统自身的 SQL 引擎计算字段特征值。分享中给出的一个示例，是先统一空值表示，再对字段执行
`murmur_hash3_32` 并聚合比较。
+
+这种方法适合高效发现迁移差异。
+
+> **编者注：** 对于关键数据，Hash 对比仍应作为多层校验的一部分，并结合行数、空值分布、关键业务聚合以及抽样明细检查共同使用。
+
+## 持续完善 Connector、并行度和可观测性
+
+执行链路统一后，同程旅行继续从三个方向完善运行底座。
+
+### Connector 可靠性
+
+分享涉及的工作包括 Paimon 类型支持和谓词下推、StarRocks 与 Doris FE 高可用、Kafka 与 RocketMQ
流式消费稳定性、HBase 范围读取以及 HDFS ViewFs
兼容等。这说明统一引擎不仅要跑通最常见的成功路径，还必须处理生产环境中不同数据源和目标端的长尾行为。
+
+### 自适应并行度
+
+任务提交前，平台会探测源端行数、目录大小、Kafka 分区数或 Paimon Bucket
数等特征，再结合集群空闲资源和目标端限流阈值推测并行度，使执行并发同时贴合数据物理分布和目标系统承载能力。
+
+### 运行可观测性
+
+平台补充了 Checkpoint 端到端耗时、状态大小和失败频率等指标，同时监控 Master 选举耗时、切换频率和 Active Master
健康状态。借助这些信息，运维人员可以更快地区分任务瓶颈、引擎问题和集群控制面异常。
+
+## 这套架构带来的启示
+
+同程旅行的实践并不只是把一种工具替换成另一种工具，更重要的是建立了一套可控的迁移方法：
+
+- 保留原有业务入口，在底层替换执行引擎；
+- 由平台集中完成任务转换，不把迁移成本分摊给每个业务团队；
+- 新老引擎双跑，验证通过后才切换生产路由；
+- 把输出校验设置为切换门禁，而不是迁移结束后的补充检查；
+- 用 AI 降低任务编写成本，但把元数据、预览、候选选择和校验保留在控制闭环中；
+- 把 Connector 覆盖、自适应执行和可观测性建设成平台能力。
+
+同程旅行下一阶段的规划包括 Kubernetes 原生弹性 Worker、Job
日志远程集中存储、自然语言生成同步任务、异常根因自动定位以及并行度自动调优。这些能力将推动数据通道从人工维护的
Pipeline，逐步演进为更具自管理能力的数据集成平台。
+
+## 讲师信息
+
+周晓晨目前负责同程旅行数据平台相关工作，并在 Apache SeaTunnel Meetup
上分享了这套架构。本文是基于公开分享事实独立撰写的技术摘要，采用了新的叙事结构，未复制原文图片或大段文字。
+
+- [周晓晨的 GitHub 主页](https://github.com/xiaochen-zhou)
