LiuHao0606 opened a new issue, #13211:
URL: https://github.com/apache/seatunnel/issues/13211

   ### Search before asking
   
   Yes, I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement. The closest ones are #9207 (upgrade 
to Paimon 1.x, closed) and #9530 (update to the latest version, closed), both 
from previous upgrade rounds.
   
   ### Description
   
   Apache Paimon 2.0.0 has been officially released: 
https://paimon.apache.org/releases/2.0.0
   
   Paimon 2.0 is a major release that brings, among other things:
   - New data types for multimodal workloads: `BLOB`, `VECTOR`, `VARIANT`
   - Data Evolution / Row Tracking (add or backfill columns without rewriting 
existing files)
   - Global index framework (BTree / Bitmap / vector / full-text)
   - REST catalog, PyPaimon, and other ecosystem improvements
   
   However, the SeaTunnel Paimon connector still pins the Paimon runtime to 
**1.1.1** (see `seatunnel-connectors-v2/connector-paimon/pom.xml`:
   `<paimon.version>1.1.1</paimon.version>`). This means:
   
   1. `connector-paimon` source/sink are not qualified against Paimon 2.0 
warehouses.
   2. Tables created with Paimon 2.0 that use new features/data types cannot be 
read or written by the connector's bundled 1.1.1 runtime.
   3. Users building a lakehouse on Paimon 2.0 cannot use SeaTunnel as their 
data integration engine against those tables.
   
   **Request:** upgrade the Paimon dependency of `connector-paimon` (source and 
sink) from 1.1.1 to 2.0.x, so SeaTunnel can read from and write to Paimon 2.0 
tables. If feasible, please keep backward compatibility with existing 1.x 
tables.
   
   ### Usage Scenario
   
   We run a streaming lakehouse built on EMR (HDFS HA) + StarRocks + Apache 
Fluss + Paimon + SeaTunnel (SeaTunnel Zeta as the batch/streaming integration 
engine). New Paimon tables are being created on Paimon 2.0 to leverage its 
improvements, and we would like to keep using SeaTunnel jobs (e.g. database CDC 
/ batch sync -> Paimon sink) against those 2.0 tables. With the connector stuck 
on the 1.1.1 runtime, these pipelines cannot target 2.0 tables, forcing us to 
either downgrade our Paimon warehouse or look for another ingestion tool.
   
   ### Related issues
   
   - https://github.com/apache/seatunnel/issues/9207 (previous upgrade to 
Paimon 1.x, closed)
   - https://github.com/apache/seatunnel/issues/9530 (update to the latest 
version, closed)
   - https://github.com/apache/seatunnel/issues/13140 (Paimon REST catalog 
support, open - REST catalog is also a 2.x-era feature, a version upgrade would 
help that effort too)
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   
   ---
   
   ### Chinese summary (补充说明)
   
   Apache Paimon 2.0.0 已正式发布(多模态类型 BLOB/VECTOR/VARIANT、Data Evolution、全局索引、REST 
Catalog 等),但 SeaTunnel 的 `connector-paimon` 目前仍固定依赖 Paimon 1.1.1,无法读写 2.0 
表。希望社区将 Paimon connector 升级到 2.0.x,以支持基于 Paimon 2.0 的湖仓读写场景(我们的流式湖仓 EMR + 
StarRocks + Fluss + Paimon + SeaTunnel 正受此限制)。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to