nzw921rx opened a new issue, #12763: URL: https://github.com/apache/seatunnel/issues/12763
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description Add support for the Apache Iceberg **v3 spec** (`format-version = 3`) to the Iceberg connector, covering table creation, read, write, schema evolution and the CDC/upsert paths. Current state (2.3.13 and `dev`): - `seatunnel-connectors-v2/connector-iceberg/pom.xml` still pins `<iceberg.version>1.6.1</iceberg.version>`. Iceberg 1.6.1 only accepts table format versions v1/v2, so `format-version = "3"` passed through `iceberg.table.auto-create-props` is rejected by Iceberg's metadata validation. - Tables created as v3 by other engines (Spark, Flink, Iceberg 1.10+) cannot be read or written by the connector, and v3-only types have no mapping to SeaTunnel's `SeaTunnelDataType`. Iceberg 1.10.0 (2025-09-11) is the first release that ships the finalized v3 spec (apache/iceberg#13175, "Spec: Mark version 3 as completed"), and the current Iceberg line is 1.12.0. The v3 spec adds: - New data types: `timestamp_ns` / `timestamptz_ns`, `unknown`, `variant`, `geometry`, `geography` - Column default values (`initial-default` / `write-default`) - Multi-argument transforms for partitioning and sorting - Row lineage tracking (`_row_id`, `_last_updated_sequence_number`) - Binary deletion vectors (position delete files are deprecated in v3) - Table encryption keys ### Usage Scenario A CDC pipeline such as `mysql-cdc → SeaTunnel (Zeta) → Iceberg` wants to write into a v3 table so that deletes/updates are represented as deletion vectors instead of positional delete files (cheaper upserts, less write amplification), or wants to ingest and consume v3 tables produced by Spark/Flink that use `variant`, geospatial types or nanosecond timestamps. Reported in apache/seatunnel#12220. ### Suggested task breakdown 1. **Dependency upgrade** — bump `iceberg.version` in `seatunnel-connectors-v2/connector-iceberg/pom.xml` to a v3-capable release (>= 1.10.0, e.g. 1.12.x) and check the whole transitive set (`iceberg-hive-metastore`, `iceberg-aws`/AWS SDK, `orc-core`, `parquet-avro`, `avro`), shading, and classpath conflicts with the Zeta/Flink/Spark runners. 2. **Table creation / upgrade** — allow `format-version = 3` via `iceberg.table.auto-create-props`, keep v1/v2 as the default, and document the `upgrade-format-version` behavior. Validate the requested version against the embedded runtime and fail fast with a clear message. 3. **Schema conversion** — map the new types both ways in `SchemaUtils` / `SeaTunnelRow` conversion (`timestamp_ns`, `timestamptz_ns`, `variant`, `geometry`, `geography`, `unknown`), and honour `initial-default` / `write-default` when creating and evolving schemas. 4. **Write path** — support binary deletion vectors in the upsert/CDC writer (currently positional/equality deletes), keep `iceberg.table.upsert-mode-enabled` semantics intact, and handle row lineage when it is enabled. 5. **Read path** — make batch and streaming source scans handle v3 metadata: deletion vectors, row lineage columns, and incremental reads on v3 tables. 6. **Partitioning / sorting** — accept multi-argument transforms in partition and sort specs. 7. **Compatibility** — v1/v2 tables and existing configurations must keep working unchanged; any behavior change has to be documented in `docs/en/introduction/concepts/incompatible-changes.md`. 8. **Docs** — update `docs/en/connectors/sink/Iceberg.md`, `docs/zh/connectors/sink/Iceberg.md`, `docs/en/connectors/source/Iceberg.md`, `docs/zh/connectors/source/Iceberg.md` and the connector changelog with the supported format versions, new types and examples. 9. **Tests** — unit tests for schema conversion and option validation, plus e2e coverage in `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e` (create a v3 table, read/write a v3 table, upsert/CDC with deletion vectors, and a v2 regression case). ### Related issues - apache/seatunnel#12220 — "Is Iceberg v3 supported? ( mysql-cdc → SeaTunnel → Iceberg )" ### References - Iceberg spec, version 3: https://iceberg.apache.org/spec/#version-3-extended-types-and-capabilities - Iceberg spec, full v3 change list (Appendix E): https://iceberg.apache.org/spec/#version-3 - Iceberg 1.10.0 release: https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.10.0 ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
