zhengxiang378928908-code opened a new pull request, #12862:
URL: https://github.com/apache/seatunnel/pull/12862
### Purpose of this pull request
Closes #12814. Unblocks #12787.
Spark's `DecimalType` stops at precision 38, so any SeaTunnel `DECIMAL(p >
38)` failed at schema creation on Spark 2.4 and 3.3 (`decimal can only support
precision up to 38`), before any row was read. Doris `LARGEINT` as `DECIMAL(39,
0)` (#12787) and MySQL `DECIMAL(65, x)` both hit this.
This PR implements the "Agreed design" in #12814:
- **Top-level fields and fields inside a `ROW`**: carried through Spark as
`STRING` plain decimal text, with `seatunnel_decimal_precision` /
`seatunnel_decimal_scale` metadata on the Spark field, and restored to
`DECIMAL(p, s)` for transforms and sinks. Scale and `NULL` are kept. This
applies both to `InternalRowConverter` and to `SeaTunnelRowConverter` (the
`GenericRow` path that transforms use).
- **Inside `ARRAY` or `MAP`** (element, key or value, at any depth):
rejected at job start with `SPARK-TRANSLATION-01`. The check runs per source
table **before** the multi-table merge renames columns to `column0..N`, so the
error names the original table and full column path, the declared type, the
Spark limit of 38 and the workarounds (leave the column out at the source, cast
it to a string in the source query, or use Zeta/Flink). A `Sql` transform
`CAST` cannot help on Spark because schema creation fails first, so the docs
only list source-side workarounds.
- **No silent changes to values**: a value that would need rounding or
overflow on write, and malformed text, a wrong scale or too many digits on
read, fail with `SPARK-TRANSLATION-02` instead of being changed. Inconsistent
carrier metadata on a Spark field fails with `SPARK-TRANSLATION-03`.
- `InternalRowConverter` no longer falls through from `DECIMAL` into the
`ARRAY` branch when it gets an unexpected value type.
Not in scope: the MySQL `DECIMAL(p > 38)` clamp to `DECIMAL(38, 18)` is a
separate source-side issue (#12841).
### Does this PR introduce _any_ user-facing change?
Yes, on the Spark engine only:
- Before: any job whose schema has `DECIMAL(p > 38)` failed at startup on
Spark.
- After: top-level and `ROW` fields work and keep their exact values.
`DECIMAL(p > 38)` inside `ARRAY`/`MAP` still fails, but at job start with a
clear message.
Docs updated in en/zh: `engines/spark.md`,
`api-design/spark-translation-layer.md`,
`introduction/concepts/incompatible-changes.md`.
### How was this patch tested?
- Unit (`WideDecimalTest`, 20 cases): schema carrier and metadata round
trip; `InternalRowConverter` and `SeaTunnelRowConverter` (through `RowEncoder`)
round trips; scale padding without rounding; rejection of rounding/overflow,
malformed carrier text, inconsistent metadata; preflight for `ARRAY`, `MAP`
key, `MAP` value, `ARRAY` inside `ROW`; narrow decimals in collections still
allowed; multi-table jobs report the original table and column.
- E2E (`FakeSourceToAssertIT`, all engines incl. Spark 2.4 / 3.3):
- `testWideDecimalToAssertSink`: `DECIMAL(39, 0)` signed 128-bit max and
min, `0`, `NULL`, `DECIMAL(50, 10)` with scale, a normal `DECIMAL(10, 2)`
beside them, and the same values inside a `ROW`, through a `Sql` transform; the
sink checks the value class, scale and exact digits.
- `testWideDecimalInMapToAssertSink`: on Spark fails with
`SPARK-TRANSLATION-01` naming `` `c_attrs.value` `` and `DECIMAL(39, 0)`; on
Zeta/Flink passes.
- Fork CI:
https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/39095286760.
Unit tests (JDK 11/17, Ubuntu/Windows) and all `all-connectors-it-*` jobs pass,
including `FakeSourceToAssertIT` on JDK 11 and 17. Unrelated failures on Zeta
only:
`SplitClusterFaultToleranceIT.testStreamJobCancelResolvesWhenWorkerCrashesBeforeCancelAck`
and
`LocalFileIT.testLocalFileBinaryUpdateModeContinuousDiscoveryWithNonRecursiveScan`;
`mysql-cdc-connector-it (17)` hit the 2-hour job timeout.
- Together with #12787: branch `tmp/largeint-with-spark-wide-decimal` merges
both PRs so the Doris Spark containers run against this change. #12787 keeps
the Doris Spark tests enabled and adds a Spark-only Doris check for the
`ARRAY`/`MAP` error (`DorisIT.testNestedLargeintFailsOnSpark`).
### Check list
* [x] No new Jar binary package.
* [x] Documentation updated (en/zh).
* [x] `incompatible-changes.md` updated (en/zh).
* [ ] Not a connector change.
This contribution is made to Apache SeaTunnel.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
