LuciferYang opened a new pull request, #9265:
URL: https://github.com/apache/paimon/pull/9265
### Purpose
Raises the `spark4` profile baseline from 4.1.2 to 4.2.0, adds a
`paimon-spark-4.2` module, and keeps Spark 4.0 and 4.1 working under the new
baseline. Same shape as #7648, which did this for 4.1. Related to #8901.
`paimon-spark-common` and `paimon-spark4-common` are compiled once, against
the newest supported Spark, and the resulting classfiles ship to every 4.x
runtime. Raising the baseline therefore changes bytecode that 4.0 and 4.1 have
to load, and Spark 4.2 made several source-compatible but binary-incompatible
changes: `CatalogManager` became an interface (a 4.2-built call site emits
`invokeinterface` and dies with `IncompatibleClassChangeError` on 4.0/4.1);
case classes gained fields (`CatalogStorageFormat`, `AppendData`,
`DataSourceV2ScanRelation`), so positional patterns and named-argument `copy`
calls no longer link across versions; `RewriteRowLevelCommand`'s
`DELTA_OPERATIONS_WITH_*` constants were renamed; `V2WriteCommand` gained a
`WriteWithSchemaEvolution` supertype; and `DESCRIBE ... PARTITION` moved out of
`DescribeRelation` into its own `DescribeTablePartition` plan (SPARK-39660).
Four mechanisms are used, in order of preference:
1. **Version-neutral construction** — match by type with named accessors
instead of positional patterns, and build placeholders through factory methods
(`CatalogStorageFormat.empty`) rather than arity-sensitive constructors.
2. **`SparkShim` methods** where only the arity differs, so each per-version
module supplies its own call.
3. **`SparkVersionCompat`** reflective accessors where the *signature* is
incompatible. Reflection is immune to the class/interface flip, since only
invoke opcodes carry that distinction.
4. **Same-FQCN forks** in `paimon-spark-4.0` / `-4.1` where a supertype or a
parameter type differs and no accessor can paper over it. Shade writes the
module's own classes first, so the fork wins.
`paimon-spark-ut-4.0` and `-4.1` recompile the shared test sources against
their own baseline. They produce test-jars only and are deliberately left out
of publish and release, unlike `paimon-spark-ut`.
One behaviour fix is included because the baseline bump caused it.
`qualifyIdentifier` has to carry the catalog so Spark 4.2's
`SimpleFunctionRegistryBase.normalizeFuncName` sees a 3-part identifier, but
the same identifier reached the expression builder, which renamed the default
output column of an unaliased v1 function call from `db.udf(...)` to
`catalog.db.udf(...)` on every version from 3.4 up. The builder name now drops
the catalog; the registry key keeps it.
### Tests
CI, plus locally:
- `mvn -Pspark4 clean install` over `paimon-spark-common`,
`paimon-spark4-common`, `paimon-spark-4.0`, `-4.1`, `-4.2` and the three ut
modules — success.
- `mvn -Pspark3 clean compile` over `paimon-spark-common` +
`paimon-spark3-common` — success.
- `spotless:check` clean on all six touched modules.
- `PaimonV1FunctionTest` 13/13 on 4.2, 4.1 and 4.0; `DescribeTableTest` 4/4
on 4.2, 4.1 and 4.0; `SparkVersionCompatTest` 14/14.
- The column-name fix was verified in both directions — reverting it turns
the new case red with `ArraySeq("paimon.test.udf_add2(3, 4)") did not equal
List("test.udf_add2(3, 4)")`.
The new `PaimonV1FunctionTestBase` case asserts the column name rather than
the row values, which is why the existing cases could not catch the regression:
they all compare rows with `checkAnswer` and spell out `AS` wherever a name is
involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
