kerwin-zk opened a new pull request, #7723:
URL: https://github.com/apache/paimon/pull/7723
### Purpose
Follow-up of #7648 (Spark 4.1 module) and a sibling of #7721. After landing
the reverse-shim layout, two of the files under `paimon-spark-4.0/src/main`
only existed as shadows because their compilation unit defined a Scala class
that **`extends BatchWrite`**. Spark 4.1 added a default method
`BatchWrite.commit(WriterCommitMessage[], WriteSummary)` whose `WriteSummary`
parameter type does not exist on Spark 4.0; a class compiled against 4.1 that
mixes in `BatchWrite` carries the inherited `commit(.., WriteSummary)`
signature in its method table, which JVM `ObjectStreamClass.getPrivateMethod`
lazy-links during Spark task serialization and crashes 4.0 with
`ClassNotFoundException: WriteSummary`.
This PR refactors both affected classes into the same **base + per-version
wrapper** pattern:
- `PaimonBatchWrite` (used by V2 writes)
- `FormatTableBatchWrite` (used by `FormatTable` V2 writes — was previously
a `private case class` inside `PaimonFormatTable.scala`)
For each, the body lives in a new abstract base in `paimon-spark-common`
that deliberately does *not* extend `BatchWrite` (renamed protected helpers:
`commitMessages`, `abortMessages`, `createPaimonDataWriterFactory`,
`createFormatTableDataWriterFactory`). Each per-version module
(`paimon-spark3-common`, `paimon-spark4-common`, `paimon-spark-4.0/src/main`)
ships a thin wrapper that mixes in `BatchWrite` and forwards the four
`BatchWrite` methods to the base helpers. Routing happens through two new
`SparkShim` factories so each Spark version's scalac compiles the right
`extends BatchWrite` mixin.
The Spark 4.0 shadow of `PaimonFormatTable.scala` is no longer needed and is
deleted; only the new thin `FormatTableBatchWrite.scala` wrapper remains under
`paimon-spark-4.0/src/main`.
### Linked issue
Follow-up of #7648; sibling of #7721.
### Tests
Local UT on top of latest master:
- `paimon-spark-4.1`: 36 suites / 728 succeeded / 0 failed
- `paimon-spark-4.0`: 36 suites / 726 succeeded / 1 canceled / 1 flaky
failure (`Data Evolution: concurrent merge and small files compact` — a
concurrency test; **passed on re-run**, the failure path is
`ConflictDetection.checkRowIdRangeConflicts`, unrelated to `BatchWrite` mixin)
Spark 3.x `compile` validated against `paimon-spark{-common,3-common,-3.5}`.
### API and Format
No new public API. Two internal factories added to
`org.apache.spark.sql.paimon.shims.SparkShim`:
- `createPaimonBatchWrite(table, writeSchema, dataSchema,
overwritePartitions, copyOnWriteScan)`
- `createFormatTableBatchWrite(table, overwriteDynamic, overwritePartitions,
writeSchema)`
### Documentation
No user-facing changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
