Copilot commented on code in PR #7721:
URL: https://github.com/apache/paimon/pull/7721#discussion_r3192724349
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala:
##########
@@ -65,6 +66,27 @@ trait SparkShim {
partitions: Array[Transform],
properties: JMap[String, String]): Table
+ /**
+ * Returns a `DataSourceV2Relation` like `relation` but with `table`
replaced. Spark 4.1 added
+ * `Option[TimeTravelSpec]` as the 6th field of `DataSourceV2Relation`, so a
`relation.copy(table
+ * = ...)` call compiled against 4.1.1 emits a `copy$default$6` reference
that crashes with
+ * `NoSuchMethodError` on Spark 4.0 runtime. Routing through this factory
lets each per-version
+ * SparkShim implementation generate the matching copy bytecode.
Review Comment:
The Scaladoc for `copyDataSourceV2Relation` says `relation.copy(table =
...)` compiled against Spark 4.1.1 emits a `copy$default$6` reference. Given
the same comment block later cites `copy$default$9` for `TableSpec`, and
because `copy$default$N` depends on the total arity of the case class, the
specific `$6` here is likely inaccurate/misleading. Please either correct the
index (if known) or reword to avoid hard-coding a specific `copy$default$N`
value and just describe the general arity-mismatch problem.
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala:
##########
@@ -65,6 +66,27 @@ trait SparkShim {
partitions: Array[Transform],
properties: JMap[String, String]): Table
+ /**
+ * Returns a `DataSourceV2Relation` like `relation` but with `table`
replaced. Spark 4.1 added
+ * `Option[TimeTravelSpec]` as the 6th field of `DataSourceV2Relation`, so a
`relation.copy(table
+ * = ...)` call compiled against 4.1.1 emits a `copy$default$6` reference
that crashes with
+ * `NoSuchMethodError` on Spark 4.0 runtime. Routing through this factory
lets each per-version
+ * SparkShim implementation generate the matching copy bytecode.
+ */
+ def copyDataSourceV2Relation(
+ relation: DataSourceV2Relation,
+ newTable: Table): DataSourceV2Relation
+
Review Comment:
The new shim factory only covers `relation.copy(table = ...)`, but there are
still `DataSourceV2Relation.copy(...)` call sites in the shared
`paimon-spark-common` code (compiled against Spark 4.1 under the spark4
profile). For example, `MergeIntoPaimonDataEvolutionTable` uses
`touchedFileTargetRelation.copy(output = ...)` and
`touchedFileTargetRelation.copy(targetRelation.table, ...)`; these will have
the same Spark 4.1 ↔ 4.0 binary-incompatibility risk and can still trigger
`NoSuchMethodError` on Spark 4.0 runtimes. Consider routing all
`DataSourceV2Relation.copy(...)` usages in cross-version code through shim
factories (one per call-site or a small set of generic helpers), or replace
them with a stable constructor API if available (e.g.,
`DataSourceV2Relation.create(...)`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
