JingsongLi commented on code in PR #9489:
URL: https://github.com/apache/paimon/pull/9489#discussion_r3892339010
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/MergeIntoPaimonDataEvolutionTable.scala:
##########
@@ -150,25 +151,65 @@ case class MergeIntoPaimonDataEvolutionTable(
*
* without any extra shuffle, join, or sort.
*/
- private lazy val isSelfMergeOnRowId: Boolean = {
+ private case class SelfMergeSpec(residualCondition: Option[Expression])
+
+ private lazy val sameSourceAndTargetTable: Boolean = {
Review Comment:
[P1] Restrict the shortcut to semantically identical source scans
This name-only check can silently ignore source projections because
`PaimonRelation.unapply` also accepts `Project(_, PaimonV2Relation(...))`. For
example, `USING (SELECT _ROW_ID, b + 1 AS b FROM target) source` with `ON
target._ROW_ID = source._ROW_ID AND target.dt = 'p1'` enters the new shortcut;
the shortcut removes the source plan and maps `source.b` to `target.b` by name,
so a value of 10 remains 10 instead of becoming 11. I reproduced this on the
current head, while changing the residual condition to non-partition `target.id
= 1` forces the regular merge path and correctly produces 11. Please only
enable this shortcut for a direct scan of the same table, or for a verified
passthrough projection; otherwise fall back to the regular merge. The Spark 4.0
copy needs the same guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
