zhuxiangyi commented on code in PR #8334:
URL: https://github.com/apache/paimon/pull/8334#discussion_r3795035258
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/MergeIntoPaimonDataEvolutionTable.scala:
##########
@@ -472,7 +472,64 @@ case class MergeIntoPaimonDataEvolutionTable(
AttributeReference(MERGE_DELETED_NAME, BooleanType, nullable = false)()
// Row metadata, _FIRST_ROW_ID added by addFirstRowId, and the delete
marker.
val fixedMergeOutputColumnCount = metadataColumns.size + 2
- val mergeOutput = (updateColumnsSorted ++ metadataColumns ++
rawBlobMarkerAttributes) :+
+
+ // Sub-field-level pruning: for a struct column whose SET only touches
some sub-fields, only the
+ // changed leaves are written (an incremental column-group file containing
the partial struct);
+ // the rest are copied from the target. Falls back to whole-column write
when the changed leaves
+ // cannot be safely determined, so behaviour never regresses.
+ val matchedUpdateActions = matchedActions.collect { case ua: UpdateAction
=> ua }
+ // Gated by data-evolution.nested-field.enabled (default off): when
disabled, no column is
+ // pruned, so every struct column is rewritten whole (behaviour identical
to before this
+ // feature). When enabled, struct columns whose SET only touches some
sub-fields are pruned.
+ val nestedFieldEnabled =
table.coreOptions().dataEvolutionNestedFieldEnabled()
+ val prunedByExprId: Map[ExprId, (Seq[Seq[String]], StructType)] =
+ if (!nestedFieldEnabled) Map.empty
+ else
+ updateColumnsSorted.flatMap {
+ attr =>
+ if (rawBlobUpdateColumns.exists(_.sameRef(attr))) {
+ None
+ } else {
+ attr.dataType match {
+ case st: StructType =>
+ val perAction = matchedUpdateActions.flatMap {
+ ua =>
+ ua.assignments
+ .find(
+ a => isModifiedAssignment(a) &&
assignmentKeyAttribute(a).sameRef(attr))
+ .map(a => changedLeaves(a.value, st, attr))
+ }
+ if (perAction.isEmpty || perAction.exists(_.isEmpty)) {
+ None
+ } else {
+ val union = perAction.flatten.flatten.map(_._1).distinct
Review Comment:
Thanks for catching this. Fixed by canonicalizing the changed-leaf set to
schema declaration order once, so the output struct, writePaths and writeType
all follow the same sequence. Spark 4 copy updated too.
Your repro shape mattered — a single clause with two assignments doesn't
reproduce, because Spark's own assignment alignment rebuilds it in schema order
first. The regression test uses two clauses as you described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
