JingsongLi commented on code in PR #9415:
URL: https://github.com/apache/paimon/pull/9415#discussion_r3930388775
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonAnalysis.scala:
##########
@@ -110,7 +110,16 @@ class PaimonAnalysis(session: SparkSession) extends
Rule[LogicalPlan] {
table: DataSourceV2Relation,
options: Options,
mergeSchemaEnabled: Boolean): LogicalPlan = {
- val query = stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val queryWithoutMarker =
stripHiveDynamicPartitionMarker(v2WriteCommand.query)
+ val query =
+ if (
+ v2WriteCommand.isByName &&
Review Comment:
[P1] Do not gate column-list semantics on Spark\x27s version-dependent
`isByName`
The current head fails its own new `Paimon Insert: column list resolves
unequal nested structs positionally` test on both Spark 3.2 and 3.3 with Scala
2.13. The CI stack reaches `resolveColumnsByPosition` at `arr.element` and
rejects the 2-field source vs 3-field target; Spark 3.4/3.5 pass. On the older
plans this `isByName` gate does not select the column-list rewrite/merge-schema
path, even though the parser marker identifies the same SQL syntax.
Please derive the effective column-list mode from the explicit
`COLUMN_LIST_WRITE` marker independently of Spark\x27s
`V2WriteCommand.isByName`, and use that effective mode consistently for nested
renaming, expected-output calculation, and missing-field resolution. The
existing test already exposes the regression; it needs to pass under every
supported Spark 3 profile before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
