peter-toth commented on code in PR #55518:
URL: https://github.com/apache/spark/pull/55518#discussion_r4478689207
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -356,6 +357,32 @@ trait RowLevelWrite extends V2WriteCommand with
SupportsSubquery {
def condition: Expression
def originalTable: NamedRelation
+ /** The projections that read query columns by ordinal. */
+ protected def queryProjections: Seq[ProjectingInternalRow]
+
+ /**
+ * The write reads the operation column at ordinal 0 and other query columns
only through
+ * projections, by ordinal. Any other query column can be pruned with
[[withPrunedQuery]].
+ */
+ override lazy val references: AttributeSet = {
+ val output = query.output.toIndexedSeq
+ val readOrdinals = 0 +: queryProjections.flatMap(_.colOrdinals)
+ AttributeSet(readOrdinals.map { ordinal =>
+ if (!output.isDefinedAt(ordinal)) {
+ throw SparkException.internalError(
Review Comment:
**Finding 34.** `references` is also read when the plan is printed. For a
resolved node, `LogicalPlan.statePrefix` reads `missingInput`
(`LogicalPlan.scala:121`, `QueryPlan.scala:506`), and `missingInput` reads
`references`. So a `RowLevelWrite` whose projections read past the query output
throws this error from `toString` and `treeString`. Those are the plan dumps
one would use to find the rule that left the ordinals stale.
I checked it on a narrow copy-on-write `UPDATE`. After
`write.withPrunedQuery(write.query, _ => write.query.output.length)` the write
is still `resolved`, and both `toString` and `treeString` fail with `A
projection of ReplaceData reads ordinal 6 of a query with 6 columns.`
Keeping `references` total, by skipping ordinals the output does not have,
would still print the plan. The check could then move to `withPrunedQuery` and
`CheckAnalysis`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
