peter-toth commented on code in PR #60008:
URL: https://github.com/apache/spark/pull/60008#discussion_r4444607745
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -832,9 +832,12 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
matched: Seq[NamedExpression],
conf: SQLConf): Expression = {
genExpr.transform {
- case u: UnresolvedAttribute if u.nameParts.size == 1 =>
+ case u @ UnresolvedAttribute(name +: nestedFields) =>
Review Comment:
**Finding 1.** A column referenced inside a lambda body still reads the
value of the query. `AstBuilder.visitLambda` turns every `UnresolvedAttribute`
in a lambda body into an `UnresolvedNamedLambdaVariable`, so this `transform`
never sees `i` in `x -> x * i`. `ResolveLambdaVariables` later turns it back
into an `UnresolvedAttribute`, which resolves against the query, before the
store assignment cast. I ran this on this head and on the base commit, with the
same result:
```sql
CREATE TABLE t (arr ARRAY<INT>, i INT,
g ARRAY<INT> GENERATED ALWAYS AS (transform(arr, x -> x * i))) USING foo;
INSERT INTO t (arr, i) SELECT array(10), 5.7D;
-- stored: [[10],5,[57]], where g should be [50]
INSERT INTO t SELECT * FROM t; -- CHECK_CONSTRAINT_VIOLATION
```
- `transform(arr, x -> x * s.f)` stores `[57]` for a stored `s.f = 5` in the
same way.
- `aggregate(arr, 0, (acc, x) -> acc + x * i)` fails the `INSERT` with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`, because `i` is still a `DOUBLE`
there.
A fix could also replace an `UnresolvedNamedLambdaVariable` here when its
first name part is not an argument of an enclosing `LambdaFunction`. The
transform would track the arguments in scope, like `parentLambdaMap` in
`ResolveLambdaVariables`. If you'd rather keep this PR to struct fields, could
you file a ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
