dongjoon-hyun commented on PR #60035:
URL: https://github.com/apache/spark/pull/60035#issuecomment-6596701093
Thank you for the detailed review, @LuciferYang. I updated the PR in
17578e8e22f.
- Following your suggestion, `TableOutputResolver` now projects the stored
values first, and the analyzer resolves the generation expressions on top of
that projection. `resolveGenerationExprReferences` is removed, including the
struct field rewrite from #60008.
- Each stored value is computed once per row, also in a lambda function.
Your example takes 40 ms locally, versus 2286 ms with the previous commit and
46 ms on master.
- Since the analyzer binds the names, lambda variables are matched in the
same way as `ResolveLambdaVariables`, and a column referenced with `IDENTIFIER`
also reads the stored value now.
- Added two tests, one for a DEFAULT column declared after the generated
column and one for `IDENTIFIER('i') * 10`, and updated the PR description and
the migration guide.
- A write with `spark.sql.caseSensitive` set to `true`, to a table whose
generation expression refers to a column in another case, now fails with
`UNRESOLVED_COLUMN` instead of `INTERNAL_ERROR`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
