dejankrak-db commented on code in PR #56237:
URL: https://github.com/apache/spark/pull/56237#discussion_r3338707622
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala:
##########
@@ -396,7 +396,13 @@ package object expressions {
// Then this will add ExtractValue("c", ExtractValue("b", a)), and
alias the final
// expression as "c".
val fieldExprs = nestedFields.foldLeft(a: Expression) { (e, name) =>
- ExtractValue(e, Literal(name), resolver)
+ // Extracting a field from a NULL (NullType) base yields NULL (SQL
NULL propagation),
+ // rather than throwing INVALID_EXTRACT_BASE_FIELD_TYPE. A
NullType column can arise
+ // e.g. from schema evolution with missing columns. This is scoped
to dotted
+ // multipart field access (`col.a`); `col['key']`/`col[0]` go
through
+ // UnresolvedExtractValue and still throw.
+ if (e.dataType == NullType) Literal(null, NullType)
Review Comment:
Thanks @cloud-fan! Fixed by making the change consistent across analyzers.
Introduced `ExtractValue.applyOrNull` (resolution-time NULL-propagation;
`extractValue` unchanged) and route both the multipart fold and
`ColumnResolutionHelper`'s `UnresolvedExtractValue` through it, and made
`isExtractable` keep a `NullType` candidate so single-pass `NameScope` reaches
the same NULL result for `col.a`. `col.a` now returns NULL under both analyzers
- added `extract-value-nulltype-single-pass.sql` (dual-run) to lock it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
