qlong commented on code in PR #54394:
URL: https://github.com/apache/spark/pull/54394#discussion_r3400568951
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:
##########
@@ -333,6 +335,20 @@ class V2ExpressionBuilder(e: Expression, isPredicate:
Boolean = false) extends L
case _ =>
None
}
+ case v: VariantGet
+ if v.path.foldable
+ && v.child.isInstanceOf[Attribute] =>
+ val colName = v.child.asInstanceOf[Attribute].name
+ val path = v.path.eval().toString
+ val typeName = v.dataType.catalogString
+ val colRef = FieldReference.column(colName)
+ val pathLit = LiteralValue(UTF8String.fromString(path), StringType)
+ val typeLit = LiteralValue(UTF8String.fromString(typeName), StringType)
+ val canonName = v.prettyName
+ Some(new UserDefinedScalarFunc(
Review Comment:
Thanks for review. This is a good call out, dedicated connector class
instead of UDSF is much cleaner and more correct. I was using UDSF to avoid
compile time dependence between iceberg and spark but i agree it is
anti-pattern. Added VariantGet by following GetArrayItem. Also fix timezone and
failOnError.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
