Github user anabranch commented on the issue:
https://github.com/apache/spark/pull/16138
I don't think this is necessarily my call but you're effectively writing a
hive udf, not a Spark one that depends on this. Writing a Spark UDF would
allow this to work just fine and the expression to be evaluated. Your work
around seems fine in your local version though.
On Fri, Aug 25, 2017 at 12:23 AM, Yann Byron <[email protected]>
wrote:
> @anabranch <https://github.com/anabranch>
>
> I met a failure case when using to_date, like this:
>
> spark-sql> select func(to_date('2017-08-25 12:00:00'));
>
> func is a custom udf which extends org.apache.hadoop.hive.ql.udf.
> generic.GenericUDF.
>
> And this following exception happened:
>
> Error in query: No handler for Hive UDF 'xxx.xxx.func':
java.lang.UnsupportedOperationException: Cannot evaluate expression:
to_date(2017-08-25 12:00:00, None);
>
> I locate this problem. When resolving function func(to_date('2017-08-25
> 12:00:00'))(at this time Spark has already finished the resolving to
func's
> child to_date('2017-08-25 12:00:00')), it will call func's
> FunctionBuilder that is defined in HiveSessionCatalog.makeFunctionBuilder
>
<https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala>
> ).
> In it, the GenericUDF instance will be forced to check input data type.
> During this process, All of the child catalyst expression which are the
> parameters of the subclass of GenericUDF, need to be mapped to
> ObjectInspector(see details: HiveInspector.toInspector
>
<https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala>).
> Inside, the child expression need to be evaluated though eval() firstly,
> if expression doesn't match with Literal.
>
> def toInspector(expr: Expression): ObjectInspector = expr match {
> case Literal(value, XXXType) =>
> //...
> // ideally, we don't test the foldable here(but in optimizer),
however, some of the
> // Hive UDF / UDAF requires its argument to be constant
objectinspector, we do it eagerly.
> case _ if expr.foldable => toInspector(Literal.create(expr.eval(),
expr.dataType))
> }
>
> In my case, the child expression is to_date('2017-08-25 12:00:00') that
> is the instance of ParseToDate
>
<https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala>
> (extends RuntimeReplaceable
>
<https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala>).
> When calling its eval(), we call the Unevaluable
>
<https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala>'s
> eval() that will throw this exception mentioned above.
>
> trait Unevaluable extends Expression {
>
> final override def eval(input: InternalRow = null): Any =
> throw new UnsupportedOperationException(s"Cannot evaluate expression:
$this")
>
> final override protected def doGenCode(ctx: CodegenContext, ev:
ExprCode): ExprCode =
> throw new UnsupportedOperationException(s"Cannot evaluate expression:
$this")
> }
>
> Considering that we barely use the second parameter pattern, I remap
> to_date to ToDate again to fix the problem.
>
> So, does my case relate to this PR?
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16138#issuecomment-324843845>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/ABkQBwa1lkNT81uAWDV3ZRSRz8XqApPHks5sbnaMgaJpZM4LDrz0>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
