Github user YannByron commented on the issue:
https://github.com/apache/spark/pull/16138
@anabranch
I met a failure case when using `to_date`, like this:
```
spark-sql> select func(to_date('2017-08-25 12:00:00'));
```
`func` is a custom udf which extends
`org.apache.hadoop.hive.ql.udf.generic.GenericUDF`.
And this following exception happened:
```
Error in query: No handler for Hive UDF 'xxx.xxx.func':
java.lang.UnsupportedOperationException: Cannot evaluate expression:
to_date(2017-08-25 12:00:00, None);
```
I locate this problem. When resolving function `func(to_date('2017-08-25
12:00:00'))`(at this time Spark has already finished the resolving to `func`'s
child `to_date('2017-08-25 12:00:00')`), it will call `func`'s FunctionBuilder
that is defined in
[HiveSessionCatalog.makeFunctionBuilder](https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala)).
In it, the GenericUDF instance will be forced to check input data type.
During this process, All of the child catalyst expression which are the
parameters of the subclass of GenericUDF, need to be mapped to
`ObjectInspector`(see details:
[HiveInspector.toInspector](https://github.com/apache/spark/blob/v2.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala)).
Inside, the child expression need to be evaluated though `eval()` firstly, if
expression doesn't match with `Literal`.
```
def toInspector(expr: Expression): ObjectInspector = expr match {
case Literal(value, XXXType) =>
//...
// ideally, we don't test the foldable here(but in optimizer), however,
some of the
// Hive UDF / UDAF requires its argument to be constant
objectinspector, we do it eagerly.
case _ if expr.foldable => toInspector(Literal.create(expr.eval(),
expr.dataType))
}
```
In my case, the child expression is `to_date('2017-08-25 12:00:00')` that
is the instance of
[ParseToDate](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala)
(extends
[RuntimeReplaceable](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala)).
When calling its `eval()`, we call the
[Unevaluable](https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala)'s
`eval()` that will throw this exception mentioned above.
```
trait Unevaluable extends Expression {
final override def eval(input: InternalRow = null): Any =
throw new UnsupportedOperationException(s"Cannot evaluate expression:
$this")
final override protected def doGenCode(ctx: CodegenContext, ev:
ExprCode): ExprCode =
throw new UnsupportedOperationException(s"Cannot evaluate expression:
$this")
}
```
Considering that we barely use the second parameter `pattern`, I remap
`to_date` to `ToDate` again to fix the problem.
So, does my case relate to this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
