Cheng Hao created SPARK-3501:
--------------------------------
Summary: Hive SimpleUDF will create duplicated type cast which
cause exception in constant folding
Key: SPARK-3501
URL: https://issues.apache.org/jira/browse/SPARK-3501
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Cheng Hao
Priority: Minor
When do the query like:
select datediff(cast(value as timestamp), cast('2002-03-21 00:00:00' as
timestamp)) from src;
SparkSQL will raise exception:
{panel}
[info] - Cast Timestamp to Timestamp in UDF *** FAILED ***
[info] scala.MatchError: TimestampType (of class
org.apache.spark.sql.catalyst.types.TimestampType$)
[info] at
org.apache.spark.sql.catalyst.expressions.Cast.castToTimestamp(Cast.scala:77)
[info] at
org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:251)
[info] at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:247)
[info] at org.apache.spark.sql.catalyst.expressions.Cast.eval(Cast.scala:263)
[info] at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:217)
[info] at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:210)
[info] at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:144)
[info] at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$2.apply(TreeNode.scala:180)
[info] at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
[info] at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
{panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
