GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/5395
[SPARK-6747][SQL] Support List<> as a return type in Hive UDF
This patch supports List<> as a return type in Hive UDF.
We assume an UDF below;
public class UDFToListString extends UDF {
public List<String> evaluate(Object o)
{ return Arrays.asList("xxx", "yyy", "zzz"); }
}
An exception of scala.MatchError is thrown as follows when the UDF used in
the current implementation.
scala.MatchError: interface java.util.List (of class java.lang.Class)
at
org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:174)
at
org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:76)
at
org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:106)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:106)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:131)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:95)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:94)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at
scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
...
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark FixBugInHiveInspectors
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5395.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5395
----
commit bd165b9f72ef8c30508423688c38b8bacc734884
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-07T16:41:17Z
Support List as a return type in Hive UDF
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
