GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/6179
[SPARK-6747] [SQL] Support List<> as a return type in Hive UDF
This patch supports List<> as a return type in Hive UDF.
We assume an UDF below;
public class UDFToListString extends UDF {
public List evaluate(Object o)
{ return Arrays.asList("xxx", "yyy", "zzz"); }
}
An exception of scala.MatchError is thrown as follows when the UDF used in
the current implementation.
scala.MatchError: interface java.util.List (of class java.lang.Class)
at
org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:174)
at
org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:76)
at
org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:106)
at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:106)
at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:131)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:95)
at
org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:94)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at
scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
...
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark FixBugInHiveInspectors
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6179.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6179
----
commit e553f10c67f129e59c526d6149e60c77348f546a
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-07T16:41:17Z
Support List as a return type in Hive UDF
commit e21ce7e33139b96581dbd37785f99854e60379fd
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-08T02:18:22Z
Add a blank line at the end of UDFToListString
commit 94064169380d129a41574e7ee3ccc994532f71f0
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-14T16:46:44Z
Apply review comments
commit f965c34bcea8efa6b58443ef31c4d857b5feb09e
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-14T18:04:31Z
Fix code-style errors
commit 1c7b9d1aae14de43ca9bf6e3414a45f1104e5043
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-04-17T00:51:14Z
Remove a new type
commit a4887127834fc73063851fd57f095ff212f9b483
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-05-15T04:36:50Z
Add StringToUtf8 to comvert String into UTF8String
commit 2b3f8a1dc9153aab0bd13767d12fc258ce5a1782
Author: Takeshi YAMAMURO <[email protected]>
Date: 2015-05-15T07:18:40Z
Add TOD comments in UDFToListString of HiveUdfSuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
