[
https://issues.apache.org/jira/browse/SPARK-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-6409:
------------------------------------
Assignee: DoingDone9
> It is not necessary that avoid old inteface of hive, because this will make
> some UDAF can not work.
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-6409
> URL: https://issues.apache.org/jira/browse/SPARK-6409
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.2.1
> Reporter: DoingDone9
> Assignee: DoingDone9
> Labels: starter
>
> I run SQL like that
> {code}
> CREATE TEMPORARY FUNCTION test_avg AS
> 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage';
>
> SELECT
> test_avg(1),
> test_avg(substr(value,5))
> FROM src;
> {code}
> then i get a exception
> {code}
> 15/03/19 09:36:45 ERROR CliDriver: org.apache.spark.SparkException: Job
> aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent
> failure: Lost task 0.3 in stage 2.0 (TID 6, HPC-3):
> java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage$AverageAggregationBuffer
> cannot be cast to
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator$AbstractAggregationBuffer
>
> at
> org.apache.spark.sql.hive.HiveUdafFunction.<init>(hiveUdfs.scala:369)
> at
> org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:214)
> at
> org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:188)
> {code}
> i find that GenericUDAFAverage used a deprecated interface AggregationBuffer
> that has been instead by AbstractAggregationBuffer. and spark avoid the old
> interface AggregationBuffer , so GenericUDAFAverage can not work.I think it
> is not necessary.
> code in spark
> {code}
> // Cast required to avoid type inference selecting a deprecated Hive API.
> private val buffer =
>
> function.getNewAggregationBuffer.asInstanceOf[GenericUDAFEvaluator.AbstractAggregationBuffer]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
