[
https://issues.apache.org/jira/browse/SPARK-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-13094:
-------------------------------------
Summary: No encoder implicits for Seq[Primitive] (was: Dataset Aggregators
do not work with complex types)
> No encoder implicits for Seq[Primitive]
> ---------------------------------------
>
> Key: SPARK-13094
> URL: https://issues.apache.org/jira/browse/SPARK-13094
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Deenar Toraskar
>
> Dataset aggregators with complex types fail with unable to find encoder for
> type stored in a Dataset. Though Datasets with these complex types are
> supported.
> {code}
> val arraySum = new Aggregator[Seq[Float], Seq[Float],
> Seq[Float]] with Serializable {
> def zero: Seq[Float] = Nil
> // The initial value.
> def reduce(currentSum: Seq[Float], currentRow: Seq[Float]) =
> sumArray(currentSum, currentRow)
> def merge(sum: Seq[Float], row: Seq[Float]) = sumArray(sum, row)
> def finish(b: Seq[Float]) = b // Return the final result.
> def sumArray(a: Seq[Float], b: Seq[Float]): Seq[Float] = {
> (a, b) match {
> case (Nil, Nil) => Nil
> case (Nil, row) => row
> case (sum, Nil) => sum
> case (sum, row) => (a, b).zipped.map { case (a, b) => a + b }
> }
> }
> }.toColumn
> {code}
> {code}
> <console>:47: error: Unable to find encoder for type stored in a Dataset.
> Primitive types (Int, String, etc) and Product types (case classes) are
> supported by importing sqlContext.implicits._ Support for serializing other
> types will be added in future releases.
> }.toColumn
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
