Deenar Toraskar created SPARK-13094:
---------------------------------------
Summary: Dataset Aggregators do not work with complex types
Key: SPARK-13094
URL: https://issues.apache.org/jira/browse/SPARK-13094
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.0
Reporter: Deenar Toraskar
Dataset aggregators with complex types fail with unable to find encoder for
type stored in a Dataset. Though Datasets with these complex types are
supported.
val arraySum = new Aggregator[Seq[Float], Seq[Float],
Seq[Float]] with Serializable {
def zero: Seq[Float] = Nil
// The initial value.
def reduce(currentSum: Seq[Float], currentRow: Seq[Float]) =
sumArray(currentSum, currentRow)
def merge(sum: Seq[Float], row: Seq[Float]) = sumArray(sum, row)
def finish(b: Seq[Float]) = b // Return the final result.
def sumArray(a: Seq[Float], b: Seq[Float]): Seq[Float] = {
(a, b) match {
case (Nil, Nil) => Nil
case (Nil, row) => row
case (sum, Nil) => sum
case (sum, row) => (a, b).zipped.map { case (a, b) => a + b }
}
}
}.toColumn
<console>:47: error: Unable to find encoder for type stored in a Dataset.
Primitive types (Int, String, etc) and Product types (case classes) are
supported by importing sqlContext.implicits._ Support for serializing other
types will be added in future releases.
}.toColumn
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
