Francisco Orchard created SPARK-25687:
-----------------------------------------
Summary: A dataset can store a column as sequence of Vectors but
not directly vectors
Key: SPARK-25687
URL: https://issues.apache.org/jira/browse/SPARK-25687
Project: Spark
Issue Type: Bug
Components: ML, SQL
Affects Versions: 2.3.1
Reporter: Francisco Orchard
A dataset can store an array of vectors but not a vector. This is inconsistent.
To reproduce:
{
import org.apache.spark.sql.Row
import org.apache.spark.ml.linalg.\{Vectors, DenseVector, Vector}
import org.apache.spark.ml.linalg.SQLDataTypes.VectorType
import org.apache.spark.sql.types._
import spark.implicits._
val rdd = sc.parallelize(Seq(Row(Seq(Vectors.dense(Array(1.0,
2.0)).toSparse))))
val arrayOfVectorsDS = spark.createDataFrame(rowRDD= rdd, schema = new
StructType(Array(StructField(name = "value", dataType = ArrayType(elementType =
VectorType))))).as[Seq[Vector]]
// val vectorsDS = arrayOfVectorsDS.flatMap(a => a)
.show
}
If the line before ".show" is uncommented this code will throw the well known
error: error: Unable to find encoder for type stored in a Dataset. Primitive
types (Int, String, etc) and Product types (case classes) are supported by
importing spark.implicits._ Support for serializing other types will be added
in future releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
