Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5048#discussion_r26516435
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
---
@@ -102,6 +107,83 @@ sealed trait Matrix extends Serializable {
private[spark] def foreachActive(f: (Int, Int, Double) => Unit)
}
+@DeveloperApi
+class MatrixUDT extends UserDefinedType[Matrix] {
+
+ override def sqlType: StructType = {
+ // type: 0 = sparse, 1 = dense
+ // the dense matrix is built by numRows, numCols, values and
isTransposed, all of which are
+ // set as not nullable. the sparse matrix needs colPtrs and
rowIndices, which are set as
+ // null, while building the dense matrix.
+ StructType(Seq(
+ StructField("type", ByteType, nullable = false),
+ StructField("numRows", IntegerType, nullable = false),
+ StructField("numCols", IntegerType, nullable = false),
+ StructField("colPtrs", ArrayType(IntegerType, containsNull = false),
nullable = true),
+ StructField("rowIndices", ArrayType(IntegerType, containsNull =
false), nullable = true),
+ StructField("values", ArrayType(DoubleType, containsNull = false),
nullable = false),
--- End diff --
Make `values` nullable too. In the future, we might want to add binary
matrix, where only indices are needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
