[
https://issues.apache.org/jira/browse/SPARK-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251308#comment-15251308
]
Vishnu Prasad commented on SPARK-14653:
---------------------------------------
Hey I've just written an example of a UDT for DenseVector I'm assumming this is
what i should do to all the classes in mllib-local that uses NumericParser and
jackson.
If what I'm doing is right which package am I supposed to put the UDT's in.
{code:borderStyle=solid}
class DenseVectorUDT extends UserDefinedType[DenseVector] {
override def sqlType: DataType = ArrayType(DoubleType, containsNull = false)
override def serialize(features: DenseVector): ArrayData = {
new GenericArrayData(features.values)
}
override def userClass: Class[DenseVector] = classOf[DenseVector]
override def deserialize(datum: Any): Vector = {
datum match {
case jValue: JValue => (jValue \ "type").extract[Int] match {
case 0 => // sparse
val size = (jValue \ "size").extract[Int]
val indices = (jValue \ "indices").extract[Seq[Int]].toArray
val values = (jValue \ "values").extract[Seq[Double]].toArray
Vectors.sparse(size, indices, values).toDense
case 1 => // dense
val values = (jValue \ "values").extract[Seq[Double]].toArray
Vectors.dense(values)
case _ =>
throw new IllegalArgumentException(s"Cannot parse $json into a
vector.")
}
}
}
}
{code}
> Remove NumericParser and jackson dependency from mllib-local
> ------------------------------------------------------------
>
> Key: SPARK-14653
> URL: https://issues.apache.org/jira/browse/SPARK-14653
> Project: Spark
> Issue Type: Sub-task
> Components: Build, ML
> Reporter: Xiangrui Meng
>
> After SPARK-14549, we should remove NumericParser and jackson from
> mllib-local, which were introduced very earlier and now replaced by UDTs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
