[
https://issues.apache.org/jira/browse/SPARK-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankur Dave updated SPARK-1552:
------------------------------
Description:
In GraphImpl, mapVertices and outerJoinVertices use a more efficient
implementation when the map function preserves vertex attribute types. This is
implemented by comparing the ClassTags of the old and new vertex attribute
types. However, ClassTags store _erased_ types, so the comparison will return a
false positive for types with different type parameters, such as Option[Int]
and Option[Double].
Thanks to Pierre-Alexandre Fonta for reporting this bug on the [mailing
list|http://apache-spark-user-list.1001560.n3.nabble.com/GraphX-Cast-error-when-comparing-a-vertex-attribute-after-its-type-has-changed-td4119.html].
Demo in the Scala shell:
scala> import scala.reflect.{classTag, ClassTag}
scala> def typesEqual[A: ClassTag, B: ClassTag](a: A, b: B): Boolean =
classTag[A] equals classTag[B]
scala> typesEqual(Some(1), Some(2.0)) // should return false
res2: Boolean = true
We can require richer TypeTags for these methods, or just take a flag from the
caller specifying whether the types are equal.
was:
In GraphImpl, mapVertices and outerJoinVertices use a more efficient
implementation when the map function preserves vertex attribute types. This is
implemented by comparing the ClassTags of the old and new vertex attribute
types. However, ClassTags store _erased_ types, so the comparison will return a
false positive for types with different type parameters, such as Option[Int]
and Option[Double].
Demo in the Scala shell:
scala> import scala.reflect.{classTag, ClassTag}
scala> def typesEqual[A: ClassTag, B: ClassTag](a: A, b: B): Boolean =
classTag[A] equals classTag[B]
scala> typesEqual(Some(1), Some(2.0)) // should return false
res2: Boolean = true
We can require richer TypeTags for these methods, or just take a flag from the
caller specifying whether the types are equal.
> GraphX performs type comparison incorrectly
> -------------------------------------------
>
> Key: SPARK-1552
> URL: https://issues.apache.org/jira/browse/SPARK-1552
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Reporter: Ankur Dave
>
> In GraphImpl, mapVertices and outerJoinVertices use a more efficient
> implementation when the map function preserves vertex attribute types. This
> is implemented by comparing the ClassTags of the old and new vertex attribute
> types. However, ClassTags store _erased_ types, so the comparison will return
> a false positive for types with different type parameters, such as
> Option[Int] and Option[Double].
> Thanks to Pierre-Alexandre Fonta for reporting this bug on the [mailing
> list|http://apache-spark-user-list.1001560.n3.nabble.com/GraphX-Cast-error-when-comparing-a-vertex-attribute-after-its-type-has-changed-td4119.html].
> Demo in the Scala shell:
> scala> import scala.reflect.{classTag, ClassTag}
> scala> def typesEqual[A: ClassTag, B: ClassTag](a: A, b: B): Boolean =
> classTag[A] equals classTag[B]
> scala> typesEqual(Some(1), Some(2.0)) // should return false
> res2: Boolean = true
> We can require richer TypeTags for these methods, or just take a flag from
> the caller specifying whether the types are equal.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
