Ming Beckwith created SPARK-17913:

             Summary: Filter/join expressions can return incorrect results when 
comparing strings to longs
                 Key: SPARK-17913
                 URL: https://issues.apache.org/jira/browse/SPARK-17913
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0, 1.6.2
            Reporter: Ming Beckwith


  case class E(subject: Long, predicate: String, objectNode: String)

  def test(sc: SparkContext) = {
    val sqlContext: SQLContext = new SQLContext(sc)
    import sqlContext.implicits._

    val broken = List(
      (19157170390056969L, "right", 19157170390056969L),
      (19157170390056973L, "wrong", 19157170390056971L),
      (19157190254313477L, "wrong", 19157190254313475L),
      (19157180859056133L, "wrong", 19157180859056131L),
      (19157170390056969L, "number", 161),
      (19157170390056971L, "string", "a string"),
      (19157190254313475L, "string", "another string"),
      (19157180859056131L, "number", 191)

    val brokenDF = sc.parallelize(broken).map(b => E(b._1, b._2, 
    val brokenFilter = brokenDF.filter($"subject" === $"objectNode")
    val fixed = brokenDF.filter(brokenDF("subject").cast("string") === 

    println("***** incorrect filter results *****")
    println("***** correct filter results *****")

    println("***** both sides cast to double *****")

Broken filter returns:

|          subject|predicate|       objectNode|
|19157170390056969|    right|19157170390056969|
|19157170390056973|    wrong|19157170390056971|
|19157190254313477|    wrong|19157190254313475|
|19157180859056133|    wrong|19157180859056131|

The physical plan shows both sides of the expression are being cast to Double 
before evaluation. So while comparing numbers to a string number appears to 
work in many cases, when the numbers are sufficiently large and close together 
there is enough loss of precision to cause incorrect results. 

== Physical Plan ==
Filter (cast(subject#0L as double) = cast(objectNode#2 as double))

After casting the left side into strings, the filter returns the expected 

|          subject|predicate|       objectNode|
|19157170390056969|    right|19157170390056969|

Expected behavior in this case is probably to choose one side and cast the 
other (compare string to string or long to long) instead of using a data type 
with less precision. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to