Maxim Gekk created SPARK-31553: ---------------------------------- Summary: Wrong result of isInCollection for large collections Key: SPARK-31553 URL: https://issues.apache.org/jira/browse/SPARK-31553 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.0.0, 3.1.0 Reporter: Maxim Gekk
If the size of a collection passed to isInCollection is bigger than spark.sql.optimizer.inSetConversionThreshold, the method can return wrong results for some inputs. For example: {code:scala} val set = (0 to 20).map(_.toString).toSet val data = Seq("1").toDF("x") println(set.contains("1")) data.select($"x".isInCollection(set).as("isInCollection")).show() {code} {code} true +--------------+ |isInCollection| +--------------+ | false| +--------------+ {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org