MaxGekk commented on pull request #28328:
URL: https://github.com/apache/spark/pull/28328#issuecomment-619904603
I have update PR's description and added a column w/o optimization. I got
the numbers by running the code:
```scala
test("isInCollection benchmark") {
def testExplainTime(collectionSize: Int) = {
val df = spark.range(10).withColumn("id2", col("id") + 1)
val list = Range(0, collectionSize).toList
val startTime = System.currentTimeMillis()
df.where(col("id").isInCollection(list)).where(col("id2").isInCollection(list)).explain()
val elapsedTime = System.currentTimeMillis() - startTime
println(s"cost time: ${elapsedTime}ms")
}
withSQLConf(SQLConf.OPTIMIZER_INSET_CONVERSION_THRESHOLD.key ->
"100000000") {
testExplainTime(1)
testExplainTime(5)
testExplainTime(10)
testExplainTime(100)
testExplainTime(1000)
testExplainTime(10000)
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
