[
https://issues.apache.org/jira/browse/SPARK-39545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-39545:
-----------------------------
Description:
{{ExpressionSet ++}} method in the master branch a little slower than the
branch-3.3 with Scala-2.13
For example, write a microbenchmark as follows and run with Scala 2.13:
{code:java}
val valuesPerIteration = 100000
val benchmark = new Benchmark("Test ExpressionSet ++ ", valuesPerIteration,
output = output)
val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 ::
Rand(0) :: Nil)
benchmark.addCase("Test ++") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
initialSet ++ setToAddWithSameDeterministicExpression
}
}
benchmark.run() {code}
*branch-3.3 result:*
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 14 16
4 7.2 139.1 1.0X
{code}
*master result :*
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 16 19
5 6.1 163.9 1.0X
{code}
was:ExpressionSet ++ with
> Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the
> performance
> -------------------------------------------------------------------------------------
>
> Key: SPARK-39545
> URL: https://issues.apache.org/jira/browse/SPARK-39545
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Yang Jie
> Priority: Minor
>
> {{ExpressionSet ++}} method in the master branch a little slower than the
> branch-3.3 with Scala-2.13
>
> For example, write a microbenchmark as follows and run with Scala 2.13:
> {code:java}
> val valuesPerIteration = 100000
> val benchmark = new Benchmark("Test ExpressionSet ++ ",
> valuesPerIteration, output = output)
> val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
> val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
> val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 ::
> Rand(0) :: Nil)
> benchmark.addCase("Test ++") { _: Int =>
> for (_ <- 0L until valuesPerIteration) {
> initialSet ++ setToAddWithSameDeterministicExpression
> }
> }
> benchmark.run() {code}
> *branch-3.3 result:*
>
> {code:java}
> OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
> Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
> Test ExpressionSet ++ : Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Test ++ 14 16
> 4 7.2 139.1 1.0X
> {code}
>
> *master result :*
> {code:java}
> OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
> Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
> Test ExpressionSet ++ : Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Test ++ 16 19
> 5 6.1 163.9 1.0X
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
