LuciferYang commented on code in PR #37843:
URL: https://github.com/apache/spark/pull/37843#discussion_r967687342
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/metric/CustomSumMetric.java:
##########
@@ -31,6 +31,10 @@
public abstract class CustomSumMetric implements CustomMetric {
@Override
public String aggregateTaskMetrics(long[] taskMetrics) {
Review Comment:
Test
```
public static long sumUseStreamApi(long[] input) {
return Arrays.stream(input).sum();
}
```
vs
```
public static long sumsUseLoopApi(long[] input) {
long sum = 0L;
for (long l : input) {
sum += l;
}
return sum;
}
```
with input `1,2,5,10,20,50,100,500,1000,10000`
Java 8
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
1 25.0 40.1 1.0X
Use Loop api 1 1
0 108.8 9.2 4.4X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
0 23.5 42.6 1.0X
Use Loop api 1 1
0 95.8 10.4 4.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
0 23.3 42.9 1.0X
Use Loop api 1 2
0 83.1 12.0 3.6X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 5
0 21.1 47.4 1.0X
Use Loop api 1 2
0 66.9 15.0 3.2X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 6 7
0 16.6 60.2 1.0X
Use Loop api 3 3
0 38.4 26.0 2.3X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 10 11
0 10.1 98.8 1.0X
Use Loop api 6 7
0 15.6 64.0 1.5X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 29 31
0 3.4 294.5 1.0X
Use Loop api 26 27
0 3.8 261.4 1.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 54 55
1 1.8 542.0 1.0X
Use Loop api 51 52
0 2.0 511.3 1.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 528 532
5 0.2 5279.2 1.0X
Use Loop api 510 511
1 0.2 5103.0 1.0X
```
Java 11
```
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2 3
0 49.7 20.1 1.0X
Use Loop api 1 1
0 115.1 8.7 2.3X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 4
1 36.5 27.4 1.0X
Use Loop api 1 1
0 102.8 9.7 2.8X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 4
1 35.1 28.5 1.0X
Use Loop api 1 1
0 97.3 10.3 2.8X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
0 24.1 41.5 1.0X
Use Loop api 2 2
0 58.4 17.1 2.4X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 7 9
1 13.4 74.6 1.0X
Use Loop api 4 5
0 25.8 38.8 1.9X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 14 16
1 7.3 136.6 1.0X
Use Loop api 6 7
1 17.1 58.4 2.3X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 65 70
2 1.5 649.1 1.0X
Use Loop api 24 27
1 4.1 244.0 2.7X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 122 131
5 0.8 1224.2 1.0X
Use Loop api 46 50
2 2.2 464.1 2.6X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 1313 1340
38 0.1 13127.9 1.0X
Use Loop api 517 545
22 0.2 5166.6 2.5X
```
Java 17
```
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2 2
0 54.2 18.4 1.0X
Use Loop api 1 1
0 108.7 9.2 2.0X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
1 26.8 37.4 1.0X
Use Loop api 2 2
0 58.4 17.1 2.2X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
0 25.3 39.5 1.0X
Use Loop api 2 2
1 53.2 18.8 2.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 5
1 22.2 45.1 1.0X
Use Loop api 2 3
0 42.6 23.5 1.9X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 6 7
1 17.0 58.8 1.0X
Use Loop api 4 5
1 27.1 37.0 1.6X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 9 10
1 10.7 93.0 1.0X
Use Loop api 7 8
1 13.7 72.9 1.3X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 29 31
2 3.4 290.9 1.0X
Use Loop api 27 28
1 3.7 270.8 1.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 54 57
2 1.9 539.5 1.0X
Use Loop api 51 54
1 2.0 512.4 1.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test sum with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 541 548
8 0.2 5405.3 1.0X
Use Loop api 548 555
8 0.2 5484.6 1.0X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
