LuciferYang commented on code in PR #37843:
URL: https://github.com/apache/spark/pull/37843#discussion_r967687501
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/metric/CustomAvgMetric.java:
##########
@@ -33,7 +33,11 @@ public abstract class CustomAvgMetric implements
CustomMetric {
@Override
public String aggregateTaskMetrics(long[] taskMetrics) {
if (taskMetrics.length > 0) {
- double average = ((double)Arrays.stream(taskMetrics).sum()) /
taskMetrics.length;
+ long sum = 0L;
Review Comment:
Test
```
public static double avgUseStreamApi(long[] input) {
if (input.length > 0) {
return Arrays.stream(input).sum() / (double) input.length;
} else {
return 0D;
}
}
```
vs
```
public static double avgUseLoopApi(long[] input) {
if (input.length > 0) {
long sum = 0L;
for (long taskMetric : input) {
sum += taskMetric;
}
return (double)sum / input.length;
} else {
return 0D;
}
}
```
with input `1,2,5,10,20,50,100,500,1000,10000`
Java 8
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
0 26.8 37.4 1.0X
Use Loop api 1 1
0 83.1 12.0 3.1X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
0 24.8 40.3 1.0X
Use Loop api 1 1
0 163.7 6.1 6.6X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 5
0 18.8 53.3 1.0X
Use Loop api 1 1
0 163.6 6.1 8.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 8 8
0 12.1 82.3 1.0X
Use Loop api 1 1
0 165.3 6.1 13.6X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 17 18
0 5.7 174.5 1.0X
Use Loop api 1 1
0 163.7 6.1 28.6X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 31 31
1 3.2 308.9 1.0X
Use Loop api 1 1
0 163.6 6.1 50.5X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 138 138
0 0.7 1382.6 1.0X
Use Loop api 1 1
0 163.6 6.1 226.2X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 272 273
1 0.4 2721.2 1.0X
Use Loop api 1 1
0 163.7 6.1 445.4X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test avg with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2691 2694
4 0.0 26914.7 1.0X
Use Loop api 1 1
0 164.2 6.1 4419.4X
```
Java 11
```
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 3
0 39.4 25.4 1.0X
Use Loop api 1 2
0 70.3 14.2 1.8X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 4
1 34.2 29.2 1.0X
Use Loop api 1 2
0 66.7 15.0 1.9X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 4
1 30.1 33.2 1.0X
Use Loop api 1 2
1 121.5 8.2 4.0X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
1 27.9 35.9 1.0X
Use Loop api 1 2
0 83.3 12.0 3.0X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 6
1 21.0 47.6 1.0X
Use Loop api 2 3
0 44.2 22.6 2.1X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 9 10
1 11.1 90.0 1.0X
Use Loop api 6 7
1 16.7 60.0 1.5X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 29 31
2 3.4 294.4 1.0X
Use Loop api 26 28
2 3.9 258.9 1.1X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 53 56
2 1.9 530.6 1.0X
Use Loop api 48 53
2 2.1 482.2 1.1X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test avg with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 537 541
3 0.2 5366.0 1.0X
Use Loop api 531 537
10 0.2 5305.1 1.0X
```
Java 17
```
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 4
0 30.3 33.0 1.0X
Use Loop api 2 2
0 65.3 15.3 2.2X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
0 27.1 36.9 1.0X
Use Loop api 2 2
0 60.3 16.6 2.2X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
1 25.7 38.9 1.0X
Use Loop api 2 2
1 62.1 16.1 2.4X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
1 22.4 44.6 1.0X
Use Loop api 2 3
1 50.1 20.0 2.2X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 7 8
1 15.0 66.6 1.0X
Use Loop api 3 4
1 30.2 33.1 2.0X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 9 10
1 10.9 91.5 1.0X
Use Loop api 7 8
0 14.4 69.4 1.3X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 28 30
2 3.6 278.9 1.0X
Use Loop api 26 28
1 3.9 258.7 1.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 52 55
2 1.9 519.7 1.0X
Use Loop api 49 53
2 2.0 493.8 1.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Test avg with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 528 534
6 0.2 5275.8 1.0X
Use Loop api 523 526
2 0.2 5232.3 1.0X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
