LuciferYang commented on code in PR #37843:
URL: https://github.com/apache/spark/pull/37843#discussion_r967653506
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java:
##########
@@ -113,10 +113,28 @@ public OneForOneBlockFetcher(
* @return whether the array contains only shuffle block IDs
*/
private boolean areShuffleBlocksOrChunks(String[] blockIds) {
- if (Arrays.stream(blockIds).anyMatch(blockId ->
!blockId.startsWith(SHUFFLE_BLOCK_PREFIX))) {
+ if (isAnyBlockNotStartWithShuffleBlockPrefix(blockIds)) {
// It comes here because there is a blockId which doesn't have
"shuffle_" prefix so we
// check if all the block ids are shuffle chunk Ids.
- return Arrays.stream(blockIds).allMatch(blockId ->
blockId.startsWith(SHUFFLE_CHUNK_PREFIX));
+ return isAllBlocksStartWithShuffleChunkPrefix(blockIds);
+ }
+ return true;
+ }
+
+ private boolean isAnyBlockNotStartWithShuffleBlockPrefix(String[] blockIds) {
Review Comment:
Java 8
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
1 23.3 43.0 1.0X
Use Loop api 1 1
0 136.7 7.3 5.9X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 6
1 19.2 52.1 1.0X
Use Loop api 2 2
0 61.4 16.3 3.2X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 7 8
1 14.7 67.9 1.0X
Use Loop api 2 2
0 55.9 17.9 3.8X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 10 11
1 9.7 103.5 1.0X
Use Loop api 2 2
0 47.3 21.1 4.9X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 18 19
1 5.6 179.5 1.0X
Use Loop api 3 3
1 32.4 30.9 5.8X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 30 32
1 3.3 303.4 1.0X
Use Loop api 5 6
0 18.7 53.4 5.7X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 116 120
3 0.9 1155.3 1.0X
Use Loop api 16 17
1 6.1 163.9 7.0X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 254 259
5 0.4 2536.7 1.0X
Use Loop api 34 35
1 3.0 337.0 7.5X
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test for AnyMatch with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2612 2660
68 0.0 26115.0 1.0X
Use Loop api 361 365
4 0.3 3609.4 7.2X
```
Java 11
```
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 3
0 35.0 28.6 1.0X
Use Loop api 1 1
0 152.2 6.6 4.4X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 4
0 27.8 36.0 1.0X
Use Loop api 1 1
0 143.6 7.0 5.2X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 5 5
0 19.8 50.5 1.0X
Use Loop api 1 1
0 87.5 11.4 4.4X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 7 8
0 13.8 72.5 1.0X
Use Loop api 2 2
0 57.2 17.5 4.1X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 16 17
0 6.4 156.6 1.0X
Use Loop api 4 4
0 28.0 35.7 4.4X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 29 32
1 3.4 293.9 1.0X
Use Loop api 7 7
0 15.1 66.0 4.5X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 113 124
5 0.9 1130.5 1.0X
Use Loop api 29 29
0 3.4 291.3 3.9X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 267 274
4 0.4 2671.7 1.0X
Use Loop api 62 62
0 1.6 618.6 4.3X
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test for AnyMatch with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2975 2985
15 0.0 29745.1 1.0X
Use Loop api 639 639
0 0.2 6390.8 4.7X
```
Java 17
```
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 3 3
0 35.6 28.1 1.0X
Use Loop api 1 1
0 129.7 7.7 3.6X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 4 5
0 22.8 43.8 1.0X
Use Loop api 1 1
0 90.4 11.1 4.0X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 6 6
0 17.8 56.2 1.0X
Use Loop api 1 1
0 76.7 13.0 4.3X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 8 8
0 12.9 77.5 1.0X
Use Loop api 2 2
0 59.8 16.7 4.6X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 50: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 16 16
0 6.4 155.4 1.0X
Use Loop api 3 3
0 37.3 26.8 5.8X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 100: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 26 27
0 3.8 264.4 1.0X
Use Loop api 4 5
0 22.8 43.9 6.0X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 500: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 103 103
0 1.0 1027.2 1.0X
Use Loop api 17 17
0 5.9 169.3 6.1X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 1000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 225 226
1 0.4 2249.3 1.0X
Use Loop api 35 35
0 2.8 352.3 6.4X
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test for AnyMatch with input size 10000: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Arrays.steam api 2201 2201
1 0.0 22005.4 1.0X
Use Loop api 343 344
0 0.3 3434.7 6.4X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
