LuciferYang edited a comment on pull request #30026:
URL: https://github.com/apache/spark/pull/30026#issuecomment-709061277
Address 724eee6 add a simple microbenchmark
**With this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: insert table with 10000 rows, one partition column, 10
partitions
Stopped after 2 iterations, 2134 ms
Running case: insert table with 10000 rows, one partition column, 50
partitions
Stopped after 2 iterations, 7206 ms
Running case: insert table with 10000 rows, one partition column, 100
partitions
Stopped after 2 iterations, 9105 ms
Running case: insert table with 10000 rows, one partition column, 200
partitions
Stopped after 2 iterations, 14778 ms
Running case: insert table with 10000 rows, one partition column, 500
partitions
Stopped after 2 iterations, 42992 ms
Running case: insert table with 10000 rows, two partition columns, 10
partitions
Stopped after 2 iterations, 2331 ms
Running case: insert table with 10000 rows, two partition columns, 50
partitions
Stopped after 2 iterations, 6768 ms
Running case: insert table with 10000 rows, two partition columns, 100
partitions
Stopped after 2 iterations, 9274 ms
Running case: insert table with 10000 rows, two partition columns, 200
partitions
Stopped after 2 iterations, 17487 ms
Running case: insert table with 10000 rows, two partition columns, 500
partitions
Stopped after 2 iterations, 54044 ms
Running case: insert table with 10000 rows, three partition columns, 10
partitions
Stopped after 2 iterations, 2368 ms
Running case: insert table with 10000 rows, three partition columns, 50
partitions
Stopped after 2 iterations, 5538 ms
Running case: insert table with 10000 rows, three partition columns, 100
partitions
Stopped after 2 iterations, 11687 ms
Running case: insert table with 10000 rows, three partition columns, 200
partitions
Stopped after 2 iterations, 22371 ms
Running case: insert table with 10000 rows, three partition columns, 500
partitions
Stopped after 2 iterations, 55828 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
insert table with 10000 rows, one partition column, 10 partitions
922 1067 206 0.0 92182.2 1.0X
insert table with 10000 rows, one partition column, 50 partitions
3265 3603 478 0.0 326535.4 0.3X
insert table with 10000 rows, one partition column, 100 partitions
4390 4553 230 0.0 438988.3 0.2X
insert table with 10000 rows, one partition column, 200 partitions
6585 7389 1137 0.0 658477.7 0.1X
insert table with 10000 rows, one partition column, 500 partitions
20220 21496 1805 0.0 2022011.5 0.0X
insert table with 10000 rows, two partition columns, 10 partitions
1114 1166 72 0.0 111432.2 0.8X
insert table with 10000 rows, two partition columns, 50 partitions
2467 3384 1297 0.0 246670.3 0.4X
insert table with 10000 rows, two partition columns, 100 partitions
4559 4637 110 0.0 455904.3 0.2X
insert table with 10000 rows, two partition columns, 200 partitions
8631 8744 159 0.0 863130.8 0.1X
insert table with 10000 rows, two partition columns, 500 partitions
23806 27022 1498 0.0 2380574.6 0.0X
insert table with 10000 rows, three partition columns, 10 partitions
1096 1184 125 0.0 109639.4 0.8X
insert table with 10000 rows, three partition columns, 50 partitions
2694 2769 107 0.0 269364.4 0.3X
insert table with 10000 rows, three partition columns, 100 partitions
5701 5844 202 0.0 570137.3 0.2X
insert table with 10000 rows, three partition columns, 200 partitions
11105 11186 115 0.0 1110452.3 0.1X
insert table with 10000 rows, three partition columns, 500 partitions
26978 27914 1324 0.0 2697786.6 0.0X
```
**Without this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: insert table with 10000 rows, one partition column, 10
partitions
Stopped after 3 iterations, 2356 ms
Running case: insert table with 10000 rows, one partition column, 50
partitions
Stopped after 2 iterations, 6328 ms
Running case: insert table with 10000 rows, one partition column, 100
partitions
Stopped after 2 iterations, 8942 ms
Running case: insert table with 10000 rows, one partition column, 200
partitions
Stopped after 2 iterations, 17401 ms
Running case: insert table with 10000 rows, one partition column, 500
partitions
Stopped after 2 iterations, 43009 ms
Running case: insert table with 10000 rows, two partition columns, 10
partitions
Stopped after 3 iterations, 2024 ms
Running case: insert table with 10000 rows, two partition columns, 50
partitions
Stopped after 2 iterations, 4862 ms
Running case: insert table with 10000 rows, two partition columns, 100
partitions
Stopped after 2 iterations, 11229 ms
Running case: insert table with 10000 rows, two partition columns, 200
partitions
Stopped after 2 iterations, 18244 ms
Running case: insert table with 10000 rows, two partition columns, 500
partitions
Stopped after 2 iterations, 54922 ms
Running case: insert table with 10000 rows, three partition columns, 10
partitions
Stopped after 3 iterations, 2173 ms
Running case: insert table with 10000 rows, three partition columns, 50
partitions
Stopped after 2 iterations, 5660 ms
Running case: insert table with 10000 rows, three partition columns, 100
partitions
Stopped after 2 iterations, 14925 ms
Running case: insert table with 10000 rows, three partition columns, 200
partitions
Stopped after 2 iterations, 28378 ms
Running case: insert table with 10000 rows, three partition columns, 500
partitions
Stopped after 2 iterations, 59941 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
insert table with 10000 rows, one partition column, 10 partitions
715 785 98 0.0 71450.9 1.0X
insert table with 10000 rows, one partition column, 50 partitions
3132 3164 46 0.0 313201.2 0.2X
insert table with 10000 rows, one partition column, 100 partitions
4375 4471 136 0.0 437546.1 0.2X
insert table with 10000 rows, one partition column, 200 partitions
7968 8701 1035 0.0 796846.9 0.1X
insert table with 10000 rows, one partition column, 500 partitions
19208 21505 NaN 0.0 1920778.4 0.0X
insert table with 10000 rows, two partition columns, 10 partitions
600 675 66 0.0 60027.5 1.2X
insert table with 10000 rows, two partition columns, 50 partitions
2372 2431 83 0.0 237244.2 0.3X
insert table with 10000 rows, two partition columns, 100 partitions
5385 5615 325 0.0 538471.6 0.1X
insert table with 10000 rows, two partition columns, 200 partitions
8496 9122 885 0.0 849591.8 0.1X
insert table with 10000 rows, two partition columns, 500 partitions
25747 27461 2424 0.0 2574722.7 0.0X
insert table with 10000 rows, three partition columns, 10 partitions
687 725 35 0.0 68748.8 1.0X
insert table with 10000 rows, three partition columns, 50 partitions
2757 2830 104 0.0 275692.0 0.3X
insert table with 10000 rows, three partition columns, 100 partitions
6336 7463 1594 0.0 633568.3 0.1X
insert table with 10000 rows, three partition columns, 200 partitions
14046 14189 202 0.0 1404645.4 0.1X
insert table with 10000 rows, three partition columns, 500 partitions
26749 29971 1520 0.0 2674929.0 0.0X
```
cc @cloud-fan seems no essential difference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
