LuciferYang edited a comment on pull request #30026:
URL: https://github.com/apache/spark/pull/30026#issuecomment-709061277
Address 724eee6 add a simple microbenchmark
**With this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: one partition column, 10 partitions
Stopped after 2 iterations, 2134 ms
Running case: one partition column, 50 partitions
Stopped after 2 iterations, 7206 ms
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 9105 ms
Running case: one partition column, 200 partitions
Stopped after 2 iterations, 14778 ms
Running case: one partition column, 500 partitions
Stopped after 2 iterations, 42992 ms
Running case: two partition columns, 10 partitions
Stopped after 2 iterations, 2331 ms
Running case: two partition columns, 50 partitions
Stopped after 2 iterations, 6768 ms
Running case: two partition columns, 100 partitions
Stopped after 2 iterations, 9274 ms
Running case: two partition columns, 200 partitions
Stopped after 2 iterations, 17487 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 54044 ms
Running case: three partition columns, 10 partitions
Stopped after 2 iterations, 2368 ms
Running case: three partition columns, 50 partitions
Stopped after 2 iterations, 5538 ms
Running case: three partition columns, 100 partitions
Stopped after 2 iterations, 11687 ms
Running case: three partition columns, 200 partitions
Stopped after 2 iterations, 22371 ms
Running case: three partition columns, 500 partitions
Stopped after 2 iterations, 55828 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
one partition column, 10 partitions 922 1067
206 0.0 92182.2 1.0X
one partition column, 50 partitions 3265 3603
478 0.0 326535.4 0.3X
one partition column, 100 partitions 4390 4553
230 0.0 438988.3 0.2X
one partition column, 200 partitions 6585 7389
1137 0.0 658477.7 0.1X
one partition column, 500 partitions 20220 21496
1805 0.0 2022011.5 0.0X
two partition columns, 10 partitions 1114 1166
72 0.0 111432.2 0.8X
two partition columns, 50 partitions 2467 3384
1297 0.0 246670.3 0.4X
two partition columns, 100 partitions 4559 4637
110 0.0 455904.3 0.2X
two partition columns, 200 partitions 8631 8744
159 0.0 863130.8 0.1X
two partition columns, 500 partitions 23806 27022
1498 0.0 2380574.6 0.0X
three partition columns, 10 partitions 1096 1184
125 0.0 109639.4 0.8X
three partition columns, 50 partitions 2694 2769
107 0.0 269364.4 0.3X
three partition columns, 100 partitions 5701 5844
202 0.0 570137.3 0.2X
three partition columns, 200 partitions 11105 11186
115 0.0 1110452.3 0.1X
three partition columns, 500 partitions 26978 27914
1324 0.0 2697786.6 0.0X
```
**Without this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: one partition column, 10 partitions
Stopped after 3 iterations, 2610 ms
Running case: one partition column, 50 partitions
Stopped after 2 iterations, 5651 ms
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 8813 ms
Running case: one partition column, 200 partitions
Stopped after 2 iterations, 16323 ms
Running case: one partition column, 500 partitions
Stopped after 2 iterations, 38269 ms
Running case: two partition columns, 10 partitions
Stopped after 3 iterations, 2730 ms
Running case: two partition columns, 50 partitions
Stopped after 2 iterations, 5574 ms
Running case: two partition columns, 100 partitions
Stopped after 2 iterations, 15787 ms
Running case: two partition columns, 200 partitions
Stopped after 2 iterations, 18852 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 52470 ms
Running case: three partition columns, 10 partitions
Stopped after 3 iterations, 2366 ms
Running case: three partition columns, 50 partitions
Stopped after 2 iterations, 8141 ms
Running case: three partition columns, 100 partitions
Stopped after 2 iterations, 12490 ms
Running case: three partition columns, 200 partitions
Stopped after 2 iterations, 26581 ms
Running case: three partition columns, 500 partitions
Stopped after 2 iterations, 64463 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
one partition column, 10 partitions 789 870
72 0.0 78864.1 1.0X
one partition column, 50 partitions 2697 2826
182 0.0 269734.5 0.3X
one partition column, 100 partitions 4254 4407
216 0.0 425382.7 0.2X
one partition column, 200 partitions 8057 8162
148 0.0 805674.5 0.1X
one partition column, 500 partitions 18896 19135
338 0.0 1889591.7 0.0X
two partition columns, 10 partitions 754 910
241 0.0 75358.7 1.0X
two partition columns, 50 partitions 2701 2787
122 0.0 270120.7 0.3X
two partition columns, 100 partitions 7341 7894
782 0.0 734065.0 0.1X
two partition columns, 200 partitions 9404 9426
32 0.0 940371.7 0.1X
two partition columns, 500 partitions 23720 26235
NaN 0.0 2371963.0 0.0X
three partition columns, 10 partitions 751 789
38 0.0 75076.4 1.1X
three partition columns, 50 partitions 3802 4071
380 0.0 380180.7 0.2X
three partition columns, 100 partitions 6072 6245
245 0.0 607224.0 0.1X
three partition columns, 200 partitions 12874 13291
590 0.0 1287360.6 0.1X
three partition columns, 500 partitions 31451 32232
1104 0.0 3145143.9 0.0X
```
cc @cloud-fan seems no essential difference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
