LuciferYang edited a comment on pull request #30026:
URL: https://github.com/apache/spark/pull/30026#issuecomment-709061277
Address 724eee6 add a simple microbenchmark
**With this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: one partition column, 10 partitions
Stopped after 2 iterations, 2134 ms
Running case: one partition column, 50 partitions
Stopped after 2 iterations, 7206 ms
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 9105 ms
Running case: one partition column, 200 partitions
Stopped after 2 iterations, 14778 ms
Running case: one partition column, 500 partitions
Stopped after 2 iterations, 42992 ms
Running case: two partition columns, 10 partitions
Stopped after 2 iterations, 2331 ms
Running case: two partition columns, 50 partitions
Stopped after 2 iterations, 6768 ms
Running case: two partition columns, 100 partitions
Stopped after 2 iterations, 9274 ms
Running case: two partition columns, 200 partitions
Stopped after 2 iterations, 17487 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 54044 ms
Running case: three partition columns, 10 partitions
Stopped after 2 iterations, 2368 ms
Running case: three partition columns, 50 partitions
Stopped after 2 iterations, 5538 ms
Running case: three partition columns, 100 partitions
Stopped after 2 iterations, 11687 ms
Running case: three partition columns, 200 partitions
Stopped after 2 iterations, 22371 ms
Running case: three partition columns, 500 partitions
Stopped after 2 iterations, 55828 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
one partition column, 10 partitions 922 1067
206 0.0 92182.2 1.0X
one partition column, 50 partitions 3265 3603
478 0.0 326535.4 0.3X
one partition column, 100 partitions 4390 4553
230 0.0 438988.3 0.2X
one partition column, 200 partitions 6585 7389
1137 0.0 658477.7 0.1X
one partition column, 500 partitions 20220 21496
1805 0.0 2022011.5 0.0X
two partition columns, 10 partitions 1114 1166
72 0.0 111432.2 0.8X
two partition columns, 50 partitions 2467 3384
1297 0.0 246670.3 0.4X
two partition columns, 100 partitions 4559 4637
110 0.0 455904.3 0.2X
two partition columns, 200 partitions 8631 8744
159 0.0 863130.8 0.1X
two partition columns, 500 partitions 23806 27022
1498 0.0 2380574.6 0.0X
three partition columns, 10 partitions 1096 1184
125 0.0 109639.4 0.8X
three partition columns, 50 partitions 2694 2769
107 0.0 269364.4 0.3X
three partition columns, 100 partitions 5701 5844
202 0.0 570137.3 0.2X
three partition columns, 200 partitions 11105 11186
115 0.0 1110452.3 0.1X
three partition columns, 500 partitions 26978 27914
1324 0.0 2697786.6 0.0X
```
**Without this pr** the result is :
```
Running benchmark: dynamic insert table benchmark
Running case: one partition column, 10 partitions
Stopped after 3 iterations, 2356 ms
Running case: one partition column, 50 partitions
Stopped after 2 iterations, 6328 ms
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 8942 ms
Running case: one partition column, 200 partitions
Stopped after 2 iterations, 17401 ms
Running case: one partition column, 500 partitions
Stopped after 2 iterations, 43009 ms
Running case: two partition columns, 10 partitions
Stopped after 3 iterations, 2024 ms
Running case: two partition columns, 50 partitions
Stopped after 2 iterations, 4862 ms
Running case: two partition columns, 100 partitions
Stopped after 2 iterations, 11229 ms
Running case: two partition columns, 200 partitions
Stopped after 2 iterations, 18244 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 54922 ms
Running case: three partition columns, 10 partitions
Stopped after 3 iterations, 2173 ms
Running case: three partition columns, 50 partitions
Stopped after 2 iterations, 5660 ms
Running case: three partition columns, 100 partitions
Stopped after 2 iterations, 14925 ms
Running case: three partition columns, 200 partitions
Stopped after 2 iterations, 28378 ms
Running case: three partition columns, 500 partitions
Stopped after 2 iterations, 59941 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark: Best
Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------
one partition column, 10 partitions 715 785
98 0.0 71450.9 1.0X
one partition column, 50 partitions 3132 3164
46 0.0 313201.2 0.2X
one partition column, 100 partitions 4375 4471
136 0.0 437546.1 0.2X
one partition column, 200 partitions 7968 8701
1035 0.0 796846.9 0.1X
one partition column, 500 partitions 19208 21505
NaN 0.0 1920778.4 0.0X
two partition columns, 10 partitions 600 675
66 0.0 60027.5 1.2X
two partition columns, 50 partitions 2372 2431
83 0.0 237244.2 0.3X
two partition columns, 100 partitions 5385 5615
325 0.0 538471.6 0.1X
two partition columns, 200 partitions 8496 9122
885 0.0 849591.8 0.1X
two partition columns, 500 partitions 25747 27461
2424 0.0 2574722.7 0.0X
three partition columns, 10 partitions 687 725
35 0.0 68748.8 1.0X
three partition columns, 50 partitions 2757 2830
104 0.0 275692.0 0.3X
three partition columns, 100 partitions 6336 7463
1594 0.0 633568.3 0.1X
three partition columns, 200 partitions 14046 14189
202 0.0 1404645.4 0.1X
three partition columns, 500 partitions 26749 29971
1520 0.0 2674929.0 0.0X
```
cc @cloud-fan seems no essential difference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
