LuciferYang edited a comment on pull request #30026:
URL: https://github.com/apache/spark/pull/30026#issuecomment-709061277
Address 724eee6 add a simple microbenchmark
EDIT: Address 9edf8ad refactor microbenchmark to test more dynamic
partitions number with JVM options `-Xmx4g -Xms4g`:
**With this pr** the result is :
```
Running benchmark: dynamic insert table benchmark, totalRows = 200000
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 10421 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 49308 ms
Running case: three partition columns, 2000 partitions
Stopped after 2 iterations, 173533 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark, totalRows = 200000: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
one partition column, 100 partitions 4946
5211 374 0.0 24731.0 1.0X
two partition columns, 500 partitions 22929
24654 2440 0.0 114645.4 0.2X
three partition columns, 2000 partitions 82092
86767 2609 0.0 410461.3 0.1X
```
**Without this pr** the result is :
```
Running benchmark: dynamic insert table benchmark, totalRows = 200000
Running case: one partition column, 100 partitions
Stopped after 2 iterations, 10252 ms
Running case: two partition columns, 500 partitions
Stopped after 2 iterations, 45089 ms
Running case: three partition columns, 2000 partitions
Stopped after 2 iterations, 198925 ms
Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7
Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
dynamic insert table benchmark, totalRows = 200000: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------
one partition column, 100 partitions 4840
5126 404 0.0 24201.4 1.0X
two partition columns, 500 partitions 20978
22545 2215 0.0 104892.0 0.2X
three partition columns, 2000 partitions 86858
99463 2043 0.0 434288.8 0.1X
```
cc @cloud-fan seems no essential difference, It looks better than expected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
