anishshri-db commented on PR #41410:
URL: https://github.com/apache/spark/pull/41410#issuecomment-1572473979
Checked the results here and basically it seems like with high overwrite
rate, the perf actually becomes worse
```
================================================================================================
put rows
================================================================================================
OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1037-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
putting 10000 rows (10000 rows to overwrite - rate 100): Best Time(ms)
Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
In-memory 8
9 1 1.3 769.8 1.0X
RocksDB (trackTotalNumberOfRows: true) 88
90 1 0.1 8804.6 0.1X
RocksDB (trackTotalNumberOfRows: false) 19
20 0 0.5 1901.4 0.4X
OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1037-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
putting 10000 rows (5000 rows to overwrite - rate 50): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------
In-memory 8
9 1 1.3 775.3 1.0X
RocksDB (trackTotalNumberOfRows: true) 63
65 1 0.2 6310.2 0.1X
RocksDB (trackTotalNumberOfRows: false) 20
21 0 0.5 1982.8 0.4X
OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1037-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
putting 10000 rows (1000 rows to overwrite - rate 10): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------
In-memory 7
8 1 1.4 727.4 1.0X
RocksDB (trackTotalNumberOfRows: true) 41
42 0 0.2 4113.9 0.2X
RocksDB (trackTotalNumberOfRows: false) 20
20 0 0.5 1967.1 0.4X
OpenJDK 64-Bit Server VM 1.8.0_362-b09 on Linux 5.15.0-1037-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
putting 10000 rows (0 rows to overwrite - rate 0): Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------
In-memory 7
8 1 1.4 722.2 1.0X
RocksDB (trackTotalNumberOfRows: true) 35
36 0 0.3 3533.8 0.2X
RocksDB (trackTotalNumberOfRows: false) 19
20 0 0.5 1929.7 0.4X
```
With low overwrite rate, the value is ~same as current. So we are not seeing
the perf gain for that case either. Closing PR as not required. Current model
seems to be the best, perf-wise. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
