nickva opened a new issue, #4650:
URL: https://github.com/apache/couchdb/issues/4650
Folsom histograms can become a bottleneck when writting docs with a high
concurrency.
Initially this was observed in a benchmark run with lock counting enabled:
```
> lcnt:rt_opt({copy_save, true}), lcnt:clear(), timer:sleep(10000),
lcnt:collect(), lcnt:conflicts().
lock id #tries #collisions collisions [%]
time [us] duration [%]
----- --- ------- ------------ ---------------
---------- -------------
db_hash_slot 2432 3073630 155606 5.0626
80812927 807.3885
run_queue 22 12050789 1823812 15.1344
74547468 744.7913
proc_main 64973 6287153 2020053 32.1299
19640746 196.2274
pix_lock 1024 320419 272 0.0849
3071989 30.6917
proc_status 64973 5666139 92365 1.6301
1792422 17.9078
alcu_allocator 10 1146957 108469 9.4571
1674152 16.7262
dirty_run_queue_sleep_list 2 1920837 276037 14.3707
1355878 13.5464
proc_msgq 64973 7011355 24101 0.3437
78924 0.7885
dist_entry_out_queue 7 367096 1676 0.4566
6023 0.0602
atom_tab 1 4760261 46 0.0010
4167 0.0416
port_sched_lock 101 390590 353 0.0904
1705 0.0170
dist_entry_links 6 23389 14 0.0599
323 0.0032
db_tab 399 4968232 13 0.0003
146 0.0015
drv_ev_state 128 72977 6 0.0082
40 0.0004
ok
```
```
> lcnt:inspect(db_hash_slot).
lock id #tries #collisions collisions [%]
time [us] duration [%] histogram [log2(us)]
----- --- ------- ------------ ---------------
---------- ------------- ---------------------
db_hash_slot folsom_slide_uniform 21808 13020 59.7029
7075598 70.6911 | ............XX. . |
db_hash_slot folsom_slide_uniform 24165 12706 52.5802
6640890 66.3480 | .............XX.. |
db_hash_slot folsom_slide_uniform 25239 10766 42.6562
5779694 57.7440 | ............XX.. |
db_hash_slot folsom_slide_uniform 23089 9987 43.2544
5082403 50.7774 | ............XX... |
```
To confirm it further, created a build with histogram updates disabled based
on https://github.com/apache/couchdb/pull/4647
Then ran a document insertion ramp-up benchmark and measured number of ops
(inserts) in 10 second windows (throughput) and latency.
<img width="654" alt="Screen Shot 2023-06-20 at 1 42 45 AM"
src="https://github.com/apache/couchdb/assets/211822/e3b6b5c8-49b8-497a-8e2b-3501f6441a1a";>
<img width="714" alt="Screen Shot 2023-06-20 at 1 41 45 AM"
src="https://github.com/apache/couchdb/assets/211822/80a712a0-2ec4-4616-a9fd-84785d7f15be";>
Started at 1000 clients concurrency our throughput improves 100% without
histogram updates.
Latency increases at a faster rate with histogram updates enabled. P95 with
3k concurrent clients is 3x as bad with histogram updates as without.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
