[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069059#comment-14069059
]
graham sanderson commented on CASSANDRA-7546:
---------------------------------------------
FYI here are the same synthetic test results for 7546.20_2.txt
{code}
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 64
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 993ms maxConcurrency = 1
[junit] GC for PS Scavenge: 34 ms for 3 collections
[junit] Approx allocation = 553MB vs 8MB; ratio to raw data size =
69.13799428571429
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 761ms maxConcurrency = 1
[junit] GC for PS Scavenge: 34 ms for 3 collections
[junit] Approx allocation = 579MB vs 8MB; ratio to raw data size =
72.31675047619048
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 780ms maxConcurrency = 1
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 436MB vs 8MB; ratio to raw data size =
54.48992095238095
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 671ms maxConcurrency = 1
[junit] GC for PS Scavenge: 24 ms for 2 collections
[junit] Approx allocation = 477MB vs 8MB; ratio to raw data size =
59.545997142857146
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 452ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 321MB vs 8MB; ratio to raw data size =
40.14510761904762
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 460ms maxConcurrency = 1
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 341MB vs 8MB; ratio to raw data size =
42.63770857142857
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 462ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 264MB vs 8MB; ratio to raw data size =
32.99879142857143
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 543ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 272MB vs 8MB; ratio to raw data size =
34.047360952380956
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 64
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 2318ms maxConcurrency = 100
[junit] GC for PS Scavenge: 119 ms for 32 collections
[junit] Approx allocation = 10547MB vs 8MB; ratio to raw data size =
1316.62704
[junit] loopRatio (closest to 1 best) 18.35448 raw 100000/1835448 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1315ms maxConcurrency = 100
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 629MB vs 8MB; ratio to raw data size =
78.62949142857143
[junit] loopRatio (closest to 1 best) 1.11563 raw 13653/13653 counted 0/0
sync 88223/97910 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 215ms maxConcurrency = 100
[junit] GC for PS Scavenge: 23 ms for 2 collections
[junit] Approx allocation = 776MB vs 8MB; ratio to raw data size =
96.92138285714286
[junit] loopRatio (closest to 1 best) 1.95927 raw 100000/195927 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 201ms maxConcurrency = 99
[junit] GC for PS Scavenge: 9 ms for 1 collections
[junit] Approx allocation = 530MB vs 8MB; ratio to raw data size =
66.21505238095239
[junit] loopRatio (closest to 1 best) 1.26719 raw 72820/72820 counted 0/0
sync 49837/53899 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 187ms maxConcurrency = 96
[junit] GC for PS Scavenge: 15 ms for 1 collections
[junit] Approx allocation = 319MB vs 8MB; ratio to raw data size =
39.93250857142857
[junit] loopRatio (closest to 1 best) 1.01989 raw 100000/101989 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 188ms maxConcurrency = 99
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 327MB vs 8MB; ratio to raw data size =
40.83917619047619
[junit] loopRatio (closest to 1 best) 1.01869 raw 99963/99963 counted 0/0
sync 1877/1906 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 169ms maxConcurrency = 98
[junit] Approx allocation = 274MB vs 8MB; ratio to raw data size =
34.27162761904762
[junit] loopRatio (closest to 1 best) 1.00376 raw 100000/100376 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 180ms maxConcurrency = 98
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 278MB vs 8MB; ratio to raw data size =
34.80582571428572
[junit] loopRatio (closest to 1 best) 1.00348 raw 99999/99999 counted 0/0
sync 349/349 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 256
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 1
[junit] original code:
[junit] Duration = 735ms maxConcurrency = 1
[junit] GC for PS Scavenge: 26 ms for 2 collections
[junit] Approx allocation = 553MB vs 26MB; ratio to raw data size =
21.025497101449275
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 384ms maxConcurrency = 1
[junit] GC for PS Scavenge: 24 ms for 2 collections
[junit] Approx allocation = 556MB vs 26MB; ratio to raw data size =
21.123911594202898
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 16
[junit] original code:
[junit] Duration = 329ms maxConcurrency = 1
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 448MB vs 26MB; ratio to raw data size =
17.03236608695652
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 324ms maxConcurrency = 1
[junit] GC for PS Scavenge: 9 ms for 1 collections
[junit] Approx allocation = 449MB vs 26MB; ratio to raw data size =
17.06260608695652
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 256
[junit] original code:
[junit] Duration = 692ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 339MB vs 26MB; ratio to raw data size =
12.898537971014493
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 681ms maxConcurrency = 1
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 338MB vs 26MB; ratio to raw data size =
12.876152753623188
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 1024
[junit] original code:
[junit] Duration = 484ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 282MB vs 26MB; ratio to raw data size =
10.73973768115942
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 463ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 282MB vs 26MB; ratio to raw data size =
10.74182811594203
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 256
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 1
[junit] original code:
[junit] Duration = 2231ms maxConcurrency = 100
[junit] GC for PS Scavenge: 121 ms for 33 collections
[junit] Approx allocation = 10752MB vs 26MB; ratio to raw data size =
408.504135942029
[junit] loopRatio (closest to 1 best) 18.49143 raw 100000/1849143 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1330ms maxConcurrency = 100
[junit] GC for PS Scavenge: 32 ms for 2 collections
[junit] Approx allocation = 587MB vs 26MB; ratio to raw data size =
22.319155362318842
[junit] loopRatio (closest to 1 best) 1.12889 raw 16479/16479 counted 0/0
sync 85804/96410 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 16
[junit] original code:
[junit] Duration = 227ms maxConcurrency = 99
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 752MB vs 26MB; ratio to raw data size =
28.593344057971013
[junit] loopRatio (closest to 1 best) 1.82137 raw 100000/182137 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 204ms maxConcurrency = 98
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 552MB vs 26MB; ratio to raw data size =
21.00069768115942
[junit] loopRatio (closest to 1 best) 1.26846 raw 72715/72715 counted 0/0
sync 49948/54131 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 256
[junit] original code:
[junit] Duration = 180ms maxConcurrency = 94
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 351MB vs 26MB; ratio to raw data size =
13.347453333333334
[junit] loopRatio (closest to 1 best) 1.01903 raw 100000/101903 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 178ms maxConcurrency = 97
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 350MB vs 26MB; ratio to raw data size =
13.320123768115941
[junit] loopRatio (closest to 1 best) 1.0193 raw 99954/99954 counted 0/0
sync 1957/1976 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 1024
[junit] original code:
[junit] Duration = 179ms maxConcurrency = 99
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 292MB vs 26MB; ratio to raw data size =
11.113111884057972
[junit] loopRatio (closest to 1 best) 1.00393 raw 100000/100393 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 178ms maxConcurrency = 93
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 293MB vs 26MB; ratio to raw data size =
11.15906231884058
[junit] loopRatio (closest to 1 best) 1.00396 raw 99997/99997 counted 0/0
sync 398/399 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 1024
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1
[junit] original code:
[junit] Duration = 997ms maxConcurrency = 1
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 586MB vs 99MB; ratio to raw data size =
5.885971954022988
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 836ms maxConcurrency = 1
[junit] GC for PS Scavenge: 42 ms for 3 collections
[junit] Approx allocation = 623MB vs 99MB; ratio to raw data size =
6.264844827586207
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 16
[junit] original code:
[junit] Duration = 600ms maxConcurrency = 1
[junit] GC for PS Scavenge: 28 ms for 2 collections
[junit] Approx allocation = 508MB vs 99MB; ratio to raw data size =
5.104469731800767
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 572ms maxConcurrency = 1
[junit] GC for PS Scavenge: 26 ms for 2 collections
[junit] Approx allocation = 513MB vs 99MB; ratio to raw data size =
5.159989042145594
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 256
[junit] original code:
[junit] Duration = 413ms maxConcurrency = 1
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 440MB vs 99MB; ratio to raw data size =
4.425642222222222
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 368ms maxConcurrency = 1
[junit] GC for PS Scavenge: 27 ms for 2 collections
[junit] Approx allocation = 393MB vs 99MB; ratio to raw data size =
3.948129195402299
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1024
[junit] original code:
[junit] Duration = 296ms maxConcurrency = 1
[junit] GC for PS Scavenge: 27 ms for 2 collections
[junit] Approx allocation = 343MB vs 99MB; ratio to raw data size =
3.4517041379310345
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 550ms maxConcurrency = 1
[junit] GC for PS Scavenge: 35 ms for 2 collections
[junit] Approx allocation = 343MB vs 99MB; ratio to raw data size =
3.4461845977011496
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 1024
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1
[junit] original code:
[junit] Duration = 2343ms maxConcurrency = 100
[junit] GC for PS Scavenge: 143 ms for 38 collections
[junit] Approx allocation = 12065MB vs 99MB; ratio to raw data size =
121.18560452107279
[junit] loopRatio (closest to 1 best) 18.22144 raw 100000/1822144 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1238ms maxConcurrency = 100
[junit] GC for PS Scavenge: 31 ms for 2 collections
[junit] Approx allocation = 569MB vs 99MB; ratio to raw data size =
5.7199810727969345
[junit] loopRatio (closest to 1 best) 1.11326 raw 13522/13522 counted 0/0
sync 88452/97804 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 16
[junit] original code:
[junit] Duration = 235ms maxConcurrency = 100
[junit] GC for PS Scavenge: 35 ms for 3 collections
[junit] Approx allocation = 792MB vs 99MB; ratio to raw data size =
7.96215540229885
[junit] loopRatio (closest to 1 best) 1.79828 raw 100000/179828 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 228ms maxConcurrency = 100
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 618MB vs 99MB; ratio to raw data size =
6.211963754789272
[junit] loopRatio (closest to 1 best) 1.26855 raw 75427/75427 counted 0/0
sync 47424/51428 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 256
[junit] original code:
[junit] Duration = 189ms maxConcurrency = 98
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 421MB vs 99MB; ratio to raw data size =
4.237997701149426
[junit] loopRatio (closest to 1 best) 1.01838 raw 100000/101838 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 199ms maxConcurrency = 98
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 405MB vs 99MB; ratio to raw data size =
4.073938620689655
[junit] loopRatio (closest to 1 best) 1.01976 raw 99960/99960 counted 0/0
sync 2000/2016 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1024
[junit] original code:
[junit] Duration = 195ms maxConcurrency = 100
[junit] GC for PS Scavenge: 32 ms for 2 collections
[junit] Approx allocation = 356MB vs 99MB; ratio to raw data size =
3.577380153256705
[junit] loopRatio (closest to 1 best) 1.00401 raw 100000/100401 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 197ms maxConcurrency = 96
[junit] GC for PS Scavenge: 31 ms for 2 collections
[junit] Approx allocation = 352MB vs 99MB; ratio to raw data size =
3.538006283524904
[junit] loopRatio (closest to 1 best) 1.00377 raw 99999/99999 counted 0/0
sync 376/378 up 0 down 0
{code}
> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-7546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: graham sanderson
> Assignee: graham sanderson
> Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_alt.txt,
> suggestion1.txt, suggestion1_21.txt
>
>
> In order to preserve atomicity, this code attempts to read, clone/update,
> then CAS the state of the partition.
> Under heavy contention for updating a single partition this can cause some
> fairly staggering memory growth (the more cores on your machine the worst it
> gets).
> Whilst many usage patterns don't do highly concurrent updates to the same
> partition, hinting today, does, and in this case wild (order(s) of magnitude
> more than expected) memory allocation rates can be seen (especially when the
> updates being hinted are small updates to different partitions which can
> happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation
> whilst not slowing down the very common un-contended case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
