[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071171#comment-14071171
]
graham sanderson commented on CASSANDRA-7546:
---------------------------------------------
In case anyone is reading them, here is the latest output - note with the
current wasted work limit of 100, we actually kick in later except under the
higher contention loads, but doing a one time flip, actually do less work
overall...
{code}
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 64
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 996ms maxConcurrency = 1
[junit] GC for PS Scavenge: 36 ms for 3 collections
[junit] Approx allocation = 563MB vs 8MB; ratio to raw data size =
70.37447428571429
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 765ms maxConcurrency = 1
[junit] GC for PS Scavenge: 38 ms for 3 collections
[junit] Approx allocation = 590MB vs 8MB; ratio to raw data size =
73.67167714285715
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 496ms maxConcurrency = 1
[junit] GC for PS Scavenge: 20 ms for 2 collections
[junit] Approx allocation = 448MB vs 8MB; ratio to raw data size =
55.95978857142857
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 574ms maxConcurrency = 1
[junit] GC for PS Scavenge: 27 ms for 2 collections
[junit] Approx allocation = 485MB vs 8MB; ratio to raw data size =
60.56426285714286
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 662ms maxConcurrency = 1
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 333MB vs 8MB; ratio to raw data size =
41.59998095238095
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 241ms maxConcurrency = 1
[junit] GC for PS Scavenge: 9 ms for 1 collections
[junit] Approx allocation = 349MB vs 8MB; ratio to raw data size =
43.65317619047619
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 222ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 273MB vs 8MB; ratio to raw data size =
34.18085428571428
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 234ms maxConcurrency = 1
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 286MB vs 8MB; ratio to raw data size =
35.788306666666664
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 64
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 1383ms maxConcurrency = 100
[junit] GC for PS Scavenge: 108 ms for 29 collections
[junit] Approx allocation = 9525MB vs 8MB; ratio to raw data size =
1189.0213895238096
[junit] loopRatio (closest to 1 best) 16.74471 raw 100000/1674471 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1728ms maxConcurrency = 100
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 572MB vs 8MB; ratio to raw data size =
71.49758761904762
[junit] loopRatio (closest to 1 best) 1.00011 raw 144/154 counted 0/0
sync 99856/99857 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 223ms maxConcurrency = 100
[junit] GC for PS Scavenge: 24 ms for 2 collections
[junit] Approx allocation = 760MB vs 8MB; ratio to raw data size =
94.87286476190476
[junit] loopRatio (closest to 1 best) 1.88353 raw 100000/188353 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 206ms maxConcurrency = 100
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 454MB vs 8MB; ratio to raw data size =
56.72647714285714
[junit] loopRatio (closest to 1 best) 1.00235 raw 1505/1720 counted 0/0
sync 98495/98515 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 189ms maxConcurrency = 98
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 342MB vs 8MB; ratio to raw data size =
42.777810476190474
[junit] loopRatio (closest to 1 best) 1.01988 raw 100000/101988 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 190ms maxConcurrency = 98
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 346MB vs 8MB; ratio to raw data size =
43.281460952380954
[junit] loopRatio (closest to 1 best) 1.01848 raw 97640/99487 counted 0/0
sync 2360/2361 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 191ms maxConcurrency = 97
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 282MB vs 8MB; ratio to raw data size =
35.20254476190476
[junit] loopRatio (closest to 1 best) 1.0037 raw 100000/100370 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 186ms maxConcurrency = 96
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 300MB vs 8MB; ratio to raw data size =
37.474085714285714
[junit] loopRatio (closest to 1 best) 1.00365 raw 100000/100365 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 256
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 1
[junit] original code:
[junit] Duration = 932ms maxConcurrency = 1
[junit] GC for PS Scavenge: 29 ms for 2 collections
[junit] Approx allocation = 560MB vs 26MB; ratio to raw data size =
21.306322898550725
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 715ms maxConcurrency = 1
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 570MB vs 26MB; ratio to raw data size =
21.673085217391304
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 16
[junit] original code:
[junit] Duration = 572ms maxConcurrency = 1
[junit] GC for PS Scavenge: 9 ms for 1 collections
[junit] Approx allocation = 457MB vs 26MB; ratio to raw data size =
17.370253623188407
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 805ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 460MB vs 26MB; ratio to raw data size =
17.51271188405797
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 256
[junit] original code:
[junit] Duration = 546ms maxConcurrency = 1
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 348MB vs 26MB; ratio to raw data size =
13.242675942028985
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 253ms maxConcurrency = 1
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 349MB vs 26MB; ratio to raw data size =
13.26744347826087
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 256) partitions = 1024
[junit] original code:
[junit] Duration = 457ms maxConcurrency = 1
[junit] GC for PS Scavenge: 15 ms for 1 collections
[junit] Approx allocation = 293MB vs 26MB; ratio to raw data size =
11.139665217391304
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 546ms maxConcurrency = 1
[junit] GC for PS Scavenge: 15 ms for 1 collections
[junit] Approx allocation = 294MB vs 26MB; ratio to raw data size =
11.20318347826087
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 256
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 1
[junit] original code:
[junit] Duration = 1444ms maxConcurrency = 100
[junit] GC for PS Scavenge: 90 ms for 30 collections
[junit] Approx allocation = 9732MB vs 26MB; ratio to raw data size =
369.75210231884057
[junit] loopRatio (closest to 1 best) 16.57088 raw 100000/1657088 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1610ms maxConcurrency = 100
[junit] GC for PS Scavenge: 17 ms for 1 collections
[junit] Approx allocation = 580MB vs 26MB; ratio to raw data size =
22.067385797101448
[junit] loopRatio (closest to 1 best) 1.00018 raw 155/172 counted 0/0
sync 99845/99846 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 16
[junit] original code:
[junit] Duration = 218ms maxConcurrency = 100
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 783MB vs 26MB; ratio to raw data size =
29.776342898550723
[junit] loopRatio (closest to 1 best) 1.93609 raw 100000/193609 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 204ms maxConcurrency = 99
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 469MB vs 26MB; ratio to raw data size =
17.852590144927536
[junit] loopRatio (closest to 1 best) 1.00213 raw 1866/2073 counted 0/0
sync 98134/98140 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 256
[junit] original code:
[junit] Duration = 187ms maxConcurrency = 98
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 361MB vs 26MB; ratio to raw data size =
13.738468695652173
[junit] loopRatio (closest to 1 best) 1.02042 raw 100000/102042 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 189ms maxConcurrency = 98
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 363MB vs 26MB; ratio to raw data size =
13.825418260869565
[junit] loopRatio (closest to 1 best) 1.01876 raw 97150/99024 counted 0/0
sync 2850/2852 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 256) partitions = 1024
[junit] original code:
[junit] Duration = 178ms maxConcurrency = 97
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 301MB vs 26MB; ratio to raw data size =
11.468054492753623
[junit] loopRatio (closest to 1 best) 1.00399 raw 100000/100399 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 188ms maxConcurrency = 100
[junit] GC for PS Scavenge: 14 ms for 1 collections
[junit] Approx allocation = 305MB vs 26MB; ratio to raw data size =
11.606495652173914
[junit] loopRatio (closest to 1 best) 1.00398 raw 100000/100398 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 1 THREAD; ELEMENT SIZE 1024
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1
[junit] original code:
[junit] Duration = 1024ms maxConcurrency = 1
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 596MB vs 99MB; ratio to raw data size =
5.994579233716475
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 848ms maxConcurrency = 1
[junit] GC for PS Scavenge: 39 ms for 3 collections
[junit] Approx allocation = 634MB vs 99MB; ratio to raw data size =
6.374710421455939
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 16
[junit] original code:
[junit] Duration = 599ms maxConcurrency = 1
[junit] GC for PS Scavenge: 24 ms for 2 collections
[junit] Approx allocation = 540MB vs 99MB; ratio to raw data size =
5.430191570881226
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 812ms maxConcurrency = 1
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 524MB vs 99MB; ratio to raw data size =
5.269081149425287
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 256
[junit] original code:
[junit] Duration = 702ms maxConcurrency = 1
[junit] GC for PS Scavenge: 34 ms for 2 collections
[junit] Approx allocation = 414MB vs 99MB; ratio to raw data size =
4.164158697318007
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 618ms maxConcurrency = 1
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 403MB vs 99MB; ratio to raw data size =
4.057552490421456
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1024
[junit] original code:
[junit] Duration = 485ms maxConcurrency = 1
[junit] GC for PS Scavenge: 30 ms for 2 collections
[junit] Approx allocation = 356MB vs 99MB; ratio to raw data size =
3.5774026053639845
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 469ms maxConcurrency = 1
[junit] GC for PS Scavenge: 28 ms for 2 collections
[junit] Approx allocation = 360MB vs 99MB; ratio to raw data size =
3.617753026819923
[junit] loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --------------------------------------------------
[junit] 100 THREADS; ELEMENT SIZE 1024
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1
[junit] original code:
[junit] Duration = 2016ms maxConcurrency = 100
[junit] GC for PS Scavenge: 139 ms for 39 collections
[junit] Approx allocation = 12348MB vs 99MB; ratio to raw data size =
124.02623823754789
[junit] loopRatio (closest to 1 best) 18.58224 raw 100000/1858224 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 1401ms maxConcurrency = 100
[junit] GC for PS Scavenge: 25 ms for 2 collections
[junit] Approx allocation = 640MB vs 99MB; ratio to raw data size =
6.437993486590038
[junit] loopRatio (closest to 1 best) 1.00019 raw 21/39 counted 0/0 sync
99979/99980 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 16
[junit] original code:
[junit] Duration = 234ms maxConcurrency = 100
[junit] GC for PS Scavenge: 36 ms for 3 collections
[junit] Approx allocation = 852MB vs 99MB; ratio to raw data size =
8.563656015325671
[junit] loopRatio (closest to 1 best) 1.9605 raw 100000/196050 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 226ms maxConcurrency = 100
[junit] GC for PS Scavenge: 29 ms for 2 collections
[junit] Approx allocation = 484MB vs 99MB; ratio to raw data size =
4.861978007662835
[junit] loopRatio (closest to 1 best) 1.00254 raw 897/1143 counted 0/0
sync 99103/99111 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 256
[junit] original code:
[junit] Duration = 214ms maxConcurrency = 98
[junit] GC for PS Scavenge: 32 ms for 2 collections
[junit] Approx allocation = 418MB vs 99MB; ratio to raw data size =
4.201469731800766
[junit] loopRatio (closest to 1 best) 1.01951 raw 100000/101951 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 216ms maxConcurrency = 97
[junit] GC for PS Scavenge: 35 ms for 2 collections
[junit] Approx allocation = 409MB vs 99MB; ratio to raw data size =
4.107927662835249
[junit] loopRatio (closest to 1 best) 1.01843 raw 96895/98735 counted 0/0
sync 3105/3108 up 0 down 0
[junit]
[junit]
[junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1024
[junit] original code:
[junit] Duration = 200ms maxConcurrency = 100
[junit] GC for PS Scavenge: 31 ms for 2 collections
[junit] Approx allocation = 367MB vs 99MB; ratio to raw data size =
3.6926790038314175
[junit] loopRatio (closest to 1 best) 1.00402 raw 100000/100402 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 201ms maxConcurrency = 95
[junit] GC for PS Scavenge: 32 ms for 2 collections
[junit] Approx allocation = 364MB vs 99MB; ratio to raw data size =
3.6609434482758623
[junit] loopRatio (closest to 1 best) 1.00399 raw 100000/100399 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit]
[junit] ==================================================
{code}
> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-7546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: graham sanderson
> Assignee: graham sanderson
> Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
> 7546.20_alt.txt, suggestion1.txt, suggestion1_21.txt
>
>
> In order to preserve atomicity, this code attempts to read, clone/update,
> then CAS the state of the partition.
> Under heavy contention for updating a single partition this can cause some
> fairly staggering memory growth (the more cores on your machine the worst it
> gets).
> Whilst many usage patterns don't do highly concurrent updates to the same
> partition, hinting today, does, and in this case wild (order(s) of magnitude
> more than expected) memory allocation rates can be seen (especially when the
> updates being hinted are small updates to different partitions which can
> happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation
> whilst not slowing down the very common un-contended case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
