[
https://issues.apache.org/jira/browse/CASSANDRA-20230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17916337#comment-17916337
]
Branimir Lambov commented on CASSANDRA-20230:
---------------------------------------------
The algorithm optimizes node ownership at the expense of individual ranges',
thus it is expected that some ranges' ownership will be very small. It does
look, however, that the results are not as good as expected in this case.
I can see that the number of racks is equal to the replication factor. This
case was not handled well before CASSANDRA-15600 and could result in very
uneven spreads, especially if nodes were not added one node from a rack at a
time.
Something like this can also happen if nodes are not decommissioned in the
opposite of the order in which they were added. In this case the algorithm has
no control over what happens, and the results may be a lot different from
desired.
Taking away ownership from the largest node(s) is what the algorithm does.
Making sure the cluster is using 4.0 or later to include CASSANDRA-15600, and
letting the algorithm choose some new tokens by adding a new node should
improve the distribution significantly (cutting vnode-many of the biggest
nodes' sizes dramatically). If updrade to 4.0 is out of the question, maybe we
can find some creative way of using the {{generatetokens}} tool or a derivative
of it to help.
> vnode algorithm occasionally generates imbalanced rings
> -------------------------------------------------------
>
> Key: CASSANDRA-20230
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20230
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jon Haddad
> Priority: Normal
> Attachments: ring.txt
>
>
> Generally, the deterministic vnode algorithm seems to make OK decisions, but
> I've been working on a cluster where the tokens it's generated look like
> they're not much better than random. I've verified
> `allocate_tokens_for_local_replication_factor=3` was used when the nodes were
> bootstrapped via the logs.
> I've parsed the nodetool ring output and added some additional info so it's
> obvious where the ranges are imbalanced. I've calculated the range for each
> token along with the avg range for each token range in the cluster, and
> listed each range size relative to the average as the %. For example:
> The node that owns the token -5646866338711190221, that range is only 9.23%
> of the average token range in the cluster.
>
> {noformat}
> 37 10.20.80.11 rack-b -5646866338711190221 9455879439619330
> 9.23% 347209701613153318 112.93%{noformat}
> Looking at the below data, it's obvious that some nodes have way more data
> (272622351113636532 266.02%) than they should.
> It doesn't look like we're trying to balance the ring by picking tokens that
> split the largest ranges, which explains why the most recent node I we added
> had less than half of the data as the largest node.
>
> {noformat}
> first token:9170886775430075796
> Ring tokens:
> # Address Rack Token Range
> Rack Range
> ---------------------------------------------------------------------------------------------------------
> 1 10.20.16.94 rack-a -9223372036854775808 52485261424700012
> 51.21% 134226759678331788 43.66%
> 2 10.20.146.47 rack-c -9189547180369111384 33824856485664424
> 33.01% 244426531911895070 79.5%
> 3 10.20.84.97 rack-b -9098397575186281096 91149605182830288
> 88.94% 177459723093194724 57.72%
> 4 10.20.17.254 rack-a -9044662703355201173 53734871831079923
> 52.43% 178709333499574635 58.13%
> 5 10.20.93.76 rack-b -8949214606862506405 95448096492694768
> 93.14% 149182968323774691 48.52%
> 6 10.20.149.213 rack-c -8739828963515584556 209385643346921849
> 204.31% 449718216853526828 146.28%
> 7 10.20.93.195 rack-b -8682698381181565144 57130582334019412
> 55.75% 266516225680941261 86.69%
> 8 10.20.88.117 rack-b -8541994507578267972 140703873603297172
> 137.3% 140703873603297172 45.77%
> 9 10.20.21.149 rack-a -8513158149005026369 28836358573241603
> 28.14% 531504554350174804 172.88%
> 10 10.20.145.96 rack-c -8353521350819354568 159636798185671801
> 155.77% 386307612696229988 125.65%
> 11 10.20.92.129 rack-b -8261404278536710957 92117072282643611
> 89.89% 280590229041557015 91.26%
> 12 10.20.26.105 rack-a -8200535838360498152 60868440176212805
> 59.39% 312622310644528217 101.68%
> 13 10.20.95.182 rack-b -8021391251878672242 179144586481825910
> 174.81% 240013026658038715 78.07%
> 14 10.20.158.149 rack-c -8006311649206201825 15079602672470417
> 14.71% 347209701613152743 112.93%
> 15 10.20.21.212 rack-a -7766062938935428803 240248710270773022
> 234.43% 434472899425069349 141.32%
> 16 10.20.144.132 rack-c -7739844482128539727 26218456806889076
> 25.58% 266467167077662098 86.67%
> 17 10.20.88.17 rack-b -7670800529894427760 69043952234111967
> 67.37% 350590721984244482 114.03%
> 18 10.20.30.86 rack-a -7466462004095374782 204338525799052978
> 199.39% 299600934840054021 97.45%
> 19 10.20.151.62 rack-c -7426908388950633006 39553615144741776
> 38.6% 312936093177906721 101.79%
> 20 10.20.19.95 rack-a -7366859901397320121 60048487553312885
> 58.59% 99602102698054661 32.4%
> 21 10.20.92.38 rack-b -7365507018146776908 1352883250543213
> 1.32% 305293511747650852 99.3%
> 22 10.20.29.147 rack-a -7227645749564266633 137861268582510275
> 134.52% 139214151833053488 45.28%
> 23 10.20.145.3 rack-c -7171605796608455430 56039952955811203
> 54.68% 255302592342177576 83.04%
> 24 10.20.80.231 rack-b -7040317985592000803 131287811016454627
> 128.11% 325189032554776105 105.77%
> 25 10.20.26.105 rack-a -6981127409214619804 59190576377380999
> 57.76% 246518340349646829 80.18%
> 26 10.20.83.58 rack-b -6882252606836285198 98874802378334606
> 96.48% 158065378755715605 51.41%
> 27 10.20.158.56 rack-c -6776630009572643883 105622597263641315
> 103.06% 394975787035811547 128.47%
> 28 10.20.30.74 rack-a -6648811345754421660 127818663818222223
> 124.72% 332316063460198144 108.09%
> 29 10.20.21.121 rack-a -6535221896394566414 113589449359855246
> 110.84% 113589449359855246 36.95%
> 30 10.20.151.96 rack-c -6420071778790659373 115150117603907041
> 112.36% 356558230781984510 115.97%
> 31 10.20.82.247 rack-b -6357720556701447159 62351222089212214
> 60.84% 524532050134838039 170.61%
> 32 10.20.28.103 rack-a -6305056011315416293 52664545386030866
> 51.39% 230165885079150121 74.86%
> 33 10.20.158.149 rack-c -6032433660201779761 272622351113636532
> 266.02% 387638118588879612 126.08%
> 34 10.20.88.117 rack-b -5994076040324343539 38357619877436222
> 37.43% 363644516377103620 118.28%
> 35 10.20.145.96 rack-c -5778229414025526901 215846626298816638
> 210.62% 254204246176252860 82.68%
> 36 10.20.26.105 rack-a -5656322218150809551 121907195874717350
> 118.95% 648733793164606742 211.01%
> 37 10.20.80.11 rack-b -5646866338711190221 9455879439619330
> 9.23% 347209701613153318 112.93%
> 38 10.20.21.149 rack-a -5394494374428312525 252371964282877696
> 246.26% 261827843722497026 85.16%
> 39 10.20.93.76 rack-b -5380399171633528006 14095202794784519
> 13.75% 266467167077662215 86.67%
> 40 10.20.144.132 rack-c -5227306901031281262 153092270602246744
> 149.38% 550922512994245639 179.19%
> 41 10.20.24.172 rack-a -5181175383115792585 46131517915488677
> 45.01% 213318991312519940 69.38%
> 42 10.20.93.195 rack-b -5118571327911031275 62604055204761310
> 61.09% 261827843722496731 85.16%
> 43 10.20.92.206 rack-b -4962413169712651344 156158158198379931
> 152.38% 156158158198379931 50.79%
> 44 10.20.144.255 rack-c -4894990837571083608 67422332141567736
> 65.79% 332316063460197654 108.09%
> 45 10.20.19.95 rack-a -4724399575827667401 170591261743416207
> 166.46% 456775807288125184 148.57%
> 46 10.20.93.195 rack-b -4683676894619741545 40722681207925856
> 39.74% 278736275092909799 90.66%
> 47 10.20.146.47 rack-c -4649179066907635035 34497827712106510
> 33.66% 245811770663448573 79.95%
> 48 10.20.16.94 rack-a -4611686018427387905 37493048480247130
> 36.59% 112713557400279496 36.66%
> 49 10.20.21.212 rack-a -4409623112414382071 202062906013005834
> 197.17% 202062906013005834 65.72%
> 50 10.20.92.38 rack-b -4371054583975213180 38568528439168891
> 37.63% 312622310644528365 101.68%
> 51 10.20.144.255 rack-c -4298588344923390323 72466239051822857
> 70.71% 350590721984244712 114.03%
> 52 10.20.150.20 rack-c -3993294833175739272 305293511747651051
> 297.9% 305293511747651051 99.3%
> 53 10.20.91.254 rack-b -3937594054506566378 55700778669172894
> 54.35% 433460529468646802 140.99%
> 54 10.20.29.147 rack-a -3937256674792971232 337379713595146
> 0.33% 472366437621410839 153.64%
> 55 10.20.151.62 rack-c -3668105800620963222 269150874172008010
> 262.63% 325189032554776050 105.77%
> 56 10.20.84.97 rack-b -3487875837653040055 180229962967923167
> 175.87% 449718216853526323 146.28%
> 57 10.20.145.96 rack-c -3435566666045219787 52309171607820268
> 51.04% 232539134575743435 75.64%
> 58 10.20.24.172 rack-a -3272020364001133020 163546302044086767
> 159.59% 665236310791838212 216.38%
> 59 10.20.30.74 rack-a -3135096996508842591 136923367492290429
> 133.61% 136923367492290429 44.54%
> 60 10.20.83.58 rack-b -3063848761308671438 71248235200171153
> 69.52% 424027076344368617 137.92%
> 61 10.20.147.229 rack-c -2911034615910381452 152814145398289986
> 149.11% 524532050134838335 170.61%
> 62 10.20.26.105 rack-a -2829803484761191279 81231131149190173
> 79.26% 305293511747651312 99.3%
> 63 10.20.80.11 rack-b -2696424851207654625 133378633553536654
> 130.15% 367423910101016813 119.51%
> 64 10.20.151.96 rack-c -2547390099533278228 149034751674376397
> 145.43% 363644516377103224 118.28%
> 65 10.20.17.254 rack-a -2504614452206415750 42775647326862478
> 41.74% 325189032554775529 105.77%
> 66 10.20.91.254 rack-b -2322859386475439970 181755065730975780
> 177.35% 373565464732214655 121.51%
> 67 10.20.22.103 rack-a -2315699623075219595 7159763400220375
> 6.99% 188914829131196155 61.45%
> 68 10.20.144.255 rack-c -2192377191226873342 123322431848346253
> 120.34% 355012908306404886 115.47%
> 69 10.20.149.213 rack-c -2056222665366193435 136154525860679907
> 132.86% 136154525860679907 44.29%
> 70 10.20.29.147 rack-a -1791167572940380850 265055092425812585
> 258.64% 524532050134838745 170.61%
> 71 10.20.93.76 rack-b -1771936873481194892 19230699459185958
> 18.76% 550922512994245078 179.19%
> 72 10.20.145.3 rack-c -1719845305934835095 52091567546359797
> 50.83% 336377359431358340 109.41%
> 73 10.20.24.172 rack-a -1569784851601295151 150060454333539944
> 146.43% 221382721339085699 72.01%
> 74 10.20.21.121 rack-a -1481596503541663932 88188348059631219
> 86.05% 88188348059631219 28.68%
> 75 10.20.88.17 rack-b -1439620810020997023 41975693520666909
> 40.96% 332316063460197869 108.09%
> 76 10.20.151.62 rack-c -1411320999486545250 28299810534451773
> 27.61% 308524306448289845 100.35%
> 77 10.20.147.229 rack-c -1194751819402005368 216569180084539882
> 211.32% 216569180084539882 70.44%
> 78 10.20.95.182 rack-b -1138690567005124535 56061252396880833
> 54.7% 300930243015872488 97.88%
> 79 10.20.30.74 rack-a -1072510148256874743 66180418748249792
> 64.58% 409086355284789189 133.06%
> 80 10.20.29.51 rack-a -936355622396193590 136154525860681153
> 132.86% 136154525860681153 44.29%
> 81 10.20.150.58 rack-c -888273494874972566 48082127521221024
> 46.92% 306478324527032802 99.69%
> 82 10.20.89.254 rack-b -763548809496363728 124724685378608838
> 121.7% 375141757508760807 122.02%
> 83 10.20.148.78 rack-c -695026190728748975 68522618767614753
> 66.86% 193247304146223591 62.86%
> 84 10.20.92.206 rack-b -454091375866946019 240934814861802956
> 235.1% 309457433629417709 100.65%
> 85 10.20.150.58 rack-c -404151275954123662 49940099912822357
> 48.73% 290874914774625313 94.61%
> 86 10.20.17.254 rack-a -333582278939827335 70568997014296327
> 68.86% 602773343456366255 196.06%
> 87 10.20.144.132 rack-c -120191231713029034 213391047226798301
> 208.22% 283960044241094628 92.36%
> 88 10.20.29.147 rack-a -117013098855287122 3178132857741912
> 3.1% 216569180084540213 70.44%
> 89 10.20.16.94 rack-a -2 117013098855287120
> 114.18% 117013098855287120 38.06%
> 90 10.20.82.247 rack-b 18275061754464473 18275061754464475
> 17.83% 472366437621410492 153.64%
> 91 10.20.30.86 rack-a 189465225671745150 171190163917280677
> 167.04% 189465225671745152 61.63%
> 92 10.20.145.96 rack-c 302842028835857714 113376803164112564
> 110.63% 423033260548886748 137.6%
> 93 10.20.21.212 rack-a 382712529817969305 79870500982111591
> 77.94% 193247304146224155 62.86%
> 94 10.20.95.182 rack-b 401657613055777279 18945083237807974
> 18.49% 383382551301312806 124.7%
> 95 10.20.158.149 rack-c 670265938936873543 268608325881096264
> 262.1% 367423910101015829 119.51%
> 96 10.20.30.86 rack-a 673587444592594063 3321505655720520
> 3.24% 290874914774624758 94.61%
> 97 10.20.92.129 rack-b 692532527830402040 18945083237807977
> 18.49% 290874914774624761 94.61%
> 98 10.20.23.199 rack-a 957547488833689288 265014961003287248
> 258.6% 283960044241095225 92.36%
> 99 10.20.93.76 rack-b 976492572071497267 18945083237807979
> 18.49% 283960044241095227 92.36%
> 100 10.20.148.78 rack-c 976675586628109273 183014556612006
> 0.18% 306409647691235730 99.66%
> 101 10.20.22.103 rack-a 1307873573641664077 331197987013554804
> 323.18% 350326084807974789 113.95%
> 102 10.20.83.58 rack-b 1312157198425561502 4283624783897425
> 4.18% 335664626354064235 109.18%
> 103 10.20.147.229 rack-c 1449042024249520056 136884825823958554
> 133.57% 472366437621410783 153.64%
> 104 10.20.24.172 rack-a 1526181194689538001 77139170440017945
> 75.27% 218307621047873924 71.01%
> 105 10.20.21.149 rack-a 1598829898146541929 72648703457003928
> 70.89% 72648703457003928 23.63%
> 106 10.20.89.254 rack-b 1735616304527059052 136786406380517123
> 133.47% 423459106101497550 137.73%
> 107 10.20.29.51 rack-a 1776289621239736865 40673316712677813
> 39.69% 177459723093194936 57.72%
> 108 10.20.23.199 rack-a 1925472589563511867 149182968323775002
> 145.57% 149182968323775002 48.52%
> 109 10.20.146.47 rack-c 1941821895281165489 16349305717653622
> 15.95% 492779871031645433 160.28%
> 110 10.20.92.38 rack-b 2032567080151914406 90745184870748917
> 88.55% 296950775624855354 96.59%
> 111 10.20.21.149 rack-a 2191988815244453426 159421735092539020
> 155.56% 266516225680941559 86.69%
> 112 10.20.158.56 rack-c 2365281001382663071 173292186138209645
> 169.1% 423459106101497582 137.73%
> 113 10.20.88.17 rack-b 2387579988458317163 22298987075654092
> 21.76% 355012908306402757 115.47%
> 114 10.20.84.97 rack-b 2523734514318999176 136154525860682013
> 132.86% 136154525860682013 44.29%
> 115 10.20.30.86 rack-a 2613282917889306604 89548403570307428
> 87.38% 421294102644853178 137.03%
> 116 10.20.150.20 rack-c 2662231777007518448 48948859118211844
> 47.76% 296950775624855377 96.59%
> 117 10.20.92.206 rack-b 2846661377600858870 184429600593340422
> 179.96% 322926863281859694 105.04%
> 118 10.20.21.121 rack-a 2883076181595991262 36414803995132392
> 35.53% 269793263706684658 87.75%
> 119 10.20.144.251 rack-c 2924059620730015183 40983439134023921
> 39.99% 261827843722496735 85.16%
> 120 10.20.145.3 rack-c 3180697442048841657 256637821318826474
> 250.42% 256637821318826474 83.47%
> 121 10.20.89.254 rack-b 3241637164636670946 60939722587829289
> 59.46% 394975787035812076 128.47%
> 122 10.20.29.51 rack-a 3362088047846622951 120450883209952005
> 117.53% 479011866250631689 155.8%
> 123 10.20.144.251 rack-c 3471653766553720026 109565718707097075
> 106.91% 290956324504878369 94.64%
> 124 10.20.88.117 rack-b 3598195395418656077 126541628864936051
> 123.48% 356558230781985131 115.97%
> 125 10.20.149.213 rack-c 3649113489646914674 50918094228258597
> 49.68% 177459723093194648 57.72%
> 126 10.20.22.103 rack-a 3792019965987400295 142906476340485621
> 139.45% 429931918140777344 139.84%
> 127 10.20.144.132 rack-c 3798296457970688935 6276491983288640
> 6.12% 149182968323774261 48.52%
> 128 10.20.80.11 rack-b 3985833514007536019 187537056036847084
> 183.0% 387638118588879942 126.08%
> 129 10.20.144.251 rack-c 4064812683651630209 78979169644094190
> 77.07% 266516225680941274 86.69%
> 130 10.20.19.95 rack-a 4139229667600553871 74416983948923662
> 72.61% 347209701613153576 112.93%
> 131 10.20.151.96 rack-c 4205516557254927886 66286889654374015
> 64.68% 140703873603297677 45.77%
> 132 10.20.83.58 rack-b 4364160686550500444 158644129295572558
> 154.8% 378327172542964425 123.05%
> 133 10.20.23.199 rack-a 4405696834678215201 41536148127714757
> 40.53% 266467167077661330 86.67%
> 134 10.20.150.58 rack-c 4486106786296485649 80409951618270448
> 78.46% 280590229041557763 91.26%
> 135 10.20.16.94 rack-a 4611686018427387901 125579232130902252
> 122.54% 205989183749172700 67.0%
> 136 10.20.92.129 rack-b 4663761621390554463 52075602963166562
> 50.81% 299600934840054019 97.45%
> 137 10.20.146.47 rack-c 4729291875508437146 65530254117882683
> 63.94% 243185089211951497 79.1%
> 138 10.20.80.11 rack-b 4763363724088609656 34071848580172510
> 33.25% 99602102698055193 32.4%
> 139 10.20.82.247 rack-b 4902577875921664056 139214151833054400
> 135.84% 139214151833054400 45.28%
> 140 10.20.144.255 rack-c 5064821116816274786 162243240894610730
> 158.31% 335529241307837640 109.13%
> 141 10.20.92.38 rack-b 5149096216271310975 84275099455036189
> 82.23% 246518340349646919 80.18%
> 142 10.20.151.96 rack-c 5344183344632085640 195087128360774665
> 190.36% 279362227815810854 90.87%
> 143 10.20.21.121 rack-a 5344190431341493990 7086709408350
> 0.01% 732504412914106089 238.25%
> 144 10.20.80.231 rack-b 5470232255249086842 126041823907592852
> 122.99% 321136038977775867 104.45%
> 145 10.20.28.103 rack-a 5599615274198389999 129383018949303157
> 126.25% 255424842856896009 83.08%
> 146 10.20.89.254 rack-b 5601038641438494310 1423367240104311
> 1.39% 130806386189407468 42.55%
> 147 10.20.149.213 rack-c 5763351906979850606 162313265541356296
> 158.38% 419168562347764966 136.34%
> 148 10.20.95.182 rack-b 5902916083282211410 139564176302360804
> 136.18% 301877441843717100 98.19%
> 149 10.20.148.78 rack-c 6095717362449884318 192801279167672908
> 188.13% 332365455470033712 108.11%
> 150 10.20.19.95 rack-a 6189475282618394230 93757920168509912
> 91.49% 589860008420004231 191.86%
> 151 10.20.88.17 rack-b 6238445324590049310 48970041971655080
> 47.78% 335529241307837900 109.13%
> 152 10.20.158.56 rack-c 6470859119958645149 232413795368595839
> 226.79% 375141757508760831 122.02%
> 153 10.20.22.103 rack-a 6516386704535559376 45527584576914227
> 44.42% 326911421917165146 106.33%
> 154 10.20.88.117 rack-b 6517807552405859479 1420847870300103
> 1.39% 279362227815810169 90.87%
> 155 10.20.145.3 rack-c 6741694105655993871 223886553250134392
> 218.46% 270834985697348722 88.09%
> 156 10.20.84.97 rack-b 6936976114753622943 195282009097629072
> 190.55% 419168562347763464 136.34%
> 157 10.20.144.251 rack-c 7020430380748903693 83454265995280750
> 81.43% 278736275092909822 90.66%
> 158 10.20.23.199 rack-a 7067309217529804976 46878836780901283
> 45.74% 550922512994245600 179.19%
> 159 10.20.91.254 rack-b 7269341570223658427 202032352693853451
> 197.14% 332365455470035484 108.11%
> 160 10.20.150.20 rack-c 7333052691393432086 63711121169773659
> 62.17% 312622310644528393 101.68%
> 161 10.20.17.254 rack-a 7388445256507581893 55392565114149807
> 54.05% 321136038977776917 104.45%
> 162 10.20.80.231 rack-b 7582277663401565387 193832406893983494
> 189.14% 312936093177906960 101.79%
> 163 10.20.28.103 rack-a 7631751918907982030 49474255506416643
> 48.28% 243306662400400137 79.14%
> 164 10.20.148.78 rack-c 7775892406540551989 144140487632569959
> 140.65% 442839715147119903 144.04%
> 165 10.20.92.206 rack-b 7912887967081619192 136995560541067203
> 133.68% 330610303680053805 107.53%
> 166 10.20.30.74 rack-a 8058249031542833245 145361064461214053
> 141.84% 426497112634851215 138.72%
> 167 10.20.150.58 rack-c 8075493341380604702 17244309837771457
> 16.83% 299600934840052713 97.45%
> 168 10.20.158.149 rack-c 8175095444078661163 99602102698056461
> 97.19% 99602102698056461 32.4%
> 169 10.20.80.231 rack-b 8221412273529908988 46316829451247825
> 45.2% 308524306448289796 100.35%
> 170 10.20.28.103 rack-a 8297681652552085364 76269379022176376
> 74.42% 239432621009252119 77.88%
> 171 10.20.147.229 rack-c 8314309595911715114 16627943359629750
> 16.23% 139214151833053951 45.28%
> 172 10.20.82.247 rack-b 8437981453614448838 123671857702733724
> 120.68% 216569180084539850 70.44%
> 173 10.20.150.20 rack-c 8560827936261361580 122846482646912742
> 119.87% 246518340349646466 80.18%
> 174 10.20.92.129 rack-b 8744459778141481139 183631841880119559
> 179.18% 306478324527032301 99.69%
> 175 10.20.29.51 rack-a 8756779821706409579 12320043564928440
> 12.02% 459098169154324215 149.33%
> 176 10.20.91.254 rack-b 8879930450925197428 123150629218787849
> 120.17% 135470672783716289 44.06%
> 177 10.20.151.62 rack-c 8881963975239137216 2033524313939788
> 1.98% 321136038977775636 104.45%
> 178 10.20.158.56 rack-c 9012770361428545162 130806386189407946
> 127.64% 130806386189407946 42.55%
> 179 10.20.21.212 rack-a 9089145277176444020 76374915747898858
> 74.53% 332365455470034441 108.11%
> 180 10.20.93.195 rack-b 9170886775430075796 81741498253631776
> 79.76% 290956324504878368 94.64%{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
