Re: very slow repair

[Oleksandr Shulgin]

On Thu, Jun 13, 2019 at 2:09 PM Léo FERLIN SUTTON
 wrote:

> Last, but not least: are you using the default number of vnodes, 256?  The
>> overhead of large number of vnodes (times the number of nodes), can be
>> quite significant.  We've seen major improvements in repair runtime after
>> switching from 256 to 16 vnodes on Cassandra version 3.0.
>
>
> Is there a recommended procedure to switch the amount of vnodes ?
>

Yes.  One should deploy a new virtual DC with desired configuration and
rebuild from the original one, then decommission the old virtual DC.

With the smaller number of vnodes you should use
allocate_tokens_for_keyspace configuration parameter to ensure uniform load
distribution.  The caveat is that the nodes allocate tokens before they
bootstrap, so the very first nodes will not have keyspace information
available.  This can be worked around, though it is not trivial.  See this
thread for our past experience:
https://lists.apache.org/thread.html/396f2d20397c36b9cff88a0c2c5523154d420ece24a4dafc9fde3d1f@%3Cuser.cassandra.apache.org%3E

--
Alex

Re: very slow repair

[Léo FERLIN SUTTON]

>
> Last, but not least: are you using the default number of vnodes, 256?  The
> overhead of large number of vnodes (times the number of nodes), can be
> quite significant.  We've seen major improvements in repair runtime after
> switching from 256 to 16 vnodes on Cassandra version 3.0.


Is there a recommended procedure to switch the amount of vnodes ?

Regards,

Leo

On Thu, Jun 13, 2019 at 12:06 PM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Thu, Jun 13, 2019 at 10:36 AM R. T. 
> wrote:
>
>>
>> Well, actually by running cfstats I can see that the totaldiskspaceused
>> is about ~ 1.2 TB per node in the DC1 and ~ 1 TB per node in DC2. DC2 was
>> off for a while thats why there is a difference in space.
>>
>> I am using Cassandra 3.0.6 and
>> my stream_throughput_outbound_megabits_per_sec is th4e default setting so
>> according to my version is (200 Mbps or 25 MB/s)
>>
>
> And the other setting: compaction_throughput_mb_per_sec?  It is also
> highly relevant for repair performance, as streamed in files need to be
> compacted with the existing files on the nodes.  In our experience change
> in compaction throughput limit is almost linearly reflected by the repair
> run time.
>
> The default 16 MB/s is too limiting for any production grade setup, I
> believe.  We go as high as 90 MB/s on AWS EBS gp2 data volumes.  But don't
> take it as a gospel, I'd suggest you start increasing the setting (e.g. by
> doubling it) and observe how it affects repair performance (and client
> latencies).
>
> Have you tried with "parallel" instead of "DC parallel" mode?  The latter
> one is really poorly named and it actually means something else, as neatly
> highlighted in this SO answer: https://dba.stackexchange.com/a/175028
>
> Last, but not least: are you using the default number of vnodes, 256?  The
> overhead of large number of vnodes (times the number of nodes), can be
> quite significant.  We've seen major improvements in repair runtime after
> switching from 256 to 16 vnodes on Cassandra version 3.0.
>
> Cheers,
> --
> Alex
>
>

Re: very slow repair

[Oleksandr Shulgin]

On Thu, Jun 13, 2019 at 10:36 AM R. T. 
wrote:

>
> Well, actually by running cfstats I can see that the totaldiskspaceused is
> about ~ 1.2 TB per node in the DC1 and ~ 1 TB per node in DC2. DC2 was off
> for a while thats why there is a difference in space.
>
> I am using Cassandra 3.0.6 and
> my stream_throughput_outbound_megabits_per_sec is th4e default setting so
> according to my version is (200 Mbps or 25 MB/s)
>

And the other setting: compaction_throughput_mb_per_sec?  It is also highly
relevant for repair performance, as streamed in files need to be compacted
with the existing files on the nodes.  In our experience change in
compaction throughput limit is almost linearly reflected by the repair run
time.

The default 16 MB/s is too limiting for any production grade setup, I
believe.  We go as high as 90 MB/s on AWS EBS gp2 data volumes.  But don't
take it as a gospel, I'd suggest you start increasing the setting (e.g. by
doubling it) and observe how it affects repair performance (and client
latencies).

Have you tried with "parallel" instead of "DC parallel" mode?  The latter
one is really poorly named and it actually means something else, as neatly
highlighted in this SO answer: https://dba.stackexchange.com/a/175028

Last, but not least: are you using the default number of vnodes, 256?  The
overhead of large number of vnodes (times the number of nodes), can be
quite significant.  We've seen major improvements in repair runtime after
switching from 256 to 16 vnodes on Cassandra version 3.0.

Cheers,
--
Alex

Re: very slow repair

[R. T.]

Hi,

Thank you for your reply,

Well, actually by running cfstats I can see that the totaldiskspaceused is 
about ~ 1.2 TB per node in the DC1 and ~ 1 TB per node in DC2. DC2 was off for 
a while thats why there is a difference in space.

I am using Cassandra 3.0.6 and my stream_throughput_outbound_megabits_per_sec 
is th4e default setting so according to my version is (200 Mbps or 25 MB/s)

Cheers

‐‐‐ Original Message ‐‐‐
On Thursday, June 13, 2019 6:04 AM, Laxmikant Upadhyay 
 wrote:

> Few queries:
> 1. What is the cassandra version ?
> 2. is the size of table 4TB per node ?
> 3. What is the value of compaction_throughput_mb_per_sec and 
> stream_throughput_outbound_megabits_per_sec ?
>
> On Thu, Jun 13, 2019 at 5:06 AM R. T.  wrote:
>
>> Hi,
>>
>> I am trying to run a repair for first time a specific column family in 
>> specific keyspace and it seems that is going super slow.
>>
>> I have 6 nodes cluster with 2 Datacenters (RF 2) and the repair is a non 
>> incremental, DC parallel one. This column family is around 4 TB and it is 
>> written heavily (compared with other CF) so since it is going to take 2 
>> months (according ETA in Reaper) does that mean that when this repair will 
>> finish the entropy will be again high in this CF ?
>>
>> How I can speed up the process ? Is there any way to diagnose bottlenecs?
>>
>> Thank you,
>>
>> W
>
> --
>
> regards,
> Laxmikant Upadhyay

Re: very slow repair

[Laxmikant Upadhyay]

Few queries:
1. What is the cassandra version ?
2. is the size of table 4TB per node ?
3. What is the value of compaction_throughput_mb_per_sec and
stream_throughput_outbound_megabits_per_sec ?

On Thu, Jun 13, 2019 at 5:06 AM R. T. 
wrote:

> Hi,
>
> I am trying to run a repair for first time a specific column family in
> specific keyspace and it seems that is going super slow.
>
> I have 6 nodes cluster with 2 Datacenters (RF 2) and the repair is a non
> incremental, DC parallel one. This column family is around 4 TB and it is
> written heavily (compared with other CF) so since it is going to take 2
> months (according ETA in Reaper) does that mean that when this repair will
> finish the entropy will be again high in this CF ?
>
> How I can speed up the process ? Is there any way to diagnose bottlenecs?
>
> Thank you,
>
> W
>
>

-- 

regards,
Laxmikant Upadhyay

very slow repair

[R. T.]

Hi,

I am trying to run a repair for first time a specific column family in specific 
keyspace and it seems that is going super slow.

I have 6 nodes cluster with 2 Datacenters (RF 2) and the repair is a non 
incremental, DC parallel one. This column family is around 4 TB and it is 
written heavily (compared with other CF) so since it is going to take 2 months 
(according ETA in Reaper) does that mean that when this repair will finish the 
entropy will be again high in this CF ?

How I can speed up the process ? Is there any way to diagnose bottlenecs?

Thank you,

W

Re: Slow repair

[Gábor Auth]

Hi,

On Wed, Mar 15, 2017 at 11:35 AM Ben Slater 
wrote:

> When you say you’re running repair to “rebalance” do you mean to populate
> the new DC? If so, the normal/correct procedure is to use nodetool rebuild
> rather than repair.
>

Oh, thank you! :)

Bye,
Gábor Auth

>

Re: Slow repair

[siddharth verma]

Hi,
We did a similar thing when a new DC was added and had to populate it
according to altered replication of keyspace.
For repair we used Tickler approach rather than actual nodetool repair.
(using the blocking read repair feature in cassandra)

You can see
1. Ticker by ckalantzis : https://github.com/ckalantzis/cassTickler
2. Modified tickler ( written in java ) :
https://github.com/siddv29/eleventh-hour-repair
clone, mvn clean install, and run it on any machine

Regards

On Wed, Mar 15, 2017 at 3:57 PM, Ben Slater 
wrote:

> When you say you’re running repair to “rebalance” do you mean to populate
> the new DC? If so, the normal/correct procedure is to use nodetool rebuild
> rather than repair. See https://docs.datastax.com/
> en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html for
> the full details.
>
> Cheers
> Ben
>
> On Wed, 15 Mar 2017 at 21:14 Gábor Auth  wrote:
>
>> Hi,
>>
>> We are working with a two DCs Cassandra cluster (EU and US), so that the
>> distance is over 160 ms between them. I've added a new DC to this cluster,
>> modified the keyspace's replication factor and trying to rebalance it with
>> repair but the repair is very slow (over 10-15 minutes per node per
>> keyspace with ~40 column families). Is it normal with this network latency
>> or something wrong with the cluster or the network connection? :)
>>
>> [2017-03-15 05:52:38,255] Starting repair command #4, repairing keyspace
>> test20151222 with repair options (parallelism: parallel, primary range:
>> true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
>> [], hosts: [], # of ranges: 32)
>> [2017-03-15 05:54:11,913] Repair session 988bd850-0943-11e7-9c1f-f5ba092c6aea
>> for range [(-3328182031191101706,-3263206086630594139],
>> (-449681117114180865,-426983008087217811], 
>> (-4940101276128910421,-4726878962587262390],
>> (-4999008077542282524,-4940101276128910421]] finished (progress: 11%)
>> [2017-03-15 05:55:39,721] Repair session 9a6fda92-0943-11e7-9c1f-f5ba092c6aea
>> for range [(7538662821591320245,7564364667721298414],
>> (8095771383100385537,8112071444788258953], 
>> (-1625703837190283897,-1600176580612824092],
>> (-1075557915997532230,-1072724867906442440], (-9152
>> 563942239372475,-9123254980705325471], 
>> (7485905313674392326,7513617239634230698]]
>> finished (progress: 14%)
>> [2017-03-15 05:57:05,718] Repair session 9de181b1-0943-11e7-9c1f-f5ba092c6aea
>> for range [(-6471953894734787784,-6420063839816736750],
>> (1372322727565611879,1480899944406172322], 
>> (1176263633569625668,1177285361971054591],
>> (440549646067640682,491840653569315468], (-43128299
>> 75221321282,-4177428401237878410]] finished (progress: 17%)
>> [2017-03-15 05:58:39,997] Repair session a18bc500-0943-11e7-9c1f-f5ba092c6aea
>> for range [(5327651902976749177,5359189884199963589],
>> (-5362946313988105342,-5348008210198062914], 
>> (-5756557262823877856,-5652851311492822149],
>> (-5400778420101537991,-5362946313988105342], (668
>> 2536072120412021,6904193483670147322]] finished (progress: 20%)
>> [2017-03-15 05:59:11,791] Repair session a44f2ac2-0943-11e7-9c1f-f5ba092c6aea
>> for range [(952873612468870228,1042958763135655298], 
>> (558544893991295379,572114658167804730]]
>> finished (progress: 22%)
>> [2017-03-15 05:59:56,197] Repair session a5e13c71-0943-11e7-9c1f-f5ba092c6aea
>> for range [(1914238614647876002,1961526714897144472],
>> (3610056520286573718,3619622957324752442], 
>> (-3506227577233676363,-3504718440405535976],
>> (-4120686433235827731,-4098515820338981500], (56515
>> 94158011135924,5668698324546997949]] finished (progress: 25%)
>> [2017-03-15 06:00:45,610] Repair session a897a9e1-0943-11e7-9c1f-f5ba092c6aea
>> for range [(-9007733666337543056,-8979974976044921941]] finished
>> (progress: 28%)
>> [2017-03-15 06:01:58,826] Repair session a927b4e1-0943-11e7-9c1f-f5ba092c6aea
>> for range [(3599745202434925817,3608662806723095677],
>> (3390003128426746316,3391135639180043521], 
>> (3391135639180043521,3529019003015169892]]
>> finished (progress: 31%)
>> [2017-03-15 06:03:15,440] Repair session aae06160-0943-11e7-9c1f-f5ba092c6aea
>> for range [(-7542303048667795773,-7300899534947316960]] finished
>> (progress: 34%)
>> [2017-03-15 06:03:17,786] Repair completed successfully
>> [2017-03-15 06:03:17,787] Repair command #4 finished in 10 minutes 39
>> seconds
>>
>> Bye,
>> Gábor Auth
>>
>> --
>
>
> *Ben Slater*
>
> *Chief Product Officer *
>
>    
>
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose 

Re: Slow repair

[Ben Slater]

When you say you’re running repair to “rebalance” do you mean to populate
the new DC? If so, the normal/correct procedure is to use nodetool rebuild
rather than repair. See
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html
for
the full details.

Cheers
Ben

On Wed, 15 Mar 2017 at 21:14 Gábor Auth  wrote:

> Hi,
>
> We are working with a two DCs Cassandra cluster (EU and US), so that the
> distance is over 160 ms between them. I've added a new DC to this cluster,
> modified the keyspace's replication factor and trying to rebalance it with
> repair but the repair is very slow (over 10-15 minutes per node per
> keyspace with ~40 column families). Is it normal with this network latency
> or something wrong with the cluster or the network connection? :)
>
> [2017-03-15 05:52:38,255] Starting repair command #4, repairing keyspace
> test20151222 with repair options (parallelism: parallel, primary range:
> true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
> [], hosts: [], # of ranges: 32)
> [2017-03-15 05:54:11,913] Repair session
> 988bd850-0943-11e7-9c1f-f5ba092c6aea for range
> [(-3328182031191101706,-3263206086630594139],
> (-449681117114180865,-426983008087217811],
> (-4940101276128910421,-4726878962587262390],
> (-4999008077542282524,-4940101276128910421]] finished (progress: 11%)
> [2017-03-15 05:55:39,721] Repair session
> 9a6fda92-0943-11e7-9c1f-f5ba092c6aea for range
> [(7538662821591320245,7564364667721298414],
> (8095771383100385537,8112071444788258953],
> (-1625703837190283897,-1600176580612824092],
> (-1075557915997532230,-1072724867906442440], (-9152
> 563942239372475,-9123254980705325471],
> (7485905313674392326,7513617239634230698]] finished (progress: 14%)
> [2017-03-15 05:57:05,718] Repair session
> 9de181b1-0943-11e7-9c1f-f5ba092c6aea for range
> [(-6471953894734787784,-6420063839816736750],
> (1372322727565611879,1480899944406172322],
> (1176263633569625668,1177285361971054591],
> (440549646067640682,491840653569315468], (-43128299
> 75221321282,-4177428401237878410]] finished (progress: 17%)
> [2017-03-15 05:58:39,997] Repair session
> a18bc500-0943-11e7-9c1f-f5ba092c6aea for range
> [(5327651902976749177,5359189884199963589],
> (-5362946313988105342,-5348008210198062914],
> (-5756557262823877856,-5652851311492822149],
> (-5400778420101537991,-5362946313988105342], (668
> 2536072120412021,6904193483670147322]] finished (progress: 20%)
> [2017-03-15 05:59:11,791] Repair session
> a44f2ac2-0943-11e7-9c1f-f5ba092c6aea for range
> [(952873612468870228,1042958763135655298],
> (558544893991295379,572114658167804730]] finished (progress: 22%)
> [2017-03-15 05:59:56,197] Repair session
> a5e13c71-0943-11e7-9c1f-f5ba092c6aea for range
> [(1914238614647876002,1961526714897144472],
> (3610056520286573718,3619622957324752442],
> (-3506227577233676363,-3504718440405535976],
> (-4120686433235827731,-4098515820338981500], (56515
> 94158011135924,5668698324546997949]] finished (progress: 25%)
> [2017-03-15 06:00:45,610] Repair session
> a897a9e1-0943-11e7-9c1f-f5ba092c6aea for range
> [(-9007733666337543056,-8979974976044921941]] finished (progress: 28%)
> [2017-03-15 06:01:58,826] Repair session
> a927b4e1-0943-11e7-9c1f-f5ba092c6aea for range
> [(3599745202434925817,3608662806723095677],
> (3390003128426746316,3391135639180043521],
> (3391135639180043521,3529019003015169892]] finished (progress: 31%)
> [2017-03-15 06:03:15,440] Repair session
> aae06160-0943-11e7-9c1f-f5ba092c6aea for range
> [(-7542303048667795773,-7300899534947316960]] finished (progress: 34%)
> [2017-03-15 06:03:17,786] Repair completed successfully
> [2017-03-15 06:03:17,787] Repair command #4 finished in 10 minutes 39
> seconds
>
> Bye,
> Gábor Auth
>
> --


*Ben Slater*

*Chief Product Officer *

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Slow repair

[Gábor Auth]

Hi,

We are working with a two DCs Cassandra cluster (EU and US), so that the
distance is over 160 ms between them. I've added a new DC to this cluster,
modified the keyspace's replication factor and trying to rebalance it with
repair but the repair is very slow (over 10-15 minutes per node per
keyspace with ~40 column families). Is it normal with this network latency
or something wrong with the cluster or the network connection? :)

[2017-03-15 05:52:38,255] Starting repair command #4, repairing keyspace
test20151222 with repair options (parallelism: parallel, primary range:
true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
[], hosts: [], # of ranges: 32)
[2017-03-15 05:54:11,913] Repair session
988bd850-0943-11e7-9c1f-f5ba092c6aea for range
[(-3328182031191101706,-3263206086630594139],
(-449681117114180865,-426983008087217811],
(-4940101276128910421,-4726878962587262390],
(-4999008077542282524,-4940101276128910421]] finished (progress: 11%)
[2017-03-15 05:55:39,721] Repair session
9a6fda92-0943-11e7-9c1f-f5ba092c6aea for range
[(7538662821591320245,7564364667721298414],
(8095771383100385537,8112071444788258953],
(-1625703837190283897,-1600176580612824092],
(-1075557915997532230,-1072724867906442440], (-9152
563942239372475,-9123254980705325471],
(7485905313674392326,7513617239634230698]] finished (progress: 14%)
[2017-03-15 05:57:05,718] Repair session
9de181b1-0943-11e7-9c1f-f5ba092c6aea for range
[(-6471953894734787784,-6420063839816736750],
(1372322727565611879,1480899944406172322],
(1176263633569625668,1177285361971054591],
(440549646067640682,491840653569315468], (-43128299
75221321282,-4177428401237878410]] finished (progress: 17%)
[2017-03-15 05:58:39,997] Repair session
a18bc500-0943-11e7-9c1f-f5ba092c6aea for range
[(5327651902976749177,5359189884199963589],
(-5362946313988105342,-5348008210198062914],
(-5756557262823877856,-5652851311492822149],
(-5400778420101537991,-5362946313988105342], (668
2536072120412021,6904193483670147322]] finished (progress: 20%)
[2017-03-15 05:59:11,791] Repair session
a44f2ac2-0943-11e7-9c1f-f5ba092c6aea for range
[(952873612468870228,1042958763135655298],
(558544893991295379,572114658167804730]] finished (progress: 22%)
[2017-03-15 05:59:56,197] Repair session
a5e13c71-0943-11e7-9c1f-f5ba092c6aea for range
[(1914238614647876002,1961526714897144472],
(3610056520286573718,3619622957324752442],
(-3506227577233676363,-3504718440405535976],
(-4120686433235827731,-4098515820338981500], (56515
94158011135924,5668698324546997949]] finished (progress: 25%)
[2017-03-15 06:00:45,610] Repair session
a897a9e1-0943-11e7-9c1f-f5ba092c6aea for range
[(-9007733666337543056,-8979974976044921941]] finished (progress: 28%)
[2017-03-15 06:01:58,826] Repair session
a927b4e1-0943-11e7-9c1f-f5ba092c6aea for range
[(3599745202434925817,3608662806723095677],
(3390003128426746316,3391135639180043521],
(3391135639180043521,3529019003015169892]] finished (progress: 31%)
[2017-03-15 06:03:15,440] Repair session
aae06160-0943-11e7-9c1f-f5ba092c6aea for range
[(-7542303048667795773,-7300899534947316960]] finished (progress: 34%)
[2017-03-15 06:03:17,786] Repair completed successfully
[2017-03-15 06:03:17,787] Repair command #4 finished in 10 minutes 39
seconds

Bye,
Gábor Auth