Re: Re: streaming stuck on joining a node with TBs of data

2020-08-05 Thread onmstester onmstester
OK. Thanks 




I'm using STCS.



Anyway, IMHO, this is one of the main bottlenecks for using big/dense node in 
Cassandra (which reduces cluster size and data center costs) and it could be 
almost solved (at least for me), if we could reduce number of sstables at 
receiver side (either by sending bigger sstables at sending side or by merging 
sstables in memtable at receiving side)



(Just fixed a wrong word in my previous question)


 On Wed, 05 Aug 2020 10:02:51 +0430 onmstester onmstester 
<mailto:onmstes...@zoho.com.INVALID> wrote 


OK. Thanks

I'm using STCS.

Anyway, IMHO, this is one of the main bottlenecks for using big/dense node in 
Cassandra (which reduces cluster size and data center costs) and it could be 
almost solved (at least for me), if we could eliminate number of sstables at 
receiver side (either by sending bigger sstables at sending side or by merging 
sstables in memtable at receiving side)


Sent using https://www.zoho.com/mail/




 On Mon, 03 Aug 2020 19:17:33 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 


Memtable really isn't involved here, each data file is copied over as-is and 
turned into a new data file, it doesn't read into the memtable (though it does 
deserialize and re-serialize, which temporarily has it in memory, but isn't in 
the memtable itself).



You can cut down on the number of data files copied in by using fewer vnodes, 
or by changing your compaction parameters (e.g. if you're using LCS, change 
sstable size from 160M to something higher), but there's no magic to join / 
compact those data files on the sending side before sending.




On Mon, Aug 3, 2020 at 4:15 AM onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:

IMHO (reading system.log) each streamed-in file from any node would be write 
down as a separate sstable to the disk and won't be wait in memtable until 
enough amount of memtable has been created inside memory, so there would be 
more compactions because of multiple small sstables. Is there any configuration 
in cassandra to force streamed-in to pass memtable-sstable cycle, to have 
bigger sstables at first place?



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: onmstester onmstester <mailto:onmstes...@zoho.com.INVALID>
To: "user"<mailto:user@cassandra.apache.org>
Date: Sun, 02 Aug 2020 08:35:30 +0430
Subject: Re: streaming stuck on joining a node with TBs of data
 Forwarded message 



Thanks Jeff,



Already used netstats and it only shows that streaming from a single node 
remained and stuck and bunch of dropped messages, next time i will check 
tpstats too.

Currently i stopped the joining/stucked node, make the auto_bootstrap false and 
started the node and its UN now, is this OK too?



What about streaming tables one by one, any idea?



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 21:44:09 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 





Nodetool tpstats and netstats should give you a hint why it’s not joining



If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).





On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/

Re: Re: streaming stuck on joining a node with TBs of data

2020-08-04 Thread onmstester onmstester
OK. Thanks

I'm using STCS.
Anyway, IMHO, this is one of the main bottlenecks for using big/dense node in 
Cassandra (which reduces cluster size and data center costs) and it could be 
almost solved (at least for me), if we could eliminate number of sstables at 
receiver side (either by sending bigger sstables at sending side or by merging 
sstables in memtable at receiving side)

Sent using https://www.zoho.com/mail/




 On Mon, 03 Aug 2020 19:17:33 +0430 Jeff Jirsa  wrote 


Memtable really isn't involved here, each data file is copied over as-is and 
turned into a new data file, it doesn't read into the memtable (though it does 
deserialize and re-serialize, which temporarily has it in memory, but isn't in 
the memtable itself).



You can cut down on the number of data files copied in by using fewer vnodes, 
or by changing your compaction parameters (e.g. if you're using LCS, change 
sstable size from 160M to something higher), but there's no magic to join / 
compact those data files on the sending side before sending.




On Mon, Aug 3, 2020 at 4:15 AM onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:

IMHO (reading system.log) each streamed-in file from any node would be write 
down as a separate sstable to the disk and won't be wait in memtable until 
enough amount of memtable has been created inside memory, so there would be 
more compactions because of multiple small sstables. Is there any configuration 
in cassandra to force streamed-in to pass memtable-sstable cycle, to have 
bigger sstables at first place?



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: onmstester onmstester <mailto:onmstes...@zoho.com.INVALID>
To: "user"<mailto:user@cassandra.apache.org>
Date: Sun, 02 Aug 2020 08:35:30 +0430
Subject: Re: streaming stuck on joining a node with TBs of data
 Forwarded message 



Thanks Jeff,



Already used netstats and it only shows that streaming from a single node 
remained and stuck and bunch of dropped messages, next time i will check 
tpstats too.

Currently i stopped the joining/stucked node, make the auto_bootstrap false and 
started the node and its UN now, is this OK too?



What about streaming tables one by one, any idea?



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 21:44:09 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 





Nodetool tpstats and netstats should give you a hint why it’s not joining



If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).





On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/

Re: Re: streaming stuck on joining a node with TBs of data

2020-08-03 Thread Jeff Jirsa
Memtable really isn't involved here, each data file is copied over as-is
and turned into a new data file, it doesn't read into the memtable (though
it does deserialize and re-serialize, which temporarily has it in memory,
but isn't in the memtable itself).

You can cut down on the number of data files copied in by using fewer
vnodes, or by changing your compaction parameters (e.g. if you're using
LCS, change sstable size from 160M to something higher), but there's no
magic to join / compact those data files on the sending side before sending.


On Mon, Aug 3, 2020 at 4:15 AM onmstester onmstester
 wrote:

> IMHO (reading system.log) each streamed-in file from any node would be
> write down as a separate sstable to the disk and won't be wait in memtable
> until enough amount of memtable has been created inside memory, so there
> would be more compactions because of multiple small sstables. Is there any
> configuration in cassandra to force streamed-in to pass memtable-sstable
> cycle, to have bigger sstables at first place?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  Forwarded message 
> From: onmstester onmstester 
> To: "user"
> Date: Sun, 02 Aug 2020 08:35:30 +0430
> Subject: Re: streaming stuck on joining a node with TBs of data
>  Forwarded message 
>
> Thanks Jeff,
>
> Already used netstats and it only shows that streaming from a single node
> remained and stuck and bunch of dropped messages, next time i will check
> tpstats too.
> Currently i stopped the joining/stucked node, make the auto_bootstrap
> false and started the node and its UN now, is this OK too?
>
> What about streaming tables one by one, any idea?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  On Sat, 01 Aug 2020 21:44:09 +0430 *Jeff Jirsa  >* wrote 
>
>
> Nodetool tpstats and netstats should give you a hint why it’s not joining
>
> If you don’t care about consistency and you just want it joined in its
> current form (which is likely strictly incorrect but I get it), “nodetool
> disablegossip && nodetool enablegossip” in rapid succession (must be less
> than 30 seconds in between commands) will PROBABLY change it from joining
> to normal (unclean, unsafe, do this at your own risk).
>
>
> On Jul 31, 2020, at 11:46 PM, onmstester onmstester <
> onmstes...@zoho.com.invalid> wrote:
>
> 
> No Secondary index, No SASI, No materialized view
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  On Sat, 01 Aug 2020 11:02:54 +0430 *Jeff Jirsa  >* wrote 
>
> Are there secondary indices involved?
>
> On Jul 31, 2020, at 10:51 PM, onmstester onmstester <
> onmstes...@zoho.com.invalid> wrote:
>
> 
> Hi,
>
> I'm going to join multiple new nodes to already existed and running
> cluster. Each node should stream in >2TB of data, and it took a few days
> (with 500Mb streaming) to almost get finished. But it stuck on streaming-in
> from one final node, but i can not see any bottleneck on any side (source
> or destination node), the only problem is 400 pending compactions on
> joining node, which i disabled auto_compaction, but no improvement.
>
> 1. How can i safely stop streaming/joining the new node and make it UN,
> then run repair on the node?
> 2. On bootstrap a new node, multiple tables would be streamed-in
> simultaneously and i think that this would increase number of compactions
> in compare with a scenario that "the joining node first stream-in one table
> then switch to another one and etc". Am i right and this would decrease
> compactions? If so, is there a config or hack in cassandra to force that?
>
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>
>
>
>
>
>
>


Fwd: Re: streaming stuck on joining a node with TBs of data

2020-08-03 Thread onmstester onmstester
IMHO (reading system.log) each streamed-in file from any node would be write 
down as a separate sstable to the disk and won't be wait in memtable until 
enough amount of memtable has been created inside memory, so there would be 
more compactions because of multiple small sstables. Is there any configuration 
in cassandra to force streamed-in to pass memtable-sstable cycle, to have 
bigger sstables at first place?



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: onmstester onmstester 
To: "user"
Date: Sun, 02 Aug 2020 08:35:30 +0430
Subject: Re: streaming stuck on joining a node with TBs of data
 Forwarded message 



Thanks Jeff,



Already used netstats and it only shows that streaming from a single node 
remained and stuck and bunch of dropped messages, next time i will check 
tpstats too.

Currently i stopped the joining/stucked node, make the auto_bootstrap false and 
started the node and its UN now, is this OK too?



What about streaming tables one by one, any idea?



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 21:44:09 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 





Nodetool tpstats and netstats should give you a hint why it’s not joining



If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).





On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa <mailto:jji...@gmail.com> 
wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
<mailto:onmstes...@zoho.com.invalid> wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/

Re: streaming stuck on joining a node with TBs of data

2020-08-01 Thread onmstester onmstester
Thanks Jeff,



Already used netstats and it only shows that streaming from a single node 
remained and stuck and bunch of dropped messages, next time i will check 
tpstats too.

Currently i stopped the joining/stucked node, make the auto_bootstrap false and 
started the node and its UN now, is this OK too?



What about streaming tables one by one, any idea?

Sent using https://www.zoho.com/mail/




 On Sat, 01 Aug 2020 21:44:09 +0430 Jeff Jirsa  wrote 



Nodetool tpstats and netstats should give you a hint why it’s not joining

If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).



On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
 wrote:



No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa  
wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
 wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/

Re: streaming stuck on joining a node with TBs of data

2020-08-01 Thread Jeff Jirsa

Nodetool tpstats and netstats should give you a hint why it’s not joining

If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).


> On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
>  wrote:
> 
> 
> No Secondary index, No SASI, No materialized view
> 
> Sent using Zoho Mail
> 
> 
> 
>  On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa  wrote 
> 
> 
> Are there secondary indices involved? 
> 
> On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
>  wrote:
> 
> 
> Hi,
> 
> I'm going to join multiple new nodes to already existed and running cluster. 
> Each node should stream in >2TB of data, and it took a few days (with 500Mb 
> streaming) to almost get finished. But it stuck on streaming-in from one 
> final node, but i can not see any bottleneck on any side (source or 
> destination node), the only problem is 400 pending compactions on joining 
> node, which i disabled auto_compaction, but no improvement.
> 
> 1. How can i safely stop streaming/joining the new node and make it UN, then 
> run repair on the node?
> 2. On bootstrap a new node, multiple tables would be streamed-in 
> simultaneously and i think that this would increase number of compactions in 
> compare with a scenario that "the joining node first stream-in one table then 
> switch to another one and etc". Am i right and this would decrease 
> compactions? If so, is there a config or hack in cassandra to force that?
> 
> 
> Sent using Zoho Mail
> 
> 
> 
> 
> 
> 


Re: streaming stuck on joining a node with TBs of data

2020-07-31 Thread onmstester onmstester
No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa  wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
 wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/

Re: streaming stuck on joining a node with TBs of data

2020-07-31 Thread Jeff Jirsa
Are there secondary indices involved? 

> On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
>  wrote:
> 
> 
> Hi,
> 
> I'm going to join multiple new nodes to already existed and running cluster. 
> Each node should stream in >2TB of data, and it took a few days (with 500Mb 
> streaming) to almost get finished. But it stuck on streaming-in from one 
> final node, but i can not see any bottleneck on any side (source or 
> destination node), the only problem is 400 pending compactions on joining 
> node, which i disabled auto_compaction, but no improvement.
> 
> 1. How can i safely stop streaming/joining the new node and make it UN, then 
> run repair on the node?
> 2. On bootstrap a new node, multiple tables would be streamed-in 
> simultaneously and i think that this would increase number of compactions in 
> compare with a scenario that "the joining node first stream-in one table then 
> switch to another one and etc". Am i right and this would decrease 
> compactions? If so, is there a config or hack in cassandra to force that?
> 
> 
> Sent using Zoho Mail
> 
> 
> 
>