Re: node decommission throttled

2016-12-08 Thread Eric Evans 
On Thu, Dec 8, 2016 at 3:27 AM, Aleksandr Ivanov  wrote:
> On sending side no high CPU/IO/etc utilization.
> But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and
> it just doesn't scale by design since "Each stream is a single thread"
> (http://www.mail-archive.com/user@cassandra.apache.org/msg42095.html)

Right, that is your bottleneck.  There is no per-host concurrency; I'm
afraid there is not much you can do about it at the moment.

https://issues.apache.org/jira/browse/CASSANDRA-4663 might be relevant

Cheers,

-- 
Eric Evans
john.eric.ev...@gmail.com

Re: node decommission throttled

2016-12-08 Thread Aleksandr Ivanov 
Nope, no MVs

On Thu, Dec 8, 2016 at 11:31 AM, Benjamin Roth 
wrote:

> Just an educated guess: you have materialized Views? They are known to
> Stream very slow
>
> Am 08.12.2016 10:28 schrieb "Aleksandr Ivanov" :
>
>> Yes, I use compression.
>> Tried without and it gave ~15% increase in speed, but is still too low
>> (~35Mbps)
>>
>> On sending side no high CPU/IO/etc utilization.
>> But on receiving node I see that one "STREAM-IN" thread takes 100% CPU
>> and it just doesn't scale by design since "Each stream is a single
>> thread" (http://www.mail-archive.com/user@cassandra.apache.org/msg42
>> 095.html)
>>
>>
>>
>>> > I'm trying to decommission one C* node from 6 nodes cluster and see
>>> that
>>> > outbound network traffic on this node doesn't go over ~30Mb/s.
>>> > Looks like it is throttled somewhere in C*
>>>
>>> Do you use compression?  Try taking a thread dump and see what the
>>> utilization of the sending threads are.
>>>
>>>
>>> --
>>> Eric Evans
>>> john.eric.ev...@gmail.com
>>>
>>
>>

Re: node decommission throttled

2016-12-08 Thread Benjamin Roth 
Just an educated guess: you have materialized Views? They are known to
Stream very slow

Am 08.12.2016 10:28 schrieb "Aleksandr Ivanov" :

> Yes, I use compression.
> Tried without and it gave ~15% increase in speed, but is still too low
> (~35Mbps)
>
> On sending side no high CPU/IO/etc utilization.
> But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and
> it just doesn't scale by design since "Each stream is a single thread" (
> http://www.mail-archive.com/user@cassandra.apache.org/msg42095.html)
>
>
>
>> > I'm trying to decommission one C* node from 6 nodes cluster and see that
>> > outbound network traffic on this node doesn't go over ~30Mb/s.
>> > Looks like it is throttled somewhere in C*
>>
>> Do you use compression?  Try taking a thread dump and see what the
>> utilization of the sending threads are.
>>
>>
>> --
>> Eric Evans
>> john.eric.ev...@gmail.com
>>
>
>

Re: node decommission throttled

2016-12-08 Thread Aleksandr Ivanov 
On sending side no high CPU/IO/etc utilization.

But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and
it just doesn't scale by design since "Each stream is a single thread" (
http://www.mail-archive.com/user@cassandra.apache.org/msg42095.html)


Maybe your System cannot Stream faster. Is your cpu or hd/ssd fully
> utilized?
>
> Am 07.12.2016 16:07 schrieb "Eric Evans" :
>
>> On Tue, Dec 6, 2016 at 9:54 AM, Aleksandr Ivanov 
>> wrote:
>> > I'm trying to decommission one C* node from 6 nodes cluster and see that
>> > outbound network traffic on this node doesn't go over ~30Mb/s.
>> > Looks like it is throttled somewhere in C*
>>
>> Do you use compression?  Try taking a thread dump and see what the
>> utilization of the sending threads are.
>>
>>
>> --
>> Eric Evans
>> john.eric.ev...@gmail.com
>>
>

Re: node decommission throttled

2016-12-08 Thread Aleksandr Ivanov 
Yes, I use compression.
Tried without and it gave ~15% increase in speed, but is still too low
(~35Mbps)

On sending side no high CPU/IO/etc utilization.
But on receiving node I see that one "STREAM-IN" thread takes 100% CPU and
it just doesn't scale by design since "Each stream is a single thread" (
http://www.mail-archive.com/user@cassandra.apache.org/msg42095.html)



> > I'm trying to decommission one C* node from 6 nodes cluster and see that
> > outbound network traffic on this node doesn't go over ~30Mb/s.
> > Looks like it is throttled somewhere in C*
>
> Do you use compression?  Try taking a thread dump and see what the
> utilization of the sending threads are.
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: node decommission throttled

2016-12-07 Thread Benjamin Roth 
Maybe your System cannot Stream faster. Is your cpu or hd/ssd fully
utilized?

Am 07.12.2016 16:07 schrieb "Eric Evans" :

> On Tue, Dec 6, 2016 at 9:54 AM, Aleksandr Ivanov  wrote:
> > I'm trying to decommission one C* node from 6 nodes cluster and see that
> > outbound network traffic on this node doesn't go over ~30Mb/s.
> > Looks like it is throttled somewhere in C*
>
> Do you use compression?  Try taking a thread dump and see what the
> utilization of the sending threads are.
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: node decommission throttled

2016-12-07 Thread Eric Evans 
On Tue, Dec 6, 2016 at 9:54 AM, Aleksandr Ivanov  wrote:
> I'm trying to decommission one C* node from 6 nodes cluster and see that
> outbound network traffic on this node doesn't go over ~30Mb/s.
> Looks like it is throttled somewhere in C*

Do you use compression?  Try taking a thread dump and see what the
utilization of the sending threads are.


-- 
Eric Evans
john.eric.ev...@gmail.com