Re: spikes in blocked native transport requests

[Nate McCall]

See the details on: https://issues.apache.org/jira/browse/CASSANDRA-11363

You may need to add -Dcassandra.max_queued_native_transport_requests=4096
as a startup parameter. YMMV though, I suggest reading through the above to
get a complete picture.

On Mon, Mar 20, 2017 at 11:10 PM, Roland Otta 
wrote:
>
> well. i checked it now.
>
> we have some stw collections from 100 to 200ms every 5 to 60 seconds.
> i am not sure whether the blocked threads are related to that but anyway
these pauses are too long for low latency applications.
>
> so i wil check gc tuning first and will check afterwards whether the
blocked threads still exist afterwards.
>
>
>
> On Mon, 2017-03-20 at 08:55 +0100, benjamin roth wrote:
>
> Did you check STW GCs?
> You can do that with 'nodetool gcstats', by looking at the gc.log or
observing GC related JMX metrics.
>
> 2017-03-20 8:52 GMT+01:00 Roland Otta :
>
> we have a datacenter which is currently used exlusively for spark batch
> jobs.
>
> in case batch jobs are running against that environment we can see very
> high peaks in blocked native transport requests (up to 10k / minute).
>
> i am concerned because i guess that will slow other queries (in case
> other applications are going to use that dc as well).
>
> i already tried increasing native_transport_max_threads +
> concurrent_reads without success.
>
> during the jobs i cant find any resource limitiations on my hardware
> (iops, disk usage, cpu, ... is fine).
>
> am i missing something? any suggestions how to cope with that?
>
> br//
> roland
>
>
>
>
>



--
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: spikes in blocked native transport requests

[benjamin roth]

Did you check STW GCs?
You can do that with 'nodetool gcstats', by looking at the gc.log or
observing GC related JMX metrics.

2017-03-20 8:52 GMT+01:00 Roland Otta :

> we have a datacenter which is currently used exlusively for spark batch
> jobs.
>
> in case batch jobs are running against that environment we can see very
> high peaks in blocked native transport requests (up to 10k / minute).
>
> i am concerned because i guess that will slow other queries (in case
> other applications are going to use that dc as well).
>
> i already tried increasing native_transport_max_threads +
> concurrent_reads without success.
>
> during the jobs i cant find any resource limitiations on my hardware
> (iops, disk usage, cpu, ... is fine).
>
> am i missing something? any suggestions how to cope with that?
>
> br//
> roland
>
>
>

spikes in blocked native transport requests

[Roland Otta]

we have a datacenter which is currently used exlusively for spark batch
jobs.

in case batch jobs are running against that environment we can see very
high peaks in blocked native transport requests (up to 10k / minute).

i am concerned because i guess that will slow other queries (in case
other applications are going to use that dc as well).

i already tried increasing native_transport_max_threads +
concurrent_reads without success.

during the jobs i cant find any resource limitiations on my hardware
(iops, disk usage, cpu, ... is fine).

am i missing something? any suggestions how to cope with that?

br//
roland