This may be your problem:
https://issues.apache.org/jira/browse/CASSANDRA-1358
The message deserializer executor is being created with a core pool size of
1. Since it uses a queue with unbounded capacity new requests are always
queued and the thread pool never grows. So the message deserializer
No, MDP is backing up because Row-Read-Stage [the stage after MDP on
reads] is full at 4096, meaning you're not able to process reads as
quickly as the requests are coming in.
On Wed, Aug 4, 2010 at 2:21 PM, Mike Malone m...@simplegeo.com wrote:
This may be your
problem:
So after 4096 messages get pushed on the row-read-stage queue (or any other
multiThreadedStage) the deserializer basically becomes a single-threaded
blocking queue that prevents any other inter-node RPC from occurring..?
Sounds like it's a problem either way. If the row read stage is what's
backed
Let's centralize that conversation over on
https://issues.apache.org/jira/browse/CASSANDRA-1358
On Wed, Aug 4, 2010 at 4:32 PM, Mike Malone m...@simplegeo.com wrote:
So after 4096 messages get pushed on the row-read-stage queue (or any other
multiThreadedStage) the deserializer basically
The output of htop shows threads as procs with a breakdown of how much cpu
/etc per thread (in ncurses color!). All of these Java procs are just Java
threads of only 1 instance of Cassandra per Server.
On Sat, Jul 31, 2010 at 3:45 PM, Benjamin Black b...@b3k.us wrote:
Sorry, I just noticed: are
When you can't get the number of threads, that means you have way too many
running (8,000+) usually.
Try running `ps -eLf | grep cassandra`. How many threads?
-Chris
On Jul 29, 2010, at 8:40 PM, Dathan Pattishall wrote:
To Follow up on this thread. I blew away the data for my entire
Off topic, but what was this tool which prints per cpu utilization?
- Garo
On Mon, Jul 26, 2010 at 10:22 PM, Dathan Pattishall datha...@gmail.com wrote:
But the 16 cores are hardly utilized. Which indicates to me there is some
bad thread thrashing, but why?
1 [|
@Garo htop.
On Mon, Jul 26, 2010 at 11:28 PM, Juho Mäkinen juho.maki...@gmail.comwrote:
Off topic, but what was this tool which prints per cpu utilization?
- Garo
On Mon, Jul 26, 2010 at 10:22 PM, Dathan Pattishall datha...@gmail.com
wrote:
But the 16 cores are hardly utilized. Which
On Mon, Jul 26, 2010 at 8:30 PM, Jonathan Ellis jbel...@gmail.com wrote:
MDP is backing up because RRS is full at 4096. This means you're not
able to process reads as quickly as the requests are coming in. Make
whatever is doing those reads be less aggressive.
So, for cassandra to function
disk may help some.
It’s not so much about “Cassandra’s” read rate but what your hardware can
manage.
/Justus
*Från:* Dathan Pattishall [mailto:datha...@gmail.com]
*Skickat:* den 27 juli 2010 08:56
*Till:* user@cassandra.apache.org
*Ämne:* Re: what causes MESSAGE-DESERIALIZER-POOL
Ah, the weird thing is I/O is assumed to be the limiting factor, but iops on
the box was very low. Service time and atime very low, and the data access
was only 6MB a second. With all of this, I'm tending to believe that the
problem may be someplace else.
You vmstat output shows idle and wait
It's usually I/O which causes backup in MESSAGE-DESERIALIZER-POOL. You
should check iostat and see what it looks like. It may be that you
need more nodes in order to deal with the read/write rate. You can also
use JMX to get latency values on reads and writes and see if the backup
has a
MESSAGE-DESERIALIZER-POOL 1 682008 1818682
On the (unverified by me) assumption that disk reads en up in this
stage as Anthony indicates, and looking at the above line and the
iostat output, makes me want to suggest that maybe the read
concurrency is set to 1 and you need to
Hpw does one set/get the read-concurrency? Also which read-concurrency, for
which pool? According to the threads tab in jconsole read-concurrency could
be among different pools. There are row-read-stag threads, lots of write
threads, row mutation stage etc.
On Mon, Jul 26, 2010 at 12:56 PM,
Okay so why would the pending be so large? I looked at iostat and the
diskload is not bad at all. The service times /etc are all good.
If it's not disk based I/O then is it thread thrashing? If so which thread
pool should I monitor more closely.
MESSAGE-DESERIALIZER-POOL:1
which would indicate
MDP is backing up because RRS is full at 4096. This means you're not
able to process reads as quickly as the requests are coming in. Make
whatever is doing those reads be less aggressive.
As to why the reads are slow in the first place, usually this means
you are disk i/o bound. Posting your
16 matches
Mail list logo
