Flush writer all time blocked
I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. http://pastebin.com/GAiu2q74 Thanks,SC
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. =Rob
RE: Flush writer all time blocked
Thanks Rob. Will it contribute to any performance problems? Thanks,SC Date: Thu, 29 Aug 2013 10:57:30 -0700 Subject: Re: Flush writer all time blocked From: rc...@eventbrite.com To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. =Rob
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 11:04 AM, S C as...@outlook.com wrote: Thanks Rob. Will it contribute to any performance problems? Yes, though perhaps more accurate would be that it is often an effect of performance problems. In general Cassandra flushes memtables because it believes it needs to. If it is unable to do so, that's usually bad. =Rob
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken
RE: Flush writer all time blocked
Ken, What queue size are you referring to? Thanks,SC From: ken.hanc...@schange.com Date: Thu, 29 Aug 2013 14:21:04 -0400 Subject: Re: Flush writer all time blocked To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken
Re: Flush writer all time blocked
On Thu, Aug 29, 2013 at 11:21 AM, Ken Hancock ken.hanc...@schange.comwrote: I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Periodic flush? Between 1.0 and 2.0, there is no time based flush mechanism for memtables. FWIW, I'm glad it is being added back in 2.0, as operators often want to be able to assert how long it will take for any given write to make it to disk. https://issues.apache.org/jira/browse/CASSANDRA-4237 (background at https://issues.apache.org/jira/browse/CASSANDRA-2427 ) =Rob
Re: Flush writer all time blocked
# the number of full memtables to allow pending flush, that is, # waiting for a writer thread. At a minimum, this should be set to # the maximum number of secondary indexes created on a single CF. memtable_flush_queue_size: 4 There was an interesting thread a while back: http://mail-archives.apache.org/mod_mbox/cassandra-user/201307.mbox/%3c17c39fe466076c46b6e83f129c7b19ce2e7ec...@hkxprd0310mb352.apcprd03.prod.outlook.com%3E On Thu, Aug 29, 2013 at 2:23 PM, S C as...@outlook.com wrote: Ken, What queue size are you referring to? Thanks, SC From: ken.hanc...@schange.com Date: Thu, 29 Aug 2013 14:21:04 -0400 Subject: Re: Flush writer all time blocked To: user@cassandra.apache.org On Thu, Aug 29, 2013 at 1:57 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Aug 29, 2013 at 10:49 AM, S C as...@outlook.com wrote: I see a high count All time blocked for Flush Writer on nodetool tpstats. Is it how many blocked ever since the server was online? Can somebody explain me what it is? I really appreciate it. Yes. Flush Writer thread pool is the thread pool responsible for the part of memtable flush that actually writes to disk. If you see it with a non-zero blocked number, you have at some time written to memory significantly faster than you could flush to disk. I don't think this is strictly true? There's also the periodic flush that can cause a storm of flushes if you have multiple column families. I sent out a query to the list last week on this topic but didn't get any responses -- I'm very interested in this topic as I've had to set my queue size fairly large to avoid this issue. Ken -- Ken Hancock | System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC Office: +1 (978) 889-3329 | ken.hanc...@schange.com | hancockks | hancockks This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International.
