Hi Asad,
You can increase it by 2 at a time. For example if its currently 2, try
increasing it to 4 and retest.
We flush 5-6 tables at a time and use 3 memtable_flush_writers. It works
great!! There were dropped mutations when it was set to one. The idea is to
make sure that writes are not blocked.
ThanksAnuj
Sent from Yahoo Mail on Android
On Fri, 21 Jul 2017 at 20:04, ZAIDI, ASAD A<az1...@att.com> wrote:
#yiv0831784205 #yiv0831784205 -- _filtered #yiv0831784205 {panose-1:2 4 5 3 5 4
6 3 2 4;} _filtered #yiv0831784205 {font-family:Calibri;panose-1:2 15 5 2 2 2 4
3 2 4;}#yiv0831784205 #yiv0831784205 p.yiv0831784205MsoNormal, #yiv0831784205
li.yiv0831784205MsoNormal, #yiv0831784205 div.yiv0831784205MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv0831784205 a:link,
#yiv0831784205 span.yiv0831784205MsoHyperlink
{color:blue;text-decoration:underline;}#yiv0831784205 a:visited, #yiv0831784205
span.yiv0831784205MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}#yiv0831784205
p.yiv0831784205msonormal, #yiv0831784205 li.yiv0831784205msonormal,
#yiv0831784205 div.yiv0831784205msonormal
{margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205
p.yiv0831784205msochpdefault, #yiv0831784205 li.yiv0831784205msochpdefault,
#yiv0831784205 div.yiv0831784205msochpdefault
{margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205
span.yiv0831784205msohyperlink {}#yiv0831784205
span.yiv0831784205msohyperlinkfollowed {}#yiv0831784205
span.yiv0831784205emailstyle17 {}#yiv0831784205 p.yiv0831784205msonormal1,
#yiv0831784205 li.yiv0831784205msonormal1, #yiv0831784205
div.yiv0831784205msonormal1
{margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv0831784205
span.yiv0831784205msohyperlink1
{color:#0563C1;text-decoration:underline;}#yiv0831784205
span.yiv0831784205msohyperlinkfollowed1
{color:#954F72;text-decoration:underline;}#yiv0831784205
span.yiv0831784205emailstyle171 {color:windowtext;}#yiv0831784205
p.yiv0831784205msochpdefault1, #yiv0831784205 li.yiv0831784205msochpdefault1,
#yiv0831784205 div.yiv0831784205msochpdefault1
{margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv0831784205
span.yiv0831784205EmailStyle27 {color:#1F497D;}#yiv0831784205
.yiv0831784205MsoChpDefault {} _filtered #yiv0831784205 {margin:1.0in 1.0in
1.0in 1.0in;}#yiv0831784205 div.yiv0831784205WordSection1 {}#yiv0831784205
Thank you for your reply. I’ll increase memTable_flush_writes and report back
if it helps.
Is there any formula that we can use to arrive at correct number of
memTable_flush_writers ? or the exercise would windup be like “try and error”
taking much time to arrive at some number that may not be optimal.
Thank you again.
From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Thursday, July 20, 2017 12:17 PM
To: ZAIDI, ASAD A <az1...@att.com>; user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node
timeout
Hi Asad,
You can do following things:
1.Increase memtable_flush_writers especially if you have a write heavy load.
2.Make sure there are no big gc pauses on your nodes. If yes, go for heap
tuning.
Please let us know whether above measures fixed your problem or not.
Thanks
Anuj
Sent from Yahoo Mail on Android
On Thu, 20 Jul 2017 at 20:57, ZAIDI, ASAD A
<az1...@att.com> wrote:
Hello Folks –
I’m using apache-cassandra 2.2.8.
I see many messages like below in my system.log file. In Cassandra.yaml file [
cross_node_timeout: true] is set and NTP server is also running correcting
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff
in tpstats output though there are bunch of MUTATIONS dropped observed.
<start timeout message >
INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 -
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and
2152 for cross node timeout
<end timeout message >
I’m seeking help here if you please let me know what I need to check in order
to address these cross node timeouts.
Thank you,
Asad
