Re: Mutation Stage does not finish
Hello, The jstack output can be seen in : http://pastebin.com/LXnNyY3U. I run the tpstats today and always get the same output: Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 *MutationStage32 5832690042 0 0* ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemoryMeter 0 0 98 0 0 MemtablePostFlusher 0 0 7 0 0 FlushWriter 0 0 5 0 0 MiscStage 0 0 0 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0 0 0 0 The OpCenter show the following status: Status: Active - Starting Gossip:Down Thrift:Down Native Transport: Down Pending Tasks: 0 Thanks Eduardo On Wed, Sep 10, 2014 at 10:30 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: No, is still running the Mutation Stage. If you're sure that it is not receiving Hinted Handoff, then the only mutations in question can be from the replay of the commit log. The commit log should take less than forever to replay. =Rob
Re: Mutation Stage does not finish
Robert/Elliot. I deleted commit logs, restarted cassandra and finally the node is up. Thanks for helps! Regards. Eduardo On Thu, Sep 11, 2014 at 12:08 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: Hello, The jstack output can be seen in : http://pastebin.com/LXnNyY3U. I run the tpstats today and always get the same output: Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 *MutationStage32 5832690042 0 0* ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemoryMeter 0 0 98 0 0 MemtablePostFlusher 0 0 7 0 0 FlushWriter 0 0 5 0 0 MiscStage 0 0 0 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0 0 0 0 The OpCenter show the following status: Status: Active - Starting Gossip:Down Thrift:Down Native Transport: Down Pending Tasks: 0 Thanks Eduardo On Wed, Sep 10, 2014 at 10:30 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: No, is still running the Mutation Stage. If you're sure that it is not receiving Hinted Handoff, then the only mutations in question can be from the replay of the commit log. The commit log should take less than forever to replay. =Rob
Re: Mutation Stage does not finish
On Thu, Sep 11, 2014 at 10:34 AM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: I deleted commit logs, restarted cassandra and finally the node is up. Do you have some crazy workload where you do a huge amount of delete or something? Replaying a commitlog should not take longer than a few tens of minutes in the worst case scenario. =Rob
Re: Mutation Stage does not finish
yes we have a huge amount insert that can be repeated, now we are working in a new data model On Thu, Sep 11, 2014 at 2:54 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Sep 11, 2014 at 10:34 AM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: I deleted commit logs, restarted cassandra and finally the node is up. Do you have some crazy workload where you do a huge amount of delete or something? Replaying a commitlog should not take longer than a few tens of minutes in the worst case scenario. =Rob
Mutation Stage does not finish
Hello, I have a node that is in MutationStage for the last 5 hours. Actually the node is *down*. The pendings task go from 776 to 110 and then to 964. There are some way to finish this stage? The last heavy write workload was 5 days ago. Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 *MutationStage32 7762583249 0 0* ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemoryMeter 0 0107 0 0 MemtablePostFlusher 0 0 9 0 0 FlushWriter 0 0 7 0 0 MiscStage 0 0 0 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0 0 0 0 Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 *MutationStage32 1102602365 0 0* ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemoryMeter 0 0107 0 0 MemtablePostFlusher 0 0 11 0 0 FlushWriter 0 0 9 0 0 MiscStage 0 0 0 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0 0 0 0 Pool NameActive Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 *MutationStage32 9642602536 0 0* ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 0 0 0 MemoryMeter 0 0107 0 0 MemtablePostFlusher 0 0 11 0 0 FlushWriter 0 0 9 0 0 MiscStage 0 0 0 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0 0 0 0
Re: Mutation Stage does not finish
On Wed, Sep 10, 2014 at 11:38 AM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: Actually the node is *down*. The node can't be that down if it's printing tpstats... https://issues.apache.org/jira/browse/CASSANDRA-4162 ? =Rob
Re: Mutation Stage does not finish
On Wed, Sep 10, 2014 at 12:03 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: Yes, the tpstats is printing. The Opcenter show the node down. Have you recently restarted it or anything? If not, try doing so? =Rob
Re: Mutation Stage does not finish
On Wed, Sep 10, 2014 at 12:16 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: Yes, I restarted the node becaouse the write latency was 2500 ms, when usually is 5 ms. And did that help? =Rob
Re: Mutation Stage does not finish
Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: No, is still running the Mutation Stage. If you're sure that it is not receiving Hinted Handoff, then the only mutations in question can be from the replay of the commit log. The commit log should take less than forever to replay. =Rob
