Re: rebuild constantly fails, 3.11

[Micha]

the nodes have 32G ram, there are no other processes running.

Thanks for the info about the g1gc.
I used bootstrap resume to finish the bootstrap, then added another two
nodes.
This worked, but I saw in munin a constantly rising memory consumption
during streaming, while on the other nodes there was a constantly amount
of memory used. Nothing was killed this time, since the streaming took
much less time to complete.
I think I'll switch to cms on the problem node and compare the memory
usage and compare the nodetool info output.



 Michael

Am 11.08.2017 um 17:55 schrieb kurt greaves:
> How much memory do these machines have?  Typically we've found that G1
> isn't worth it until you get to around 24G heaps, and even at that it's
> not really better than CMS. You could try CMS with an 8G heap and 2G new
> size. 
> 
> However as the oom is only happening on one node have you ensured there
> are no extra processes running on that node that could be consuming
> extra memory? Note that the oom killer will kill the process with the
> highest oom score, which generally corresponds to the process using the
> most memory, but not necessarily the problem.
> 
> Also could you run nodetool info on the problem node and 1 other and
> dump the output in a gist? It would be interesting to see if there is a
> significant difference in off-heap.
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: rebuild constantly fails, 3.11

[kurt greaves]

cc'ing user back in...

On 12 Aug. 2017 01:55, "kurt greaves"  wrote:

> How much memory do these machines have?  Typically we've found that G1
> isn't worth it until you get to around 24G heaps, and even at that it's not
> really better than CMS. You could try CMS with an 8G heap and 2G new size.
>
> However as the oom is only happening on one node have you ensured there
> are no extra processes running on that node that could be consuming extra
> memory? Note that the oom killer will kill the process with the highest oom
> score, which generally corresponds to the process using the most memory,
> but not necessarily the problem.
>
> Also could you run nodetool info on the problem node and 1 other and dump
> the output in a gist? It would be interesting to see if there is a
> significant difference in off-heap.
>
> On 11 Aug. 2017 17:30, "Micha"  wrote:
>
>> It's an oom issue, the kernel kills the cassandra job.
>> The config was to use offheap buffers and 20G java heap, I changed this
>> to use heap buffers and 16G java heap. I added a  new node yesterday
>> which got streams from 4 other nodes. They all succeeded except on the
>> one node which failed before. This time again the db was killed by the
>> kernel. At the moment I don't know what is the reason here, since the
>> nodes are equal.
>>
>> For me it seems the g1gc is not able to free the memory fast enough.
>> The settings were for  MaxGCPauseMillis=600 and ParallelGCThreads=10
>> ConcGCThreads=10 which maybe are too high since the node has only 8
>> cores..
>> I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned
>> in the comments of jvm.options
>>
>> Since the bootstrap of the fifth node did not complete I will start it
>> again and check if the memory is still decreasing over time.
>>
>>
>>
>>  Michael
>>
>>
>>
>> On 11.08.2017 01:25, Jeff Jirsa wrote:
>> >
>> >
>> > On 2017-08-08 01:00 (-0700), Micha  wrote:
>> >> Hi,
>> >>
>> >> it seems I'm not able to add add 3 node dc to a 3 node dc. After
>> >> starting the rebuild on a new node, nodetool netstats show it will
>> >> receive 1200 files from node-1 and 5000 from node-2. The stream from
>> >> node-1 completes but the stream from node-2 allways fails, after
>> sending
>> >> ca 4000 files.
>> >>
>> >> After restarting the rebuild it again starts to send the 5000 files.
>> >> The whole cluster is connected via one switch only , no firewall
>> >> between, the networks shows no errors.
>> >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
>> >> the logs show no errors. The size of the data is ca 1TB.
>> >
>> > Is there anything in `dmesg` ?  System logs? Nothing? Is node2 running?
>> Is node3 running?
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: dev-h...@cassandra.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>

Re: rebuild constantly fails, 3.11

[kurt greaves]

How much memory do these machines have?  Typically we've found that G1
isn't worth it until you get to around 24G heaps, and even at that it's not
really better than CMS. You could try CMS with an 8G heap and 2G new size.

However as the oom is only happening on one node have you ensured there are
no extra processes running on that node that could be consuming extra
memory? Note that the oom killer will kill the process with the highest oom
score, which generally corresponds to the process using the most memory,
but not necessarily the problem.

Also could you run nodetool info on the problem node and 1 other and dump
the output in a gist? It would be interesting to see if there is a
significant difference in off-heap.

On 11 Aug. 2017 17:30, "Micha"  wrote:

> It's an oom issue, the kernel kills the cassandra job.
> The config was to use offheap buffers and 20G java heap, I changed this
> to use heap buffers and 16G java heap. I added a  new node yesterday
> which got streams from 4 other nodes. They all succeeded except on the
> one node which failed before. This time again the db was killed by the
> kernel. At the moment I don't know what is the reason here, since the
> nodes are equal.
>
> For me it seems the g1gc is not able to free the memory fast enough.
> The settings were for  MaxGCPauseMillis=600 and ParallelGCThreads=10
> ConcGCThreads=10 which maybe are too high since the node has only 8 cores..
> I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned
> in the comments of jvm.options
>
> Since the bootstrap of the fifth node did not complete I will start it
> again and check if the memory is still decreasing over time.
>
>
>
>  Michael
>
>
>
> On 11.08.2017 01:25, Jeff Jirsa wrote:
> >
> >
> > On 2017-08-08 01:00 (-0700), Micha  wrote:
> >> Hi,
> >>
> >> it seems I'm not able to add add 3 node dc to a 3 node dc. After
> >> starting the rebuild on a new node, nodetool netstats show it will
> >> receive 1200 files from node-1 and 5000 from node-2. The stream from
> >> node-1 completes but the stream from node-2 allways fails, after sending
> >> ca 4000 files.
> >>
> >> After restarting the rebuild it again starts to send the 5000 files.
> >> The whole cluster is connected via one switch only , no firewall
> >> between, the networks shows no errors.
> >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
> >> the logs show no errors. The size of the data is ca 1TB.
> >
> > Is there anything in `dmesg` ?  System logs? Nothing? Is node2 running?
> Is node3 running?
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: rebuild constantly fails, 3.11

[Micha]

It's an oom issue, the kernel kills the cassandra job.
The config was to use offheap buffers and 20G java heap, I changed this
to use heap buffers and 16G java heap. I added a  new node yesterday
which got streams from 4 other nodes. They all succeeded except on the
one node which failed before. This time again the db was killed by the
kernel. At the moment I don't know what is the reason here, since the
nodes are equal.

For me it seems the g1gc is not able to free the memory fast enough.
The settings were for  MaxGCPauseMillis=600 and ParallelGCThreads=10
ConcGCThreads=10 which maybe are too high since the node has only 8 cores..
I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned
in the comments of jvm.options

Since the bootstrap of the fifth node did not complete I will start it
again and check if the memory is still decreasing over time.



 Michael



On 11.08.2017 01:25, Jeff Jirsa wrote:
> 
> 
> On 2017-08-08 01:00 (-0700), Micha  wrote: 
>> Hi,
>>
>> it seems I'm not able to add add 3 node dc to a 3 node dc. After
>> starting the rebuild on a new node, nodetool netstats show it will
>> receive 1200 files from node-1 and 5000 from node-2. The stream from
>> node-1 completes but the stream from node-2 allways fails, after sending
>> ca 4000 files.
>>
>> After restarting the rebuild it again starts to send the 5000 files.
>> The whole cluster is connected via one switch only , no firewall
>> between, the networks shows no errors.
>> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
>> the logs show no errors. The size of the data is ca 1TB.
> 
> Is there anything in `dmesg` ?  System logs? Nothing? Is node2 running? Is 
> node3 running? 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

RE: rebuild constantly fails, 3.11

[Bob Dourandish]

I don't know if this is going to help or not but we had a problem disk (was
going bad) and would develop bad blocks, requiring raid repair. As nearly as
we could guess, that would cause "some sort of timing issue" for nodetool -
we were never able to duplicate the exact occurrence on demand. The clue
came when the same exact process worked on a different setup that did not
use the same disk subsystem. Once we isolated and replaced the degrading
disk, the problem went away. 

Beyond what Jeff suggested, you may want to look at any logs your raid
produces for potential clues. Also, if it is always failing when sending the
same file, you might want to review what is going on there.

I know this is fairly "general" response but I hope it gives you some ideas.

Good luck!

Bob

-Original Message-
From: Jeff Jirsa [mailto:jji...@apache.org] 
Sent: Thursday, August 10, 2017 7:26 PM
To: dev@cassandra.apache.org
Subject: Re: rebuild constantly fails, 3.11



On 2017-08-08 01:00 (-0700), Micha <mich...@fantasymail.de> wrote: 
> Hi,
> 
> it seems I'm not able to add add 3 node dc to a 3 node dc. After 
> starting the rebuild on a new node, nodetool netstats show it will 
> receive 1200 files from node-1 and 5000 from node-2. The stream from
> node-1 completes but the stream from node-2 allways fails, after 
> sending ca 4000 files.
> 
> After restarting the rebuild it again starts to send the 5000 files.
> The whole cluster is connected via one switch only , no firewall 
> between, the networks shows no errors.
> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
> the logs show no errors. The size of the data is ca 1TB.

Is there anything in `dmesg` ?  System logs? Nothing? Is node2 running? Is
node3 running? 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: rebuild constantly fails, 3.11

[Jeff Jirsa]

On 2017-08-08 01:00 (-0700), Micha  wrote: 
> Hi,
> 
> it seems I'm not able to add add 3 node dc to a 3 node dc. After
> starting the rebuild on a new node, nodetool netstats show it will
> receive 1200 files from node-1 and 5000 from node-2. The stream from
> node-1 completes but the stream from node-2 allways fails, after sending
> ca 4000 files.
> 
> After restarting the rebuild it again starts to send the 5000 files.
> The whole cluster is connected via one switch only , no firewall
> between, the networks shows no errors.
> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0.
> the logs show no errors. The size of the data is ca 1TB.

Is there anything in `dmesg` ?  System logs? Nothing? Is node2 running? Is 
node3 running? 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org