It's an oom issue, the kernel kills the cassandra job. The config was to use offheap buffers and 20G java heap, I changed this to use heap buffers and 16G java heap. I added a new node yesterday which got streams from 4 other nodes. They all succeeded except on the one node which failed before. This time again the db was killed by the kernel. At the moment I don't know what is the reason here, since the nodes are equal.
For me it seems the g1gc is not able to free the memory fast enough. The settings were for MaxGCPauseMillis=600 and ParallelGCThreads=10 ConcGCThreads=10 which maybe are too high since the node has only 8 cores.. I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned in the comments of jvm.options Since the bootstrap of the fifth node did not complete I will start it again and check if the memory is still decreasing over time. Michael On 11.08.2017 01:25, Jeff Jirsa wrote: > > > On 2017-08-08 01:00 (-0700), Micha <mich...@fantasymail.de> wrote: >> Hi, >> >> it seems I'm not able to add add 3 node dc to a 3 node dc. After >> starting the rebuild on a new node, nodetool netstats show it will >> receive 1200 files from node-1 and 5000 from node-2. The stream from >> node-1 completes but the stream from node-2 allways fails, after sending >> ca 4000 files. >> >> After restarting the rebuild it again starts to send the 5000 files. >> The whole cluster is connected via one switch only , no firewall >> between, the networks shows no errors. >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. >> the logs show no errors. The size of the data is ca 1TB. > > Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? Is > node3 running? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
