[
https://issues.apache.org/jira/browse/CASSANDRA-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15072053#comment-15072053
]
Peter Kovgan edited comment on CASSANDRA-10937 at 12/27/15 6:10 AM:
--------------------------------------------------------------------
Unfortunatelly retest with lesser load did not help.
Now average CPU usage on all nodes was 60-70%.(when test started, not sure
about entire test)
There are still some nodes down.
GC is very long (30-40 sec) before OOM occurs.
I send logs and heap shapshot attachment (test3.rar)
First (2015-12-24 20:51:58) OOM occured on node 69 (logs attached in test3.rar)
Then (2015-12-25 03:20:28) hints ate all the disk space on node 65 and node
shutdown
Then (2015-12-25 13:10:20) the node 67 gone with OOM
-- Address Load Tokens Owns Host ID
Rack
UN 10.20.42.68 285.06 GB 256 ?
6ce9d0bf-9569-4f4a-9769-c96f76a1c87f rack1
DN 10.20.42.69 119.4 GB 256 ?
8452fbfb-c692-403e-9c43-83493ab39201 rack1
UN 10.20.42.64 317.97 GB 256 ?
b09eb63c-0d24-4373-8379-85d0616f488e rack1
DN 10.20.42.65 197.49 GB 256 ?
61a1561a-15a4-412b-b831-b47a6e5902a9 rack1
UN 10.20.42.66 276.96 GB 256 ?
22a3f2d8-39ea-4aa9-9b38-be8e850823f4 rack1
DN 10.20.42.67 243.97 GB 256 ?
05672e1a-581a-45b1-92fe-9c3ce9409752 rack1
UN 10.20.42.62 332.89 GB 256 ?
e3e74ff4-c8f9-4186-b4ec-b66a6e05d9b4 rack1
UN 10.20.42.63 330.99 GB 256 ?
e51a871f-6aac-415e-9928-2ead4d3cbf6b rack1
was (Author: tierhetze):
Unfortunatelly retest with lesser load did not help.
Now average CPU usage on all nodes was 60-70%.(when test started, not sure
about entire test)
There are still some nodes down.
GC is very long (30-40 sec) before OOM occurs.
I send logs and heap shapshot attachment (test3.rar)
-- Address Load Tokens Owns Host ID
Rack
UN 10.20.42.68 285.06 GB 256 ?
6ce9d0bf-9569-4f4a-9769-c96f76a1c87f rack1
DN 10.20.42.69 119.4 GB 256 ?
8452fbfb-c692-403e-9c43-83493ab39201 rack1
UN 10.20.42.64 317.97 GB 256 ?
b09eb63c-0d24-4373-8379-85d0616f488e rack1
DN 10.20.42.65 197.49 GB 256 ?
61a1561a-15a4-412b-b831-b47a6e5902a9 rack1
UN 10.20.42.66 276.96 GB 256 ?
22a3f2d8-39ea-4aa9-9b38-be8e850823f4 rack1
DN 10.20.42.67 243.97 GB 256 ?
05672e1a-581a-45b1-92fe-9c3ce9409752 rack1
UN 10.20.42.62 332.89 GB 256 ?
e3e74ff4-c8f9-4186-b4ec-b66a6e05d9b4 rack1
UN 10.20.42.63 330.99 GB 256 ?
e51a871f-6aac-415e-9928-2ead4d3cbf6b rack1
> OOM on multiple nodes during test of insert load
> ------------------------------------------------
>
> Key: CASSANDRA-10937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10937
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra : 3.0.0
> Installed as open archive, no connection to any OS specific installer.
> Java:
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> OS :
> Linux version 2.6.32-431.el6.x86_64
> ([email protected]) (gcc version 4.4.7 20120313 (Red
> Hat 4.4.7-4) (GCC) ) #1 SMP Sun Nov 10 22:19:54 EST 2013
> We have:
> 8 guests ( Linux OS as above) on 2 (VMWare managed) physical hosts. Each
> physical host keeps 4 guests.
> Physical host parameters(shared by all 4 guests):
> Model: HP ProLiant DL380 Gen9
> Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
> 46 logical processors.
> Hyperthreading - enabled
> Each guest assigned to have:
> 1 disk 300 Gb for seq. log (NOT SSD)
> 1 disk 4T for data (NOT SSD)
> 11 CPU cores
> Disks are local, not shared.
> Memory on each host - 24 Gb total.
> 8 Gb - cassandra heap
> (lshw and cpuinfo attached in file)
> Reporter: Peter Kovgan
> Attachments: more-logs.rar, some-heap-stats.rar, test2.rar, test3.rar
>
>
> 8 cassandra nodes.
> Load test started with 4 clients(different and not equal machines), each
> running 1000 threads.
> Each thread assigned in round-robin way to run one of 4 different inserts.
> Consistency->ONE.
> I attach the full CQL schema of tables and the query of insert.
> Replication factor - 2:
> create keyspace OBLREPOSITORY_NY with replication =
> {'class':'NetworkTopologyStrategy','NY':2};
> Initiall throughput is:
> 215.000 inserts /sec
> or
> 54Mb/sec, considering single insert size a bit larger than 256byte.
> Data:
> all fields(5-6) are short strings, except one is BLOB of 256 bytes.
> After about a 2-3 hours of work, I was forced to increase timeout from 2000
> to 5000ms, for some requests failed for short timeout.
> Later on(after aprox. 12 hous of work) OOM happens on multiple nodes.
> (all failed nodes logs attached)
> I attach also java load client and instructions how set-up and use it.
> The test is important for our strategic project and we hope it is curable.
> Attachments:
> test2.rar -contains most of material
> more-logs.rar - contains additional nodes logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
