We have a 2.4 ignite peristence cluster on AWS with 8 i3.xlarge nodes, where the default memory region size is 160GB. When loading a large amount of data through a custom StreamReceiver, over time we see the throughput decrease, as well as network and SSD I/O, and one node with higher CPU usage than the others, but still only 33%. Looking at that node, I see (kill -3) that out of 27 sys-stripe* threads, 21 are blocked on locks, 17 of which are memory segment locks. The running threads have the following call stacks: acquirePage -> get/put ->getKey/Putkey . The 2.4 documentation says I should have 32 (one per CPU) segments. I wouldn't expect such high contention if there was random distribution of load across the segments. I'm only using the default region. Is there a explaination for such a high contention level.
It looks as if I can set the number of segments higher by configuring the data region setConcurrencyLevel(). The 2.0 documentation for memory configuration said the default was 4 * # CPUS, but the 2.4 Data Region code seems to default to # CPUs, and we are only using default region. What would be a reasonable value? It looks like the current data region settings get communicated to visor, but I do not see a way to actually display them. I'm I missing something? If I change the xml configuration, it is unclear to me whether this will be applied when I restart the node or is that setting only used when the data region is originally created. (As a general documentation thing, it would be helpful to identify when settings will actually be applied. ) Thanks, -DH PS. Reading the code, my first thought was that locks were configured as fair, which ensures order but increases lock held time, but the default seems to be unfair, which is good. Fair locks have the characteristic of allowing high throughput until contended, then locking in low throughput until the load completely stops. Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
