Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Mahesh Renduchintala]

Hi,

We have been searching for a cause of memory leak in ignite server nodes for 
many weeks now.
The memory leak exists and below is the scenario.

Scenario

  *   Scenario
     *   Our ignite servers have about 50GB of data. Two servers were baselined
     *   There are about 10 client nodes connected
     *   OffHeap memory - 64GB per node, 48GB heap (XMS, XMS parameters)
  *   Test
     *   start all 10 nodes
     *   wait for 10 nodes to be connected
     *   sleep for 30 mins
     *   Stop all 10 nodes (docker rm -f X)
     *   Loop 1 to 4 steps infinitely

    Observations

     *   With every iteration, about 1GB of memory disappears from the heap f 
servers
     *   After about 100 iterations, server nodes crash reporting OOM

Workaround

     *   The memory leak does not occur when directBuffer is made false as 
below. Meaning TCP buffers are only onHeap
        *

<property name="communicationSpi">
    <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
        <property name="connectTimeout" value="90000"/>
        <property name="directBuffer" value="false"/>
        <property name="directSendBuffer" value="false"/>
        <property name="messageQueueLimit" value="10000"/>
        <property name="idleConnectionTimeout" value="#{315360000000L}"/>
        <property name="selectorSpins" value="2"/>
        <property name="usePairedConnections" value="true"/>
        <property name="filterReachableAddresses" value="true"/>
        <property name="socketWriteTimeout" value="20000"/>
    </bean>
</property>

Hope you can reproduce this problem at your end.
This occurs with any default configuration of ignite servers. This is why, I am 
not sending mine.

regards
Mahesh



                             *   The only work around we found is to use on