Hey Ilya,
Yes you are right, the LinkedHashMaps entries are the dominate part.
I was concentrating on the reader because I thought there may be an implicit
connection..
Those LinkedHashMaps entries are part of the object structure I'm loading to
cache, if needed I can send the model code (Java).
As you can see from my config, I use onheapCacheEnabled=true, my objects can be
pretty heavy (2 MG each), but I've place a rigid eviction policy of only 1000
entries.
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="1000"/>
</bean>
</property>
My heap is 16GB (can be more..)
if I set onheapCacheEnabled=false does it mean that the object loaded will not
reside in the heap..?
My general assumption is that I have problem with my GC tuning and it does not
handle the cache load well and struggles to clear/allocate memory.
What will help you to identify my problem more precisely ? (GC Logs, behavioral
code, model code) ?
Thank you very much.
Nadav
From: Ilya Kasnacheev <[email protected]>
Sent: Friday, February 15, 2019 4:33 PM
To: [email protected]
Subject: Re: Apache Ignite starts fast and then become really slow with out of
memory
Hello!
As far as I can see, the highlighted class does not dominate heap in any
meaningful way.
You seem to have huge LinkedHashMaps, any idea where they are used/held?
Regards,
--
Ilya Kasnacheev
чт, 14 февр. 2019 г. в 22:11, <[email protected]
<mailto:[email protected]> >:
Hey,
Guys is this normal…?
So many binary readers are opened without reuse? I saw an improvement issue on
this https://jira.apache.org/jira/browse/IGNITE-5721
I'm running a simple compute tasks that split around 4K ids to compute job that
each one takes from cache bulk of 500 ids.
Each Object size return is ~1.8MB.
When I run 3-4 iterations it is super fasts completing the tasks under 100 ms.
But then memory rises and it becomes really slow, fully occupied by GC and
after a while it caves and get OOME.
I'm evaluating ignite for a POC and this is a real showstopper.
I'm sure that I'm missing something because I do nothing out of the ordinary.
I tried different GC tuning G1 and CMS, different evection policies, different
region sizes. Nothing helped.
I'm using the latest ignite 2.7, my machine is windows server 12 R2 with 8
cores and 32GB RAM, this is my node config:
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:context="http://www.springframework.org/schema/context";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd
">
<context:annotation-config/>
<context:property-placeholder location="classpath:application.properties"/>
<bean id="similarityConfiguration"
class="com.cla.filecluster.service.files.pv.SimilarityConfiguration"/>
<bean id="similarityCalculator"
class="com.cla.filecluster.service.files.pv.SimilarityCalculator"
factory-method="create">
<constructor-arg ref="similarityConfiguration" />
</bean>
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="userAttributes">
<map key-type="java.lang.String" value-type="java.lang.Boolean">
<entry key="data.node" value="true"/>
</map>
</property>
<property name="binaryConfiguration">
<bean
class="org.apache.ignite.configuration.BinaryConfiguration">
<property
name="compactFooter" value="false"/>
</bean>
</property>
<!-- property name="asyncCallbackPoolSize"
value="2"/ -->
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="pageSize"
value="#{4 * 1024}"/>
<property
name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="analysis_region"/>
<property name="persistenceEnabled" value="true"/>
<property name="initialSize" value="#{100 * 1024 * 1024}"/>
<property name="maxSize" value="8589934592"/>
</bean>
</property>
<property name="storagePath" value="E:\asm-repo\db"/>
<property name="walPath" value="E:\asm-repo\wal"/>
<property
name="walArchivePath" value="E:\asm-repo\wal\archive"/>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="storeKeepBinary" value="true"/>
<property name="eagerTtl" value="true"/>
<property name="name" value="PvAnalysisData"/>
<property name="dataRegionName" value="analysis_region"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
<property name="onheapCacheEnabled" value="true"/>
<property
name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy"
factory-method="factoryOf">
<constructor-arg>
<bean
class="javax.cache.expiry.Duration">
<constructor-arg value="SECONDS"/>
<constructor-arg value="5"/>
</bean>
</constructor-arg>
</bean>
</property>
<property
name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<!-- property name="maxMemorySize" value="104857600"/ -->
<property name="maxSize" value="1000"/>
</bean>
</property>
<property name="nodeFilter">
<bean
class="com.cla.filecluster.common.DataNodeFilter"/>
</property>
</bean>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial
nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery
that can be used
instead os static IP based discovery. For information
on all options refer
to our documentation:
http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based
discovery of initial nodes. -->
<!--<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with
actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Thanks,
Nadav
System Architect
+972-544821606
<http://www.docauthority.com/> DocAuthority.com
