Hey,
Which thread pools are responsible for the compute jobs and cache operations ? Thanks, Nadav From: Ilya Kasnacheev <ilya.kasnach...@gmail.com> Sent: Wednesday, February 20, 2019 12:26 PM To: user@ignite.apache.org Subject: Re: Apache Ignite starts fast and then become really slow with out of memory Hello! Maybe you could do that by decreasing thread pool sizes as well as limiting message queue lengths. However, it might not be worth the hassle. Regards, -- Ilya Kasnacheev вт, 19 февр. 2019 г. в 21:26, <nadav.h...@docauthority.com <mailto:nadav.h...@docauthority.com> >: Hey Ilya, I managed to stable the memory and avoid extreme slowness by send compute jobs one by one and not in bulk. The compute jobs takes objects from cache by bulks, so I guess that firing 50 compute jobs on a node where each job takes bulk of 100 is too heavy for it to handle… My node has 32GB (Can go even higher) and 16 Cores. The one by one bits the purpose of scaling (i.e. multiple nodes serving the same request), Is there a way to send a bulk of compute jobs and manage it won't "explode" Thank you very much!! Nadav From: Ilya Kasnacheev <ilya.kasnach...@gmail.com <mailto:ilya.kasnach...@gmail.com> > Sent: Monday, February 18, 2019 11:39 AM To: user@ignite.apache.org <mailto:user@ignite.apache.org> Subject: Re: Apache Ignite starts fast and then become really slow with out of memory Hello! I recommend starting from simplest configuration to more complex one. Remove onheap cache, see if problem goes away. Regards, -- Ilya Kasnacheev вс, 17 февр. 2019 г. в 12:09, <nadav.h...@docauthority.com <mailto:nadav.h...@docauthority.com> >: Hey Ilya, Yes you are right, the LinkedHashMaps entries are the dominate part. I was concentrating on the reader because I thought there may be an implicit connection.. Those LinkedHashMaps entries are part of the object structure I'm loading to cache, if needed I can send the model code (Java). As you can see from my config, I use onheapCacheEnabled=true, my objects can be pretty heavy (2 MG each), but I've place a rigid eviction policy of only 1000 entries. <property name="evictionPolicy"> <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy"> <property name="maxSize" value="1000"/> </bean> </property> My heap is 16GB (can be more..) if I set onheapCacheEnabled=false does it mean that the object loaded will not reside in the heap..? My general assumption is that I have problem with my GC tuning and it does not handle the cache load well and struggles to clear/allocate memory. What will help you to identify my problem more precisely ? (GC Logs, behavioral code, model code) ? Thank you very much. Nadav From: Ilya Kasnacheev <ilya.kasnach...@gmail.com <mailto:ilya.kasnach...@gmail.com> > Sent: Friday, February 15, 2019 4:33 PM To: user@ignite.apache.org <mailto:user@ignite.apache.org> Subject: Re: Apache Ignite starts fast and then become really slow with out of memory Hello! As far as I can see, the highlighted class does not dominate heap in any meaningful way. You seem to have huge LinkedHashMaps, any idea where they are used/held? Regards, -- Ilya Kasnacheev чт, 14 февр. 2019 г. в 22:11, <nadav.h...@docauthority.com <mailto:nadav.h...@docauthority.com> >: Hey, Guys is this normal…? So many binary readers are opened without reuse? I saw an improvement issue on this https://jira.apache.org/jira/browse/IGNITE-5721 I'm running a simple compute tasks that split around 4K ids to compute job that each one takes from cache bulk of 500 ids. Each Object size return is ~1.8MB. When I run 3-4 iterations it is super fasts completing the tasks under 100 ms. But then memory rises and it becomes really slow, fully occupied by GC and after a while it caves and get OOME. I'm evaluating ignite for a POC and this is a real showstopper. I'm sure that I'm missing something because I do nothing out of the ordinary. I tried different GC tuning G1 and CMS, different evection policies, different region sizes. Nothing helped. I'm using the latest ignite 2.7, my machine is windows server 12 R2 with 8 cores and 32GB RAM, this is my node config: <beans xmlns="http://www.springframework.org/schema/beans"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns:context="http://www.springframework.org/schema/context"; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd "> <context:annotation-config/> <context:property-placeholder location="classpath:application.properties"/> <bean id="similarityConfiguration" class="com.cla.filecluster.service.files.pv.SimilarityConfiguration"/> <bean id="similarityCalculator" class="com.cla.filecluster.service.files.pv.SimilarityCalculator" factory-method="create"> <constructor-arg ref="similarityConfiguration" /> </bean> <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration"> <property name="userAttributes"> <map key-type="java.lang.String" value-type="java.lang.Boolean"> <entry key="data.node" value="true"/> </map> </property> <property name="binaryConfiguration"> <bean class="org.apache.ignite.configuration.BinaryConfiguration"> <property name="compactFooter" value="false"/> </bean> </property> <!-- property name="asyncCallbackPoolSize" value="2"/ --> <property name="dataStorageConfiguration"> <bean class="org.apache.ignite.configuration.DataStorageConfiguration"> <property name="pageSize" value="#{4 * 1024}"/> <property name="defaultDataRegionConfiguration"> <bean class="org.apache.ignite.configuration.DataRegionConfiguration"> <property name="name" value="analysis_region"/> <property name="persistenceEnabled" value="true"/> <property name="initialSize" value="#{100 * 1024 * 1024}"/> <property name="maxSize" value="8589934592"/> </bean> </property> <property name="storagePath" value="E:\asm-repo\db"/> <property name="walPath" value="E:\asm-repo\wal"/> <property name="walArchivePath" value="E:\asm-repo\wal\archive"/> </bean> </property> <property name="cacheConfiguration"> <list> <!-- Partitioned cache example configuration (Atomic mode). --> <bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="storeKeepBinary" value="true"/> <property name="eagerTtl" value="true"/> <property name="name" value="PvAnalysisData"/> <property name="dataRegionName" value="analysis_region"/> <property name="atomicityMode" value="ATOMIC"/> <property name="backups" value="0"/> <property name="onheapCacheEnabled" value="true"/> <property name="expiryPolicyFactory"> <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf"> <constructor-arg> <bean class="javax.cache.expiry.Duration"> <constructor-arg value="SECONDS"/> <constructor-arg value="5"/> </bean> </constructor-arg> </bean> </property> <property name="evictionPolicy"> <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy"> <!-- property name="maxMemorySize" value="104857600"/ --> <property name="maxSize" value="1000"/> </bean> </property> <property name="nodeFilter"> <bean class="com.cla.filecluster.common.DataNodeFilter"/> </property> </bean> </list> </property> <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. --> <property name="discoverySpi"> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> <property name="ipFinder"> <!-- Ignite provides several options for automatic discovery that can be used instead os static IP based discovery. For information on all options refer to our documentation: http://apacheignite.readme.io/docs/cluster-config --> <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. --> <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">--> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"> <property name="addresses"> <list> <!-- In distributed environment, replace with actual host IP address. --> <value>127.0.0.1:47500..47509</value> </list> </property> </bean> </property> </bean> </property> </bean> </beans> Thanks, Nadav System Architect +972-544821606 <http://www.docauthority.com/> DocAuthority.com
