Hello! Since we don't seem to have much experience in running YARN here, can you please elaborate on this problem? Is the number of consumed resources wrong or are resources actually get consumed by Ignite?
BTW, if you plan to store data inside Ignite, you should also set IGNITE_MEMORY_OVERHEAD_PER_NODE, see discussion at http://apache-ignite-developers.2346864.n4.nabble.com/YARN-parameters-and-default-values-td29899.html Regards, -- Ilya Kasnacheev 2018-05-17 14:31 GMT+03:00 Murugar <[email protected]>: > Hi, > > Ignite-yarn[Ignition] uses almost all memory and vcores available in YARN > cluster. The allocated memory to the Ignition server was 40GB and 16 > VCores. > > *When we checked the actual usage with yarn top command the Ignition server > is using 380 GB and 140 VCores!!! * When we checked the cache, it is having > only less number of records - 3000 records. > > *Information about the application:* > We load data into Ignite Cache, process them periodically and delete > processed records periodically. > > We are running Ignite-yarn in Cloudera cluster with 5 data nodes with below > cluster.properties > > IGNITE_NODE_COUNT=4 > IGNITE_RUN_CPU_PER_NODE=4 > IGNITE_MEMORY_PER_NODE=10096 > IGNITE_VERSION=2.3.0 > IGNITE_PATH=/apache-ignite-fabric-2.3.0-bin.zip > > IGNITE_RELEASES_DIR=/releases > IGNITE_WORKING_DIR=/work > IGNITE_XML_CONFIG=/ignite-config.xml > IGNITE_USERS_LIBS=/libs > IGNITE_LOCAL_WORK_DIR=/home/user > IGNITE_JVM_OPTS="-server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC" > > > Please find our Ignite-config.xml below > > <?xml version="1.0" encoding="UTF-8"?> > <beans xmlns="http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xmlns:util="http://www.springframework.org/schema/util"; > xsi:schemaLocation=" > http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.5.xsd > http://www.springframework.org/schema/util > http://www.springframework.org/schema/util/spring-util-2.0.xsd";> > <bean class="org.apache.ignite.configuration.IgniteConfiguration"> > <property name="clientMode" value="false"/> > <property name="peerClassLoadingEnabled" value="true"/> > <property name="networkTimeout" value="100000"/> > <property name="networkSendRetryCount" value="5"/> > <property name="discoverySpi"> > <bean > class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> > <property name="ipFinder"> > <bean > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm. > TcpDiscoveryVmIpFinder"> > <property name="addresses"> > <list> > > <value>ipaddress:47500</value> > </list> > </property> > </bean> > </property> > <property name="networkTimeout" > value="10000000"/> > <property name="joinTimeout" > value="10000000"/> > <property name="maxAckTimeout" > value="10000000"/> > <property name="reconnectCount" > value="50"/> > <property name="socketTimeout" > value="10000000"/> > </bean> > </property> > </bean> > </beans> > > Cache Configuration: > > var userGridConf: CacheConfiguration[Long, User] = new > CacheConfiguration[Long, User](userCacheName) > userGridConf.setCacheMode(CacheMode.REPLICATED) > userGridConf.setEagerTtl(false) > userGridConf.setOnheapCacheEnabled(false) > userGridConf.setBackups(0) > userGridConf.setIndexedTypes(classOf[Long], classOf[User]) > val cache = ignite.getOrCreateCache(userGridConf) > cache.size(CachePeekMode.OFFHEAP) > > Could you please let me know what is wrong with the above configurations? > > Thanks in Advance!!! > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
