Now， because I created my cache in java code, so changing the config.xml
doesn't work.
I already change the java code of creating my cache to change the backups to
zero from one:
CacheConfiguration<K, V> cfg =
CacheConfig.cache(cacheName, pojoStoreFactory);
// cfg.setBackups(1);
cfg.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cfg.setCacheMode(CacheMode.PARTITIONED);
After that, the same data consumed memory: 130G, the detail is the following:
[root@ignite15 apache-ignite-fabric-1.6.0-bin]# free -g
total used free shared buffers cached
Mem: 125 66 59 0 0 0
-/+ buffers/cache: 65 60
Swap: 3 0 3
[root@localhost apache-ignite-fabric-1.6.0-bin]# free -g
total used free shared buffers cached
Mem: 125 65 60 0 0 0
-/+ buffers/cache: 64 61
Swap: 3 0 3
Can anyone sugguest me how to reduce the number of used memory?
Ignite consume too much memory!!!
Thanks.
Bob
发件人： 胡永亮/Bob
发送时间： 2016-07-12 17:22
收件人： [email protected]
主题： Ignite used memory 7 times greater than imported data
Hi, everyone
I meet a problem: Ignite used 220G+ memory, but the imported data is only
31G. Why?
Basic info: I used Ignite 1.6, and deployed ignite cluster using 2 machine
whose memory is 128G.
I run 3 ignite instance in every machine. I am using JDK8.
I has 31G data in csv file. First, I import the data into oracle's one
table, total records: 47,535,542
Then I using ignite-schema-import.sh tool to produce POJO files, and put
them info my java project.
Then I used IgniteCache.loadCache to load data from oracle to ignite. It
used time: 01:43:17
After importing data, I used linux command free and ps aux, get the
following info:
free -g:
machine1:
total used free shared buffers
cached
Mem: 125 114 11 0 0 0
-/+ buffers/cache: 114 11
Swap: 3 0 3
machine2:
total used free shared buffers
cached
Mem: 125 122 3 0 0 0
-/+ buffers/cache: 121 4
Swap: 3 0 3
ps aux:
machine1:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 23234 48.2 28.8 56278844 38088436 pts/1 Sl 14:29 70:08
/usr/java/jdk1.8.0_91/bin/java -server -Xms1g -Xmx1g -XX:NewSize=512m
-XX:SurvivorRatio=6 -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:MaxGCPauseMillis=2000
-XX:GCTimeRatio=4 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=16 -XX:G1HeapWastePercent=10 -XX:+UseTLAB
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -DIGNITE_QUIET=true
-DIGNITE_SUCCESS_FILE=/root/apache-ignite-fabric-1.6.0-bin/work/ignite_success_6f17738f-6b73-4681-a06e-a9d270c0fe9c
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49164
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_HOME=/root/apache-ignite-fabric-1.6.0-bin
-DIGNITE_PROG_NAME=bin/ignite.sh -cp
/root/apache-ignite-fabric-1.6.0-bin/libs/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-indexing/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-rest-http/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-spring/*:/root/apache-ignite-fabric-1.6.0-bin/libs/licenses/*
org.apache.ignite.startup.cmdline.CommandLineStartup config_poc.xml
root 22259 46.5 32.7 61434264 43266272 pts/1 Sl 14:28 68:14
/usr/java/jdk1.8.0_91/bin/java ...
root 22729 44.7 28.3 55533284 37374932 pts/1 Sl 14:28 65:29
/usr/java/jdk1.8.0_91/bin/java ...
machine2:
root 182359 76.1 32.0 60570656 42322148 pts/2 Sl 14:34 87:02
/usr/java/jdk1.8.0_91/bin/java ...
root 181882 75.8 31.7 60219724 41962436 pts/2 Sl 14:34 87:04
/usr/java/jdk1.8.0_91/bin/java ...
root 182867 56.8 31.8 60243104 42005888 pts/2 Sl 14:34 64:54
/usr/java/jdk1.8.0_91/bin/java ...
The ignite config:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
Ignite Spring configuration file to startup Ignite cache.
This file demonstrates how to configure cache using Spring. Provided cache
will be created on node startup.
Use this configuration file when running HTTP REST examples (see
'examples/rest' folder).
When starting a standalone node, you need to execute the following command:
{IGNITE_HOME}/bin/ignite.{bat|sh} examples/config/example-cache.xml
When starting Ignite from Java IDE, pass path to this file to Ignition:
Ignition.start("examples/config/example-cache.xml");
-->
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd";>
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<!-- Set cache mode. -->
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="1"/>
<property name="memoryMode" value="OFFHEAP_TIERED" />
<!-- Enable off-heap storage with unlimited size. -->
<!--property name="offHeapMaxMemory" value="0"/-->
<!-- Set initial cache capacity to ~ 100M. -->
<!--property name="startSize" value="#{500 * 1024 * 1024}"/-->
<!--property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="100000"/>
</bean>
</property-->
</bean>
</list>
</property>
<!-- Enable only some events and leave other ones disabled. -->
<!--property name="includeEventTypes">
<list>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
</list>
</property-->
<!-- Explicitly configure TCP discovery SPI to provide list of initial
nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery
that can be used
instead os static IP based discovery. For information
on all options refer
to our documentation:
http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based
discovery of initial nodes. -->
<!--<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with
actual host IP address. -->
<value>10.9.22.15:47500..47509</value>
<value>10.9.22.16:47500..47509</value>
<value>192.168.175.39:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
I don't know how analye this problem, please help me . Thanks very much.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of
this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing,
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this
communication in error,please
immediately notify the sender by return e-mail, and delete the original message
and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of
this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing,
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this
communication in error,please
immediately notify the sender by return e-mail, and delete the original message
and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
