Hi, everyone
I meet a problem: Ignite used 220G+ memory, but the imported data is only
31G. Why?
Basic info: I used Ignite 1.6, and deployed ignite cluster using 2 machine
whose memory is 128G.
I run 3 ignite instance in every machine. I am using JDK8.
I has 31G data in csv file. First, I import the data into oracle's one
table, total records: 47,535,542
Then I using ignite-schema-import.sh tool to produce POJO files, and put
them info my java project.
Then I used IgniteCache.loadCache to load data from oracle to ignite. It
used time: 01:43:17
After importing data, I used linux command free and ps aux, get the
following info:
free -g:
machine1:
total used free shared buffers
cached
Mem: 125 114 11 0 0 0
-/+ buffers/cache: 114 11
Swap: 3 0 3
machine2:
total used free shared buffers
cached
Mem: 125 122 3 0 0 0
-/+ buffers/cache: 121 4
Swap: 3 0 3
ps aux:
machine1:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 23234 48.2 28.8 56278844 38088436 pts/1 Sl 14:29 70:08
/usr/java/jdk1.8.0_91/bin/java -server -Xms1g -Xmx1g -XX:NewSize=512m
-XX:SurvivorRatio=6 -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:MaxGCPauseMillis=2000
-XX:GCTimeRatio=4 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=16 -XX:G1HeapWastePercent=10 -XX:+UseTLAB
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -DIGNITE_QUIET=true
-DIGNITE_SUCCESS_FILE=/root/apache-ignite-fabric-1.6.0-bin/work/ignite_success_6f17738f-6b73-4681-a06e-a9d270c0fe9c
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49164
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_HOME=/root/apache-ignite-fabric-1.6.0-bin
-DIGNITE_PROG_NAME=bin/ignite.sh -cp
/root/apache-ignite-fabric-1.6.0-bin/libs/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-indexing/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-rest-http/*:/root/apache-ignite-fabric-1.6.0-bin/libs/ignite-spring/*:/root/apache-ignite-fabric-1.6.0-bin/libs/licenses/*
org.apache.ignite.startup.cmdline.CommandLineStartup config_poc.xml
root 22259 46.5 32.7 61434264 43266272 pts/1 Sl 14:28 68:14
/usr/java/jdk1.8.0_91/bin/java ...
root 22729 44.7 28.3 55533284 37374932 pts/1 Sl 14:28 65:29
/usr/java/jdk1.8.0_91/bin/java ...
machine2:
root 182359 76.1 32.0 60570656 42322148 pts/2 Sl 14:34 87:02
/usr/java/jdk1.8.0_91/bin/java ...
root 181882 75.8 31.7 60219724 41962436 pts/2 Sl 14:34 87:04
/usr/java/jdk1.8.0_91/bin/java ...
root 182867 56.8 31.8 60243104 42005888 pts/2 Sl 14:34 64:54
/usr/java/jdk1.8.0_91/bin/java ...
The ignite config:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
Ignite Spring configuration file to startup Ignite cache.
This file demonstrates how to configure cache using Spring. Provided cache
will be created on node startup.
Use this configuration file when running HTTP REST examples (see
'examples/rest' folder).
When starting a standalone node, you need to execute the following command:
{IGNITE_HOME}/bin/ignite.{bat|sh} examples/config/example-cache.xml
When starting Ignite from Java IDE, pass path to this file to Ignition:
Ignition.start("examples/config/example-cache.xml");
-->
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd";>
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<!-- Set cache mode. -->
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="1"/>
<property name="memoryMode" value="OFFHEAP_TIERED" />
<!-- Enable off-heap storage with unlimited size. -->
<!--property name="offHeapMaxMemory" value="0"/-->
<!-- Set initial cache capacity to ~ 100M. -->
<!--property name="startSize" value="#{500 * 1024 * 1024}"/-->
<!--property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="100000"/>
</bean>
</property-->
</bean>
</list>
</property>
<!-- Enable only some events and leave other ones disabled. -->
<!--property name="includeEventTypes">
<list>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
</list>
</property-->
<!-- Explicitly configure TCP discovery SPI to provide list of initial
nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery
that can be used
instead os static IP based discovery. For information
on all options refer
to our documentation:
http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based
discovery of initial nodes. -->
<!--<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with
actual host IP address. -->
<value>10.9.22.15:47500..47509</value>
<value>10.9.22.16:47500..47509</value>
<value>192.168.175.39:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
I don't know how analye this problem, please help me . Thanks very much.
Bob
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of
this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing,
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this
communication in error,please
immediately notify the sender by return e-mail, and delete the original message
and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
