Re: How IGFS keep sync with HDFS?
Hi, IGFS always propagate updates to HDFS immediately (or with slight delay in case of writes in DUAL_ASYNC mode). It doesn't remove data from memory after flushing it to HDFS. You can try configuring *org.apache.ignite.cache.eviction.igfs.IgfsPerBlockLruEvictionPolicy *to evict some data blocks from IGFS data cache. As per expiry policy, it doesn't affect flushing logic anyhow. If eviction happens after flush, then data will not be lost. Otherwise it can be lost. Though, this is possible only in DUAL_ASYNC mode with very low TTL. Vladimir. On Tue, Nov 29, 2016 at 2:17 PM, Kaiming Wan <344277...@qq.com> wrote: > If IGFS can't hold more input data, will it flush data to HDFS? How IGFS > flush data to HDFS? Async or sync? > > If it a async mode, when IGFS can't hold more data, it will using cache > expire policy. And if the data which is expired is not persisted to HDFS > timely, the data will be lost. Is is possible? > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/How-IGFS-keep-sync-with-HDFS-tp9258p9262.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: How IGFS keep sync with HDFS?
If IGFS can't hold more input data, will it flush data to HDFS? How IGFS flush data to HDFS? Async or sync? If it a async mode, when IGFS can't hold more data, it will using cache expire policy. And if the data which is expired is not persisted to HDFS timely, the data will be lost. Is is possible? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-IGFS-keep-sync-with-HDFS-tp9258p9262.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
How IGFS keep sync with HDFS?
I am using ignite hadoop accelerate to improve my map-reduce job. I am always encountered "java heap size" problem, though I have offer more than 80GB to ignite. The map-reduce job will write more than 150GB data to IGFS. When IGFS will sync data to HDFS? When processing a map-reduce job, do I have to make IGFS big enough to hold all the output data from my map-reduce job? Here is my configuration: http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xmlns:util="http://www.springframework.org/schema/util; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd;> Spring file for Ignite node configuration with IGFS and Apache Hadoop map-reduce support enabled. Ignite node will start with this configuration by default. /home/appadmin/hadoop-2.7.2/etc/hadoop/core-site.xml JVM option: if [ -z "$JVM_OPTS" ] ; then if [[ `"$JAVA" -version 2>&1 | egrep "1\.[7]\."` ]]; then JVM_OPTS="-Xms1g -Xmx1g -server -XX:+AggressiveOpts -XX:MaxPermSize=256m" else JVM_OPTS="-server -Xms20g -Xmx80g -server -XX:+AggressiveOpts -XX:MaxMetaspaceSize=10g -XX:NewSize=512m -XX:SurvivorRatio=6 -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=4 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M -XX:ConcGCThreads=16 -XX:G1HeapWastePercent=10 -XX:+UseTLAB -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC" fi fi -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-IGFS-keep-sync-with-HDFS-tp9258.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
