Re: Proper config for IGFS eviction
Hi, There are metaCacheCfg and dataCacheCfg in IgniteConfiguration.FileSystemConfiguration, which allows to configure caches for IGFS. Inside this configuration you can set DataRegion with persistence. Have you tried this? Evgenii -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Proper config for IGFS eviction
Hello Engrdean, You should enable persistence on your DataRegionConfiguration to make it possible to evict files metadata pages from memory to disk. 2018-08-09 19:49 GMT+03:00 engrdean : > I've been struggling to find a configuration that works successfully for > IGFS > with hadoop filesystem caching. Anytime I attempt to load more data than > what will fit into memory on my Ignite node, the ignite process crashes. > > The behavior I am looking for is that old cache entries will be evicted > when > I try to write new data to IGFS that exceeds the available memory on the > server. I can see that my data is being persisted into HDFS, but I seem to > be limited to the amount of physical memory on my Ignite server at the > moment. I am using the teragen example to generate the files on hadoop for > the purposes of this test like so: > > time hadoop-ig jar > /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar > teragen 1 igfs://i...@myserver.com/tmp/output1 > > If I have systemRegionMaxSize set to a value less than the physical memory > on my ignite server, then the message is something like this: > > /class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of > memory in data region [name=sysMemPlc, initSize=1.0 GiB, maxSize=14.0 GiB, > persistenceEnabled=false] Try the following: > ^-- Increase maximum off-heap memory size > (DataRegionConfiguration.maxSize) > ^-- Enable Ignite persistence (DataRegionConfiguration. > persistenceEnabled) > ^-- Enable eviction or expiration policies > / > If I increase the systemRegionMaxSize to a value greater than the physical > memory on my ignite server, the message is something like this: > > /[2018-08-09 12:16:08,174][ERROR][igfs-#171][GridNearTxLocal] Heuristic > transaction failure. > class > org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: > Failed to locally write to cache (all transaction entries will be > invalidated, however there was a window when entries for this transaction > were visible to others): GridNearTxLocal [mappings=IgniteTxMappingsImpl [], > nearLocallyMapped=false, colocatedLocallyMapped=true, needCheckBackup=null, > hasRemoteLocks=false, trackTimeout=false, lb=null, thread=igfs-#171, > mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter > [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], > explicitLock=false, > super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, > depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=[-313790114], > recovery=false, txMap=[IgniteTxEntry [key=KeyCacheObjectImpl [part=504, > val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, > blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], > cacheId=-313790114, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=504, > val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, > blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], > cacheId=-313790114], val=[op=CREATE, val=CacheObjectByteArrayImpl > [arrLen=65536]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], > entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, > explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, > filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=504, > super=GridDistributedCacheEntry [super=GridCacheMapEntry > [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey > [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, > affKey=null, evictExclude=true], hasValBytes=true], val=null, > startVer=1533830728270, ver=GridCacheVersion [topVer=145310277, > order=1533830728270, nodeOrder=1], hash=-915370253, > extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc > [locs=[GridCacheMvccCandidate [nodeId=6ed33eb9-2103-402c- > afab-a415c8f08f2f, > ver=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1], > threadId=224, id=258264, topVer=AffinityTopologyVersion [topVer=1, > minorTopVer=0], reentry=null, > otherNodeId=6ed33eb9-2103-402c-afab-a415c8f08f2f, > otherVer=GridCacheVersion > [topVer=145310277, order=1533830728268, nodeOrder=1], mappedDhtNodes=null, > mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl > [part=504, val=IgfsBlockKey > [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, > affKey=null, evictExclude=true], hasValBytes=true], > masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_ > implicit=0|dht_local=1|near_local=0|removed=0|read=0, > prevVer=GridCacheVersion [topVer=145310277, order=1533830728268, > nodeOrder=1], nextVer=GridCacheVersion [topVer=145310277, > order=1533830728268, nodeOrder=1]]], rmts=null]], flags=2]]], prepared=1, > locked=false, nodeId=6ed33eb9-2103-402c-afab-a415c8f08f2f, > locMapped=false, > expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, > serReadVer=null, xidVer=GridCacheVersion [topVer=145310277, > order=1533830728268, nodeOrder=1]],
Proper config for IGFS eviction
I've been struggling to find a configuration that works successfully for IGFS with hadoop filesystem caching. Anytime I attempt to load more data than what will fit into memory on my Ignite node, the ignite process crashes. The behavior I am looking for is that old cache entries will be evicted when I try to write new data to IGFS that exceeds the available memory on the server. I can see that my data is being persisted into HDFS, but I seem to be limited to the amount of physical memory on my Ignite server at the moment. I am using the teragen example to generate the files on hadoop for the purposes of this test like so: time hadoop-ig jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen 1 igfs://i...@myserver.com/tmp/output1 If I have systemRegionMaxSize set to a value less than the physical memory on my ignite server, then the message is something like this: /class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of memory in data region [name=sysMemPlc, initSize=1.0 GiB, maxSize=14.0 GiB, persistenceEnabled=false] Try the following: ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize) ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled) ^-- Enable eviction or expiration policies / If I increase the systemRegionMaxSize to a value greater than the physical memory on my ignite server, the message is something like this: /[2018-08-09 12:16:08,174][ERROR][igfs-#171][GridNearTxLocal] Heuristic transaction failure. class org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Failed to locally write to cache (all transaction entries will be invalidated, however there was a window when entries for this transaction were visible to others): GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMapped=true, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, lb=null, thread=igfs-#171, mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=[-313790114], recovery=false, txMap=[IgniteTxEntry [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114], val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=504, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], val=null, startVer=1533830728270, ver=GridCacheVersion [topVer=145310277, order=1533830728270, nodeOrder=1], hash=-915370253, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=[GridCacheMvccCandidate [nodeId=6ed33eb9-2103-402c-afab-a415c8f08f2f, ver=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1], threadId=224, id=258264, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], reentry=null, otherNodeId=6ed33eb9-2103-402c-afab-a415c8f08f2f, otherVer=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52879, affKey=null, evictExclude=true], hasValBytes=true], masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1], nextVer=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1]]], rmts=null]], flags=2]]], prepared=1, locked=false, nodeId=6ed33eb9-2103-402c-afab-a415c8f08f2f, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=145310277, order=1533830728268, nodeOrder=1]], IgniteTxEntry [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52880, affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=504, val=IgfsBlockKey [fileId=c976b6f1561-689b0ba5-6920-4b52-a614-c2360d0acff4, blockId=52880,
