Hi, Sergey, 1) OOME happens because Ignite 2.0 by default uses off-heap memory for caches. So, you should not give Ignite large heap any more. Instead, you should properly configure the offheap memory policies for the caches. This is the configuration defined by bean class org.apache.ignite.configuration.MemoryConfiguration . Please see class JavaDoc and properties systemCacheMaxSize , systemCacheInitialSize .
2) In Ignite 2.0 metaCacheName and dataCacheName properties can be left null, in which case default values will be used, please see javadocs of org.apache.ignite.configuration.FileSystemConfiguration#setMetaCacheConfiguration and org.apache.ignite.configuration.FileSystemConfiguration#setDataCacheConfiguration . WRT used memory: there is a request to expose IGFS metrics : https://issues.apache.org/jira/browse/IGNITE-5168 , seggected patch exposes them via MXBeans, it is done, but not yet accepted. You can take the patch and build Ignite with it. 3) It looks like the advice "*Consider configuring MemoryEventStorageSpi or another EventStorageSpi implementation via IgniteConfiguration.setEventStorageSpi() configuration property.*" is quite relevant there -- please try to configure the event storage spi. On Tue, May 30, 2017 at 11:06 PM, Sergey Semichev <[email protected]> wrote: > I have configured and run IGFS as Hadoop FileSystem successfully. > There are three issues, that I couldn't solve. Any help will be > appreciated! > > 1) Out of memory error when I try to load to IGFS 400Mb file > 2) FileSystemConfiguration - metaCacheName and dataCacheName properties > 3) includeEventTypes and task tasks execution statistics > > See details below > > Environment > -MacOS Sierra > -YARN 2.8 in Pseudo-distributed mode > -Ignite 2.0 > > -JVM setting in ignite.sh > JVM_OPTS="-server -Xms6g -Xmx6g -XX:NewSize=512m -XX:SurvivorRatio=6 > -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 > -XX:GCTimeRatio=4 > -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M > -XX:ConcGCThreads=8 -XX:G1HeapWastePercent=10 -XX:+UseTLAB > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC" > > -Ignite configuration files > https://github.com/ssemichev/ignite-configs/blob/master/ > igfs-hadoop-fs/igfs-hadoop-fs-config.xml > https://github.com/ssemichev/ignite-configs/blob/master/ > igfs-hadoop-fs/ignite_conf/core-site.xml > https://github.com/ssemichev/ignite-configs/blob/master/ > igfs-hadoop-fs/ignite_conf/hdfs-site.xml > https://github.com/ssemichev/ignite-configs/blob/master/ > igfs-hadoop-fs/ignite_conf/mapred-site.xml > > ============================================= > 1) Out of memory error when I try to load to IGFS 400Mb file > I can copy a 60Mb file, but it fails when I try to copy large 410Mb file > > >./bin/ignite.sh ./configs/igfs-hadoop-fs/igfs-hadoop-fs-config.xml > Ignite node started OK (id=9c9a20a5) > Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=6.0GB] > > visor> top > Hosts: 1 > Summary: > +-------------------------------------+ > | Total hosts | 1 | > | Total nodes | 1 | > | Total CPUs | 8 | > | Avg. CPU load | 0.17 % | > | Avg. free heap | 95.00 % | > | Avg. Up time | 00:05:46 | > | Snapshot time | 05/30/17, 14:49:22 | > > visor> node > | OS info | Mac OS X x86_64 10.12.5 > | > | OS user | ssemichev > | > | Deployment mode | SHARED > | > | Language runtime | Java Platform API Specification ver. 1.8 > | > | Ignite version | 2.0.0 > | Busy time % | 4.07% > | > | Current CPU load % | 0.23% > | > | Average CPU load % | 0.42% > | > | Heap memory initialized | 6gb > | > | Heap memory used | 224mb > | > | Heap memory committed | 6gb > | > | Heap memory maximum | 6gb > | > | Non-heap memory initialized | 2mb > | > | Non-heap memory used | 51mb > | > | Non-heap memory committed | 52mb > | > | Non-heap memory maximum | 0 > | > | Current thread count | 55 > | > | Maximum thread count | 61 > | > | Total started thread count | 131 > | > | Current daemon thread count | 13 > | > > > >ls -alh /opt/hadoop-2.8.0.tar.gz > 410M May 22 12:25 /opt/hadoop-2.8.0.tar.gz > > >hdfs --config ./ignite_conf dfs -put /opt/hadoop-2.8.0.tar.gz /in > put: Failed to flush data during stream close [path=/in._COPYING_, > fileInfo=IgfsFileInfo [len=0, blockSize=65536, > lockId=fdc3caa5c51-9c9a20a5-2b17-460a-bec4-ea909f8e49a0, affKey=null, > fileMap=IgfsFileMap [ranges=null], evictExclude=true]] > > >From Ignite terminal output > [ERROR][igfs-igfs-ipc-#81%null%][GridNearTxLocal] Heuristic transaction > failure. > class > org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: > Failed to locally write to cache (all transaction entries will be > invalidated, however there was a window when entries for this transaction > were visible to others): GridNearTxLocal [mappings=IgniteTxMappingsImpl [], > nearLocallyMapped=false, colocatedLocallyMapped=true, needCheckBackup=null, > hasRemoteLocks=false, thread=igfs-igfs-ipc-#81%null%, > mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter > [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], > explicitLock=false, > super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, > depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=GridIntList > [idx=1, arr=[-313518151]], txMap={IgniteTxKey [key=KeyCacheObjectImpl > [part=1, val=cdc3caa5c51-0d6b159c-291a-4630-a948-dcd9fa98ec1c, > hasValBytes=true], cacheId=-313518151]=IgniteTxEntry > [key=KeyCacheObjectImpl > [part=1, val=cdc3caa5c51-0d6b159c-291a-4630-a948-dcd9fa98ec1c, > hasValBytes=true], cacheId=-313518151, txKey=IgniteTxKey > [key=KeyCacheObjectImpl [part=1, > val=cdc3caa5c51-0d6b159c-291a-4630-a948-dcd9fa98ec1c, hasValBytes=true], > cacheId=-313518151], val=[op=TRANSFORM, val=CacheObjectImpl > [val=IgfsFileInfo [len=0, blockSize=65536, > lockId=fdc3caa5c51-9c9a20a5-2b17-460a-bec4-ea909f8e49a0, affKey=null, > fileMap=IgfsFileMap [ranges=null], evictExclude=true], hasValBytes=true]], > prevVal=[op=TRANSFORM, val=CacheObjectImpl [val=IgfsFileInfo [len=0, > blockSize=65536, lockId=fdc3caa5c51-9c9a20a5-2b17-460a-bec4-ea909f8e49a0, > affKey=null, fileMap=IgfsFileMap [ranges=null], evictExclude=true], > hasValBytes=true]], oldVal=[op=NOOP, val=null], > entryProcessorsCol=[IgniteBiTuple [val1=IgfsMetaFileUnlockProcessor > [modificationTime=1496170357665, updateSpace=false, space=0, > affRange=null], > val2=[Ljava.lang.Object;@6a5ba764]], ttl=-1, conflictExpireTime=-1, > conflictVer=null, explicitVer=null, dhtVer=null, filters=[], > filtersPassed=false, filtersSet=true, entry=GridDhtColocatedCacheEntry > [super=GridDhtCacheEntry [rdrs=[], locPart=GridDhtLocalPartition > [rmvQueueMaxSize=256, rmvdEntryTtl=10000, id=1, > store=org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$CacheDataStoreImpl@6f95f794, > lastApplied=0, shouldBeRenting=false, state=OWNING, reservations=0, > empty=false, createTime=05/30/2017 14:43:42], > super=GridDistributedCacheEntry [super=GridCacheMapEntry > [key=KeyCacheObjectImpl [part=1, > val=cdc3caa5c51-0d6b159c-291a-4630-a948-dcd9fa98ec1c, hasValBytes=true], > val=CacheObjectImpl [val=IgfsFileInfo [len=0, blockSize=65536, > lockId=fdc3caa5c51-9c9a20a5-2b17-460a-bec4-ea909f8e49a0, affKey=null, > fileMap=IgfsFileMap [ranges=null], evictExclude=true], hasValBytes=true], > startVer=1496169896316, ver=GridCacheVersion [topVer=107649824, > order=1496169890081, nodeOrder=1], hash=-684243145, > extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc > [locs=[GridCacheMvccCandidate [nodeId=9c9a20a5-2b17-460a- > bec4-ea909f8e49a0, > ver=GridCacheVersion [topVer=107649824, order=1496169896315, nodeOrder=1], > threadId=209, id=1568, topVer=AffinityTopologyVersion [topVer=2, > minorTopVer=0], reentry=null, > otherNodeId=9c9a20a5-2b17-460a-bec4-ea909f8e49a0, > otherVer=GridCacheVersion > [topVer=107649824, order=1496169896315, nodeOrder=1], mappedDhtNodes=null, > mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl > [part=1, val=cdc3caa5c51-0d6b159c-291a-4630-a948-dcd9fa98ec1c, > hasValBytes=true], > masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_ > implicit=0|dht_local=1|near_local=0|removed=0|read=0, > prevVer=null, nextVer=null]], rmts=null]], flags=2]]]], prepared=0, > locked=true, nodeId=9c9a20a5-2b17-460a-bec4-ea909f8e49a0, locMapped=false, > expiryPlc=null, transferExpiryPlc=false, flags=4, partUpdateCntr=0, > serReadVer=null, xidVer=GridCacheVersion [topVer=107649824, > order=1496169896315, nodeOrder=1]]}, recovery=false], super=IgniteTxAdapter > [xidVer=GridCacheVersion [topVer=107649824, order=1496169896315, > nodeOrder=1], writeVer=GridCacheVersion [topVer=107649824, > order=1496169896317, nodeOrder=1], implicit=false, loc=true, threadId=209, > startTime=1496170357663, nodeId=9c9a20a5-2b17-460a-bec4-ea909f8e49a0, > startVer=GridCacheVersion [topVer=107649824, order=1496169896315, > nodeOrder=1], endVer=null, isolation=REPEATABLE_READ, > concurrency=PESSIMISTIC, timeout=0, sysInvalidate=false, sys=true, plc=2, > commitVer=GridCacheVersion [topVer=107649824, order=1496169896315, > nodeOrder=1], finalizing=NONE, invalidParts=null, state=COMMITTING, > timedOut=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], > duration=0ms, onePhaseCommit=true], size=1]]] > > [ERROR][igfs-igfs-ipc-#81%null%][GridNearTxLocal] Failed to invalidate > transaction entries while reverting a commit. > class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Not > enough > memory allocated (consider increasing memory policy size or enabling > evictions) [policyName=sysMemPlc, size=104.9 MB] > > After the failed attempt > visor> node > | Heap memory initialized | 6gb > | > | Heap memory used | 1gb > | > | Heap memory committed | 6gb > | > | Heap memory maximum | 6gb > | > | Non-heap memory initialized | 2mb > | > | Non-heap memory used | 60mb > | > | Non-heap memory committed | 61mb > | > | Non-heap memory maximum | 0 > | > | Current thread count | 63 > | > | Maximum thread count | 73 > | > | Total started thread count | 336 > | > | Current daemon thread count | 13 > | > > > >ls -alh /opt/apache-ignite-hadoop-2.0.0-bin.zip > 56M May 23 13:10 /opt/apache-ignite-hadoop-2.0.0-bin.zip > > >hdfs --config ./ignite_conf dfs -put > /opt/apache-ignite-hadoop-2.0.0-bin.zip / > > >hdfs --config ./ignite_conf dfs -ls -h / > 55.7 M 2017-05-30 15:05 /apache-ignite-hadoop-2.0.0-bin.zip > > > ============================================= > 2) FileSystemConfiguration - metaCacheName and dataCacheName properties > It seems that Ignite 2.0 doesn't have them. Do I need to configure a > cache(s) for IGFS setup manually? > How can I check how much memory is used for IGFS? Visor statistics is not > very useful, it shows only total Heap and Non-heap values. > > ============================================= > 3) includeEventTypes and task tasks execution statistics > I added includeEventTypes section in the config > https://github.com/ssemichev/ignite-configs/blob/master/ > igfs-hadoop-fs/igfs-hadoop-fs-config.xml > > When I run > visor>tasks > it prints the error in the ignite.sh terminal > > Failed to execute job > [jobId=4ceffda5c51-05e150ae-87db-427c-8e68-e6bc83158476, > ses=GridJobSessionImpl [ses=GridTaskSessionImpl > [taskName=o.a.i.i.v.node.VisorNodeEventsCollectorTask, dep=LocalDeployment > [super=GridDeployment [ts=1496171110859, depMode=SHARED, > clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, > clsLdrId=9ed5fba5c51-53c69937-a6b3-483d-ba88-0ba5c7f86d35, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=false, usage=0]], > taskClsName=o.a.i.i.v.node.VisorNodeEventsCollectorTask, > sesId=3ceffda5c51-05e150ae-87db-427c-8e68-e6bc83158476, > startTime=1496173301316, endTime=9223372036854775807, > taskNodeId=05e150ae-87db-427c-8e68-e6bc83158476, > clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, > cpSpi=null, > failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, > subjId=05e150ae-87db-427c-8e68-e6bc83158476, mapFut=IgniteFuture > [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, > hash=1743289220]], execName=null], > jobId=4ceffda5c51-05e150ae-87db-427c-8e68-e6bc83158476]] > class org.apache.ignite.IgniteException: Failed to query events because > default no-op event storage SPI is used. Consider configuring > MemoryEventStorageSpi or another EventStorageSpi implementation via > IgniteConfiguration.setEventStorageSpi() configuration property. > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/IGFS-as-Hadoop-FileSystem-tp13249.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
