Hi Usein, Which Java version do you have? There was a similar thread already where this exception was fixed by upgrading to the latest one: http://apache-ignite-users.70518.x6.nabble.com/Caused-by-org-h2-jdbc-JdbcSQLException-General-error-quot-java-lang-IllegalMonitorStateException-Attt-td15684.html
Can you try to upgrade as well? If you confirm that it indeed helps, then it needs to be documented. -Val On Thu, Aug 17, 2017 at 3:58 AM, Usein Faradzhev <usein_faradz...@epam.com> wrote: > Hello. > > > > We are trying to use the Ignite Memory File System and sometimes Ignite > can’t write file to IGFS and can’t read. What is this happens? > > Below is an example for Cloudera Quick Start VM 5.10.0 and error, also > configuration and full log in attachments. This problems arise on our > cluster with CentOS 7 and CDH 5.11.1 too. > > > > In-Memory Hadoop Accelerator: > > Version 2.1.0 > > Date 2017-07-27 > > File http://apache-mirror.rbc.ru/pub/apache//ignite/2.1.0/ > apache-ignite-hadoop-2.1.0-bin.zip > <http://apache-mirror.rbc.ru/pub/apache/ignite/2.1.0/apache-ignite-hadoop-2.1.0-bin.zip> > > > > [cloudera@quickstart ~]$ ls -l dtm_ekp_scoring_plan_oper75.csv > > -rw-r--r-- 1 cloudera cloudera 19579883 Aug 16 03:53 > dtm_ekp_scoring_plan_oper75.csv > > > > [cloudera@quickstart ~]$ hdfs dfs -mkdir -p igfs://igfs@/user/cloudera/ > dtm_ekp_scoring_plan_oper/ > > [cloudera@quickstart ~]$ hdfs dfs -put dtm_ekp_scoring_plan_oper75.csv > igfs://igfs@/user/cloudera/dtm_ekp_scoring_plan_oper/ > > put: Failed to flush data during stream close [path=/user/cloudera/dtm_ekp_ > scoring_plan_oper/dtm_ekp_scoring_plan_oper75.csv._COPYING_, > fileInfo=IgfsFileInfo [len=0, blockSize=65536, > lockId=4600eafed51-15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, > affKey=null, fileMap=IgfsFileMap [ranges=null], evictExclude=true]] > > > > > > [2017-08-17 03:13:07,951][ERROR][igfs-#47%null%][GridNearTxLocal] > Heuristic transaction failure. > > class org.apache.ignite.internal.transactions. > IgniteTxHeuristicCheckedException: Failed to locally write to cache (all > transaction entries will be invalidated, however there was a window when > entries for this transaction were visible to others): GridNearTxLocal > [mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping > [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114, > txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114], > val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]], > prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], > entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, > explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, > filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=954, > super=GridDistributedCacheEntry > [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=954, > val=IgfsBlockKey [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, > blockId=130, affKey=null, evictExclude=true], hasValBytes=true], val=null, > startVer=1502964754897, ver=GridCacheVersion [topVer=114444755, > order=1502964754897, nodeOrder=1], hash=236544549, > extras=GridCacheMvccEntryExtras > [mvcc=GridCacheMvcc [locs=[GridCacheMvccCandidate > [nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, ver=GridCacheVersion > [topVer=114444755, order=1502964754896, nodeOrder=1], threadId=69, id=152, > topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], reentry=null, > otherNodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, > otherVer=GridCacheVersion [topVer=114444755, order=1502964754896, > nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, > serOrder=null, key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], > masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_ > implicit=1|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, > nextVer=null]], rmts=null]], flags=2]]], prepared=1, locked=false, > nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, locMapped=false, > expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, > serReadVer=null, xidVer=GridCacheVersion [topVer=114444755, > order=1502964754896, nodeOrder=1]]], explicitLock=false, dhtVer=null, > last=false, nearEntries=0, clientFirst=false, > node=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6]], > nearLocallyMapped=false, colocatedLocallyMapped=true, needCheckBackup=null, > hasRemoteLocks=false, thread=igfs-#47%null%, > mappings=IgniteTxMappingsSingleImpl > [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry > [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114, > txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114], > val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]], > prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], > entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, > explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, > filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=954, > super=GridDistributedCacheEntry > [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=954, > val=IgfsBlockKey [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, > blockId=130, affKey=null, evictExclude=true], hasValBytes=true], val=null, > startVer=1502964754897, ver=GridCacheVersion [topVer=114444755, > order=1502964754897, nodeOrder=1], hash=236544549, > extras=GridCacheMvccEntryExtras > [mvcc=GridCacheMvcc [locs=[GridCacheMvccCandidate > [nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, ver=GridCacheVersion > [topVer=114444755, order=1502964754896, nodeOrder=1], threadId=69, id=152, > topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], reentry=null, > otherNodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, > otherVer=GridCacheVersion [topVer=114444755, order=1502964754896, > nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, > serOrder=null, key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey > [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130, > affKey=null, evictExclude=true], hasValBytes=true], > masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_ > implicit=1|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, > nextVer=null]], rmts=null]], flags=2]]], prepared=1, locked=false, > nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, locMapped=false, > expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, > serReadVer=null, xidVer=GridCacheVersion [topVer=114444755, > order=1502964754896, nodeOrder=1]]], explicitLock=false, dhtVer=null, > last=false, nearEntries=0, clientFirst=false, > node=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6]], > super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], > dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter > [completedBase=null, sndTransformedVals=false, depEnabled=false, txState= > IgniteTxImplicitSingleStateImpl [init=true, recovery=false], > super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=114444755, > order=1502964754896, nodeOrder=1], writeVer=GridCacheVersion > [topVer=114444755, order=1502964754898, nodeOrder=1], implicit=true, > loc=true, threadId=69, startTime=1502964787910, > nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, > startVer=GridCacheVersion [topVer=114444755, order=1502964754896, > nodeOrder=1], endVer=null, isolation=READ_COMMITTED, > concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=true, plc=2, > commitVer=GridCacheVersion [topVer=114444755, order=1502964754896, > nodeOrder=1], finalizing=NONE, invalidParts=null, state=COMMITTING, > timedOut=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], > duration=0ms, onePhaseCommit=true], size=1]]] > > at org.apache.ignite.internal. > processors.cache.transactions.IgniteTxLocalAdapter.userCommit( > IgniteTxLocalAdapter.java:834) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal. > localFinish(GridNearTxLocal.java:3030) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxFinishFuture. > finish(GridNearTxFinishFuture.java:405) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal$16.apply( > GridNearTxLocal.java:3180) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal$16.apply( > GridNearTxLocal.java:3172) > > at org.apache.ignite.internal. > util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382) > > at org.apache.ignite.internal. > util.future.GridFutureAdapter.listen(GridFutureAdapter.java:352) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync( > GridNearTxLocal.java:3172) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal.optimisticPutFuture( > GridNearTxLocal.java:2382) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal. > putAllAsync0(GridNearTxLocal.java:803) > > at org.apache.ignite.internal. > processors.cache.distributed.near.GridNearTxLocal. > putAllAsync(GridNearTxLocal.java:362) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter$35.op(GridCacheAdapter.java:2814) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter$AsyncOp.op(GridCacheAdapter.java:5076) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.asyncOp(GridCacheAdapter.java:4225) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter$AsyncOpRetryFuture.execute(GridCacheAdapter.java:4797) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.asyncOp(GridCacheAdapter.java:4154) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.putAllAsync0(GridCacheAdapter.java:2812) > > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.putAllAsync(GridCacheAdapter.java:2804) > > at org.apache.ignite.internal.processors.cache. > GridCacheProxyImpl.putAllAsync(GridCacheProxyImpl.java:827) > > at org.apache.ignite.internal.processors.igfs. > IgfsDataManager.storeBlocksAsync(IgfsDataManager.java:1084) > > at org.apache.ignite.internal.processors.igfs. > IgfsDataManager.access$1500(IgfsDataManager.java:96) > > at org.apache.ignite.internal.processors.igfs. > IgfsDataManager$5.run(IgfsDataManager.java:1000) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: class org.apache.ignite.IgniteException: Runtime failure on > search row: org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$SearchRow@7b30c9d0 > > at org.apache.ignite.internal. > processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1632) > > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke( > IgniteCacheOffheapManagerImpl.java:1204) > > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl. > java:352) > > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3179) > > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.innerSet(GridCacheMapEntry.java:979) > > at org.apache.ignite.internal. > processors.cache.transactions.IgniteTxLocalAdapter.userCommit( > IgniteTxLocalAdapter.java:666) > > ... 24 more > > Caused by: java.lang.IllegalMonitorStateException: Attempted to release > write lock while not holding it [lock=00007f1bd4be0270, > state=000103ba000025c3 > > at org.apache.ignite.internal.util.OffheapReadWriteLock. > writeUnlock(OffheapReadWriteLock.java:263) > > at org.apache.ignite.internal.pagemem.impl. > PageMemoryNoStoreImpl.writeUnlock(PageMemoryNoStoreImpl.java:490) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.util.PageHandler. > writeUnlock(PageHandler.java:377) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.util.PageHandler. > writePage(PageHandler.java:287) > > at org.apache.ignite.internal. > processors.cache.persistence.DataStructure.write(DataStructure.java:281) > > at org.apache.ignite.internal. > processors.cache.persistence.freelist.FreeListImpl. > insertDataRow(FreeListImpl.java:493) > > at org.apache.ignite.internal. > processors.cache.persistence.RowStore.addRow(RowStore.java:84) > > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow( > IgniteCacheOffheapManagerImpl.java:1258) > > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3875) > > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3833) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.BPlusTree$Invoke. > invokeClosure(BPlusTree.java:2988) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.BPlusTree$Invoke.access$ > 6200(BPlusTree.java:2882) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.BPlusTree.invokeDown( > BPlusTree.java:1713) > > at org.apache.ignite.internal. > processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1602) > > ... 29 more > > > > > > Best Regards, > > *Usein Faradzhev* > Chief Software Engineer > > EPAM Systems > > Togliatti office, Russia > > GMT+3 (Standard) / GMT+4 (Daylight) > > > > Mobile phone: +7 919 815 00 32 <+7%20919%20815-00-32> > > Skype: usein.faradgev > > E-mail: usein_faradz...@epam.com > > > <http://www.epam.com> > > CONFIDENTIALITY CAUTION AND DISCLAIMER > This message is intended only for the use of the individual(s) or > entity(ies) to which it is addressed and contains information that is > legally privileged and confidential. If you are not the intended recipient, > or the person responsible for delivering the message to the intended > recipient, you are hereby notified that any dissemination, distribution or > copying of this communication is strictly prohibited. All unintended > recipients are obliged to delete this message and destroy any printed > copies. > > >
