Hi,
I am using onHeap along with the default offHeap memory in my Ignite 2.0
server nodes. I am getting out of memory error even though there is enough
memory available offHeap.
My understanding (based on my earlier posts) is that onHeap is just a cache
for offHeap. If that's how it works then the heap space should never fill up
if there is space available offHeap.
Could you please help me understand how these 2 memories work with each
other. The exception I have got is pasted below.
Thanks
[19:30:14,582][SEVERE][grid-nio-worker-tcp-comm-2-#23%NRO%][GridDirectParser]
Failed to read message [msg=GridIoMessage [plc=0, topic=null, topicOrd=-1,
ordered=false, timeout=0, skipOnTimeout=false, msg=null],
buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
reader=DirectMessageReader [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
baseOff=139667108316592, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=GridDhtTxPrepareRequest [nearNodeId=null, futId=null,
miniId=0, topVer=null, invalidateNearEntries=null, nearWrites=null,
owned=null, nearXidVer=null, subjId=null, taskNameHash=0, preloadKeys=null,
super=GridDistributedTxPrepareRequest [threadId=164,
concurrency=PESSIMISTIC, isolation=REPEATABLE_READ,
writeVer=GridCacheVersion [topVer=113311658, order=1501950546676,
nodeOrder=6], timeout=56261, reads=null, writes=null, dhtVers=null,
txSize=0, plc=2, txState=null, flags=onePhaselast,
super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=113311658,
order=1501950546597, nodeOrder=6], committedVers=null, rolledbackVers=null,
cnt=0, super=GridCacheMessage [msgId=1294187, depInfo=null, err=null,
skipPrepare=false, cacheId=0, cacheId=0]]]], mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
baseOff=139667108316592, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=IgniteTxEntry [key=BinaryObjectImpl [arr= true,
ctx=false, start=0], cacheId=1593518529, txKey=null, val=[op=NOOP,
val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=null, filtersPassed=false,
filtersSet=false, entry=null, prepared=0, locked=false, nodeId=null,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0,
partUpdateCntr=0, serReadVer=null, xidVer=null], mapIt=null, it=null,
arrPos=-1, keyDone=false, readSize=1, readItems=0, prim=0, primShift=0,
uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true],
state=0], StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
baseOff=139667108316592, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=[op=UPDATE, val=null], mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
baseOff=139667108316592, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=true, msg=BinaryObjectImpl [arr= false, ctx=false, start=0],
mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0,
prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0,
lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
baseOff=139667108316592, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null,
null, null]], lastRead=true], ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@31cec787, idx=2,
bytesRcvd=428406858156, bytesSent=278366, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2,
igniteInstanceName=NRO, finished=false, hashCode=426420385,
interrupted=false, runner=grid-nio-worker-tcp-comm-2-#23%NRO%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=176 lim=20272 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=139784,
sentCnt=0, reserved=true, lastAck=139776, nodeLeft=false,
node=TcpDiscoveryNode [id=02c2d1ae-3676-4c21-a953-60d81a41c47d,
addrs=[10.120.249.50, 127.0.0.1], sockAddrs=[/127.0.0.1:47500,
/10.120.249.50:47500], discPort=47500, order=6, intOrder=6,
lastExchangeTime=1501831543578, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=0, queueLimit=131072,
reserveCnt=2, pairedConnections=true], outRecovery=null,
super=GridNioSessionImpl [locAddr=/10.120.194.51:47100,
rmtAddr=/10.120.249.50:35310, createTime=1502047810795, closeTime=0,
bytesSent=28, bytesRcvd=20314, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1502047810795, lastSndTime=1502047811434,
lastRcvTime=1502047811434, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@16786b96, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.lang.OutOfMemoryError: Java heap space
[19:30:14,581][SEVERE][query-#23216%NRO%][GridCacheIoManager] Failed to
process message [senderId=9d1131b7-a590-4e7f-b0ab-29bcbd5716d0,
messageType=class o.a.i.i.processors.cache.query.GridCacheQueryRequest]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.ignite.internal.util.io.GridUnsafeDataOutput.array(GridUnsafeDataOutput.java:98)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:203)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9969)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:179)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:245)
at
org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9969)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10033)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:879)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:852)
at
org.apache.ignite.internal.processors.cache.GridCacheMessage.marshalCollection(GridCacheMessage.java:492)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.prepareMarshal(GridCacheQueryResponse.java:137)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onSend(GridCacheIoManager.java:912)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1108)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.sendQueryResponse(GridCacheDistributedQueryManager.java:324)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.onPageReady(GridCacheDistributedQueryManager.java:469)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1349)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:231)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114)
Exception in thread "query-#23216%NRO%" java.lang.OutOfMemoryError: Java
heap space
at
org.apache.ignite.internal.util.io.GridUnsafeDataOutput.array(GridUnsafeDataOutput.java:98)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:203)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9969)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:179)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:245)
at
org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9969)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10033)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:879)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:852)
at
org.apache.ignite.internal.processors.cache.GridCacheMessage.marshalCollection(GridCacheMessage.java:492)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.prepareMarshal(GridCacheQueryResponse.java:137)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onSend(GridCacheIoManager.java:912)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1108)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.sendQueryResponse(GridCacheDistributedQueryManager.java:324)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.onPageReady(GridCacheDistributedQueryManager.java:469)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1349)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:231)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114)
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/OOM-in-Heap-though-offHeap-is-available-free-tp16070.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
