Hi Ilya Kasnacheev, On Ingite 2.7 No code is using affinityCall() Haven’t enabled cache statistics
Could you please answer my questions in red below: Try to increase Xmx up to 4Gb or higher or avoid G1GC usage on small heaps as it is very sensitive to free heap memory. James>> Yes, it helps. As I understand all the data is kept in non-heap memory, But could you advise when the heap memory will be used, like “order”, “group by” operation ? Looks like you have Visor node (or may be web-console) in grid. Is OOM happened only when Visor attached to grid? James>> Most like to occur if execute a complicate query. Another question is: if I set Xmx=6GB, is it possible to avoide out-of-memory failure. I regard this very critical because it make node halt. Best Regards, James Wang M/WeChat: +86 135 1215 1134 From: Ilya Kasnacheev <ilya.kasnach...@gmail.com<mailto:ilya.kasnach...@gmail.com>> Sent: Tuesday, March 5, 2019 09:44 PM To: user@ignite.apache.org<mailto:user@ignite.apache.org> Subject: Re: java.lang.OutOfMemoryError: GC overhead limit exceeded Hello! Is there a chance you were using affinityCall() and you're on Ignite 2.7? We have a bug which can cause spurious OOMs here: https://issues.apache.org/jira/browse/IGNITE-10925 - disabling cache statistics will help Regards, -- Ilya Kasnacheev сб, 2 мар. 2019 г. в 18:13, James Wang 王升平 (edvance CN) <james.w...@edvancesecurity.com<mailto:james.w...@edvancesecurity.com>>: Hi Support, Try to increase Xmx up to 4Gb or higher or avoid G1GC usage on small heaps as it is very sensitive to free heap memory. James>> will increase heap memory and retry. As I understand all the data is kept in non-heap memory, But could you advise when the heap memory will be used, like “order”, “group by” operation ? Looks like you have Visor node (or may be web-console) in grid. Is OOM happened only when Visor attached to grid? James>> Most like to occur if execute a complicate query. Another question is: if I set Xmx=6GB, is it possible to avoide out-of-memory failure. I regard this very critical because it make node halt. Thank you. Best Regards, James Wang M/WeChat: +86 135 1215 1134 From: Andrey Mashenkov <andrey.mashen...@gmail.com<mailto:andrey.mashen...@gmail.com>> Sent: Friday, March 1, 2019 06:50 PM To: user@ignite.apache.org<mailto:user@ignite.apache.org> Subject: Re: java.lang.OutOfMemoryError: GC overhead limit exceeded Hi, Most likely heap size is too low. Try to increase Xmx up to 4Gb or higher or avoid G1GC usage on small heaps as it is very sensitive to free heap memory. Looks like you have Visor node (or may be web-console) in grid. Is OOM happened only when Visor attached to grid? On Fri, Mar 1, 2019 at 7:17 AM James Wang 王升平 (edvance CN) <james.w...@edvancesecurity.com<mailto:james.w...@edvancesecurity.com>> wrote: OS: 4C +8GB Data Region = 4GB start command: nohup $IGNITE_HOME/bin/ignite.sh<http://ignite.sh> -Xmx1024m -XX:+UseG1GC -XX:MaxDirectMemorySize=1G grtip-config.xml > ignite.log 2>&1 & How to adjust the the memory tunning. [21:29:22,777][SEVERE][mgmt-#33519%234-236-237-241%][GridJobWorker] Runtime error caught during grid runnable execution: GridJobWorker [createTime=1551360450243, startTime=1551360453071, finishTime=1551360483944, taskNode=TcpDiscoveryNode [id=a7839266-6396-4f7c-9ef7-c3a4b2355782, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.1.236], sockAddrs=[/192.168.1.236:47500<http://192.168.1.236:47500>, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500<http://127.0.0.1:47500>], discPort=47500, order=56, intOrder=31, lastExchangeTime=1551151457630, loc=false, ver=2.7.0#20181201-sha1:256ae401, isClient=false], internal=true, marsh=BinaryMarshaller [], ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.v.service.VisorServiceTask, dep=GridDeployment [ts=1551094729510, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6<mailto:clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6>, clsLdrId=3aef2742961-5a63d018-0eb4-493e-91a4-be6d41caff85, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.CachesRegistry, pendingUndeploy=false, undeployed=false, usage=0], taskClsName=o.a.i.i.v.service.VisorServiceTask, sesId=8a35b792961-a7839266-6396-4f7c-9ef7-c3a4b2355782, startTime=1551360450899, endTime=9223372036854775807, taskNodeId=a7839266-6396-4f7c-9ef7-c3a4b2355782, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6<mailto:clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6>, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=ContainsNodeIdsPredicate [], subjId=a7839266-6396-4f7c-9ef7-c3a4b2355782, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=327805292]], execName=null], jobId=9a35b792961-a7839266-6396-4f7c-9ef7-c3a4b2355782], jobCtx=GridJobContextImpl [jobId=9a35b792961-a7839266-6396-4f7c-9ef7-c3a4b2355782, timeoutObj=null, attrs={}], dep=GridDeployment [ts=1551094729510, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6<mailto:clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6>, clsLdrId=3aef2742961-5a63d018-0eb4-493e-91a4-be6d41caff85, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.CachesRegistry, pendingUndeploy=false, undeployed=false, usage=0], finishing=true, masterLeaveGuard=false, timedOut=false, sysCancelled=false, sysStopping=false, isStarted=true, job=VisorServiceJob [], held=0, partsReservation=null, reqTopVer=null, execName=null] java.lang.OutOfMemoryError: GC overhead limit exceeded [21:29:22,781][SEVERE][nio-acceptor-tcp-comm-#28%234-236-237-241%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError: GC overhead limit exceeded]] [21:29:22,814][SEVERE][grid-timeout-worker-#23%234-236-237-241%][GridTimeoutProcessor] Error when executing timeout callback: CancelableTask [id=94ef2742961-0742c515-5b96-4aad-b07a-9f1ec60af5f3, endTime=1551360514795, period=3000, cancel=false, task=MetricsUpdater [prevGcTime=38806, prevCpuTime=2195727, super=o.a.i.i.managers.discovery.GridDiscoveryManager$MetricsUpdater@73766331<mailto:super=o.a.i.i.managers.discovery.GridDiscoveryManager$MetricsUpdater@73766331>]] java.lang.OutOfMemoryError: GC overhead limit exceeded James Wang / edvance Mobile/WeChat: +86 135 1215 1134 This message contains information that is deemed confidential and privileged. Unless you are the addressee (or authorized to receive for the addressee), you may not use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message. -- Best regards, Andrey V. Mashenkov This message contains information that is deemed confidential and privileged. Unless you are the addressee (or authorized to receive for the addressee), you may not use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message. This message contains information that is deemed confidential and privileged. Unless you are the addressee (or authorized to receive for the addressee), you may not use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message.
