I will try turning on GC logging. torsdag 4. juli 2019 23.04.55 UTC+2 skrev Baptiste Mathus følgende: > > Did you enable GC logging to have a better understanding of the profile of > your memory consumption? If not, I would recommend you do it first and > analyze them. > https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and > much more) quite well. > > Then, once you understand better when it crashes, possibly you'll want to > analyze a heap dump to see what is causing the problem. > > Cheers > > > Le mar. 2 juil. 2019 à 15:30, Sverre Moe <[email protected] <javascript:>> > a écrit : > >> Today it has been chaotic. >> Several build agents disconnected >> >> Unexpected termination of the channel >> >> Many builds failed because of Memory error. >> >> I have tried restarting Jenkins several times today. >> >> Anyone have any suggestions? >> >> tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende: >>> >>> We have assigned 8GB of memory to our Jenkins instance. >>> JAVA_OPTIONS=-Xmx8g >>> >>> Still we experience memory issues after a while running. >>> java.lang.OutOfMemoryError: unable to create new native thread >>> >>> We have: >>> aprox 40 connected build agents >>> aprox 400 pipeline jobs >>> >>> We have a test Jenkins instance running with the same jobs, this one >>> connects to the same build agents (though on a different home directory). >>> >>> Lately we have been getting disconnected build agents, that we cannot >>> get up again without restarting Jenkins. >>> >>> Can we assign more memory to a build agent? Would it have any affect on >>> this issue? >>> >>> This we got from one of our latest Pipeline builds that failed on a >>> sh("find **** -exec ***") step. It failed on that build agent that is now >>> disconnected. >>> >>> >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:714) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) >>> at >>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368) >>> at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288) >>> at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:257) >>> at java.lang.ProcessImpl.start(ProcessImpl.java:134) >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) >>> at hudson.Proc$LocalProc.<init>(Proc.java:249) >>> Also: java.io.IOException: error=11, Resource temporarily unavailable >>> >>> >>> >>> >>> SEVERE: Unexpected error when retrieving changeset >>> hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M >>> "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> % >>> ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 >>> b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/ >>> project_user_work >>> at >>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012) >>> >>> >>> at >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) >>> >>> >>> at >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) >>> >>> >>> at hudson.remoting.UserRequest.perform(UserRequest.java:212) >>> at hudson.remoting.UserRequest.perform(UserRequest.java:54) >>> at hudson.remoting.Request$2.run(Request.java:369) >>> at >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) >>> >>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> >>> >>> at java.lang.Thread.run(Thread.java:748) >>> Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote >>> call to master-sles12.3-x86_64_3 >>> at >>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) >>> at >>> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) >>> >>> >>> at hudson.remoting.Channel.call(Channel.java:955) >>> at >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) >>> >>> >>> at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown >>> Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) >>> >>> >>> at com.sun.proxy.$Proxy104.execute(Unknown Source) >>> at >>> io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159) >>> >>> >>> at >>> io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84) >>> >>> >>> at >>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140) >>> at >>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) >>> >>> >>> at >>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) >>> >>> >>> at >>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS >>> tepExecution.java:47) >>> >>> Jul 01, 2019 11:51:12 AM >>> hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler >>> >>> uncaughtException >>> SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an >>> uncaught exception, this may leave your Jenkins in a bad way and >>> is usually indicative of a bug in the code. >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:714) >>> at java.util.Timer.<init>(Timer.java:160) >>> at java.util.Timer.<init>(Timer.java:132) >>> at >>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296) >>> >>> >>> at >>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437) >>> >>> >>> at >>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299) >>> >>> >>> at java.util.TimerThread.mainLoop(Timer.java:555) >>> at java.util.TimerThread.run(Timer.java:505) >>> >>> INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3. >>> java.util.concurrent.TimeoutException: Ping started at 1561982408948 >>> hasn't completed by 1561982648948 >>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>> at hudson.remoting.PingThread.run(PingThread.java:90) >>> >>> Jul 01, 2019 2:04:11 PM >>> hudson.remoting.SynchronousCommandTransport$ReaderThread run >>> INFO: I/O error in channel master-sles12.3-x86_64_3 >>> java.io.IOException: Unexpected termination of the channel >>> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space >>> >>> Jul 01, 2019 2:04:11 PM >>> hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed >>> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space >>> >>> >>> >>> The latest problem we got. It did not take down the build node. On all >>> occasions of this problem it happens when the Pipeline is doing some IO on >>> the Jenkins master. Here we manually restart the build again, and it builds >>> fine. >>> >>> Running on Jenkins <https://build-ci.spacetec.no:8443/computer/(master)/> >>> in /var/lib/jenkins/workspace/project_master[Pipeline] { >>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>> parallel >>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>> { (Branch: Setup) >>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>> End of Pipelinejava.lang.OutOfMemoryError: unable to create new native >>> thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:714) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) >>> at >>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366) >>> at >>> com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90) >>> at >>> com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81) >>> at com.google.common.eventbus.EventBus.post(EventBus.java:264) >>> at >>> org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70) >>> at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141) >>> at >>> io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196) >>> at >>> io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85) >>> at >>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) >>> at >>> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) >>> at >>> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) >>> at >>> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) >>> at >>> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) >>> at >>> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> Finished: FAILURE >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >
-- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f9cdac34-e674-4749-a425-495ec20932a1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
