Today it has been chaotic.
Several build agents disconnected
Unexpected termination of the channel
Many builds failed because of Memory error.
I have tried restarting Jenkins several times today.
Anyone have any suggestions?
tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
>
> We have assigned 8GB of memory to our Jenkins instance.
> JAVA_OPTIONS=-Xmx8g
>
> Still we experience memory issues after a while running.
> java.lang.OutOfMemoryError: unable to create new native thread
>
> We have:
> aprox 40 connected build agents
> aprox 400 pipeline jobs
>
> We have a test Jenkins instance running with the same jobs, this one
> connects to the same build agents (though on a different home directory).
>
> Lately we have been getting disconnected build agents, that we cannot get
> up again without restarting Jenkins.
>
> Can we assign more memory to a build agent? Would it have any affect on
> this issue?
>
> This we got from one of our latest Pipeline builds that failed on a
> sh("find **** -exec ***") step. It failed on that build agent that is now
> disconnected.
>
>
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
> at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
> at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> at hudson.Proc$LocalProc.<init>(Proc.java:249)
> Also: java.io.IOException: error=11, Resource temporarily unavailable
>
>
>
>
> SEVERE: Unexpected error when retrieving changeset
> hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M
> "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
> ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1
> b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
> project_user_work
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
>
>
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
>
>
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
>
>
> at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> at hudson.remoting.Request$2.run(Request.java:369)
> at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call
> to master-sles12.3-x86_64_3
> at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
>
> at hudson.remoting.Channel.call(Channel.java:955)
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
>
>
> at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
>
>
> at com.sun.proxy.$Proxy104.execute(Unknown Source)
> at
> io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
>
>
> at
> io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
>
>
> at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
> at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
>
>
> at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
>
>
> at
> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
> tepExecution.java:47)
>
> Jul 01, 2019 11:51:12 AM
> hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler
>
> uncaughtException
> SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught
> exception, this may leave your Jenkins in a bad way and
> is usually indicative of a bug in the code.
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at java.util.Timer.<init>(Timer.java:160)
> at java.util.Timer.<init>(Timer.java:132)
> at
> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
>
>
> at
> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
>
>
> at
> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
>
>
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
> java.util.concurrent.TimeoutException: Ping started at 1561982408948
> hasn't completed by 1561982648948
> at hudson.remoting.PingThread.ping(PingThread.java:134)
> at hudson.remoting.PingThread.run(PingThread.java:90)
>
> Jul 01, 2019 2:04:11 PM
> hudson.remoting.SynchronousCommandTransport$ReaderThread run
> INFO: I/O error in channel master-sles12.3-x86_64_3
> java.io.IOException: Unexpected termination of the channel
> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space
>
> Jul 01, 2019 2:04:11 PM
> hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space
>
>
>
> The latest problem we got. It did not take down the build node. On all
> occasions of this problem it happens when the Pipeline is doing some IO on
> the Jenkins master. Here we manually restart the build again, and it builds
> fine.
>
> Running on Jenkins <https://build-ci.spacetec.no:8443/computer/(master)/> in
> /var/lib/jenkins/workspace/project_master[Pipeline] {
> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline]
> parallel
> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline]
> { (Branch: Setup)
> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline]
> End of Pipelinejava.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
> at
> com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
> at
> com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
> at com.google.common.eventbus.EventBus.post(EventBus.java:264)
> at
> org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
> at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
> at
> io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
> at
> io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
> at
> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
> at
> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
> at
> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
> at
> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
> at
> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Finished: FAILURE
>
>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
