Hi all,
Our GoCD server is acting slow. We have >1000 pipelines that trigger
automatically on updates to various branches of a Git repository. For each
branch that needs to be picked up by GoCD, we have a distinct git material
with shallow clone set to true. The time to trigger from Git updates is
quite slow. The UI is also a bit sluggish in general.
Environment info:
Linux
GoCD server 21.1.0
PostgreSQL 12.8 database
The server was recently recreated updating the OS from Ubuntu 16.04 to
Ubuntu 20.04. The most important pieces of configuration were restored from
the previous server. However, it's possible some configuration was lost, or
some package version was changed, or something like that. I'd appreciate
any pointers.
While trying to debug this, I noticed a few things:
CPU utilization on the machine is hovering around 60%, when there are no
jobs running.
The top consumers of CPU on the machine are git commands run by the GoCD
server. At any given time there are about 8 instances of git processes like
"git branch -r --contains <commit-sha>". Each of these git commands can be
using a whole CPU.
Looking at things related to git, I noticed:
The GoCD server logs in /var/log/go-server/go-server.log have a lot these
messages (referencing the different deployment branches):
2023-01-11 06:44:04,168 WARN [ThreadPoolTaskScheduler-7]
MaterialUpdateService:204 - [Material Update] Skipping update of material
GitMaterial{url=g...@github.com:repo/app.git, branch='abranch',
shallowClone=true, submoduleFolder='null'} which has been in-progress since
Wed Jan 11 06:43:04 UTC 2023
Using jstack to get thread dumps from the server, this seems to be the java
stack trace where those git processes are launched:
"130@MessageListener for MaterialUpdateListener" #130 daemon prio=5
os_prio=0 cpu=14618.25ms elapsed=100474.11s tid=0x00007fa2ded9dca0
nid=0xea6f in Object.wait() [0x00007fa20c5ad000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@15.0.1/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait(java.base@15.0.1/Unknown Source)
at java.lang.ProcessImpl.waitFor(java.base@15.0.1/Unknown Source)
- locked <0x00000005370e58e0> (a java.lang.ProcessImpl)
at
com.thoughtworks.go.util.ProcessWrapper.waitForExit(ProcessWrapper.java:54)
at
com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:354)
at
com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:378)
at
com.thoughtworks.go.domain.materials.SCMCommand.runOrBomb(SCMCommand.java:38)
at
com.thoughtworks.go.domain.materials.git.GitCommand.containsRevisionInBranch(GitCommand.java:364)
at
com.thoughtworks.go.config.materials.git.GitMaterial.modificationsSince(GitMaterial.java:132)
at
com.thoughtworks.go.server.service.materials.GitPoller.modificationsSince(GitPoller.java:35)
at
com.thoughtworks.go.server.service.materials.GitPoller.modificationsSince(GitPoller.java:26)
at
com.thoughtworks.go.server.service.MaterialService.modificationsSince(MaterialService.java:134)
at
com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:56)
at
com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.insertLatestOrNewModifications(MaterialDatabaseUpdater.java:157)
at
com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterialWithNewRevisions(MaterialDatabaseUpdater.java:149)
at
com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$2.doInTransaction(MaterialDatabaseUpdater.java:108)
at
com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23)
at
com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$2(TransactionTemplate.java:43)
at
com.thoughtworks.go.server.transaction.TransactionTemplate$$Lambda$1842/0x00000008045df9c8.doInTransaction(Unknown
Source)
at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at
com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40)
at
com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:105)
- locked <0x0000000537f34488> (a java.lang.String)
at
com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64)
at
com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32)
at
com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
at
com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
at java.lang.Thread.run(java.base@15.0.1/Unknown Source)
Any help is appreciated, thanks!
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/go-cd/3c3fe87e-5639-4316-9dfa-bc3dcf68d901n%40googlegroups.com.
