Greetings Jenkins community!
My team uses Jenkins to run long validation cycles. We use Kubernetes
plugin (we run our own bare-metal cluster) and many quick single test case
builds.
Recently our number of test cases have grown drastically and we've noticed
that scheduling of new builds is very slow despite we have enough compute
resources in cluster.
To summarize the problem:
- Jenkins has over 1000 builds in queue
- They weren't started at the same time - number of builds grows from
zero to 1000 within 14 hours. We've added this delay to, well, prevent
Jenkins from hanging :)
- Those builds schedule very slowly (so queue size decreases not as fast
as we desire)
- There are enough resources in Kubernetes cluster - therefore slowness
isn't caused by long waiting time. Other Jenkins instances we have can
schedule builds quickly into the same Kubernetes (so Kubernetes is fine)
- UI is responsive, but actions like configSubmit on jobs or nodes are
slow
This server runs only validation cycles, no builds are executed directly on
master. We use Jenkins from official Docker image 2.222.3-jdk11 with
Kubernetes plugin version 1.24.
Hardware-wise we have server with two Intel(R) Xeon(R) CPU E5-2670 2.60GHz
(8 cores each with hyper-threading enabled - 32 vCPU in total) and 128GB of
RAM.
Jenkins container uses --network=host option, inbound (JNLP) agents
provisioned by Kubernetes plugin use random port.
Settings of Kubernetes cloud we use:
[image: Annotation 2020-06-05 210612.png]
We've increased Max connections to Kubernetes API when the problem appeared
(we thought we may have been bounded by number of connections).
I've attached plugins.txt file also and here are JVM options we use:
JAVA_OPTS=-Xms64G -Xmx64G
-XX:+HeapDumpOnOutOfMemoryError
-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=84000
-Dpermissive-script-security.enabled=no_security # don't do this in
production
-Dorg.apache.commons.jelly.tags.fmt.timeZone=Europe/Moscow
-Dhudson.model.DirectoryBrowserSupport.CSP=
-Dfile.encoding=UTF-8
-Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider
-Xlog:gc*,gc+ref=debug,gc+heap=debug,gc+age=trace:file=/var/jenkins_home/gc-%p-%t.log:tags,uptime,time,level:filecount=10,filesize=500m
-XX:-UseCompressedOops
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9090
-Dcom.sun.management.jmxremote.rmi.port=9090
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
As you can see we've tried to profile Jenkins with VisualVM. I have thread
dump and snapshot, but I don't want to share them publicly. I can send it
to you personally, if you want to take a look.
CPU was completely fine, around 20% and GC takes as little as 0.1%
[image: peak_cpu.png]
At this moment my team and me are out of guesses, so I'd appreciate any
advice you have
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/7ea695e9-889f-486a-91b9-974a68e0f3d9o%40googlegroups.com.
ace-editor:1.1
ansicolor:0.6.2
ant:1.11
antisamy-markup-formatter:1.8
apache-httpcomponents-client-4-api:4.5.10-2.0
audit-trail:3.3
authentication-tokens:1.3
authorize-project:1.3.0
azure-commons:1.0.4
azure-container-agents:1.1.1
azure-credentials:2.0.1
badge:1.8
basic-branch-build-strategies:1.3.2
blueocean-autofavorite:1.2.4
blueocean-commons:1.22.0
blueocean-config:1.22.0
blueocean-core-js:1.22.0
blueocean-dashboard:1.22.0
blueocean-display-url:2.3.1
blueocean-events:1.22.0
blueocean-git-pipeline:1.22.0
blueocean-github-pipeline:1.22.0
blueocean-i18n:1.22.0
blueocean-jwt:1.22.0
blueocean-personalization:1.22.0
blueocean-pipeline-api-impl:1.22.0
blueocean-pipeline-editor:1.22.0
blueocean-pipeline-scm-api:1.22.0
blueocean-rest-impl:1.22.0
blueocean-rest:1.22.0
blueocean-web:1.22.0
bootstrap4-api:4.4.1-15
bouncycastle-api:2.18
branch-api:2.5.5
build-user-vars-plugin:1.5
build-with-parameters:1.4
cloudbees-folder:6.11.1
command-launcher:1.4
conditional-buildstep:1.3.6
configuration-as-code:1.35
copyartifact:1.43
credentials-binding:1.21
credentials:2.3.1
dashboard-view:2.12
data-tables-api:1.10.20-15
discard-old-build:1.05
display-url-api:2.3.2
docker-commons:1.16
docker-java-api:3.0.14
docker-workflow:1.21
dtkit-api:2.1.2
durable-task:1.33
email-ext:2.68
favorite:2.3.2
git-client:3.1.1
git-server:1.9
git:4.2.2
github-autostatus:3.6.1
gitlab-api:1.0.6
gitlab-branch-source:1.5.1
google-oauth-plugin:1.0.0
htmlpublisher:1.21
http_request:1.8.24
influxdb:2.1
jackson2-api:2.11.0
javadoc:1.5
jenkins-design-language:1.22.0
job-dsl:1.77
jsch:0.1.55.2
junit:1.28
kubernetes-client-api:4.6.4-2
kubernetes-credentials:0.6.2
kubernetes:1.24.0
ldap:1.21
locale:1.4
lockable-resources:2.7
mailer:1.30
matrix-auth:2.5
matrix-project:1.14
maven-plugin:3.4
modernstatus:1.2
momentjs:1.1.1
oauth-credentials:0.4
pam-auth:1.6
parameterized-trigger:2.36
permissive-script-security:0.6
pipeline-build-step:2.11
pipeline-graph-analysis:1.10
pipeline-input-step:2.11
pipeline-milestone-step:1.3.1
pipeline-model-api:1.5.1
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:1.5.1
pipeline-model-extensions:1.5.1
pipeline-rest-api:2.13
pipeline-stage-step:2.3
pipeline-stage-tags-metadata:1.5.1
pipeline-stage-view:2.13
pipeline-utility-steps:2.5.0
plain-credentials:1.7
plugin-util-api:1.2.0
purge-build-queue-plugin:1.0
rebuild:1.31
resource-disposer:0.14
role-strategy:2.16
run-condition:1.2
run-selector:1.1.1
saferestart:0.3
scm-api:2.6.3
script-security:1.71
show-build-parameters:1.0
sse-gateway:1.21
ssh-agent:1.19
ssh-credentials:1.18.1
ssh-slaves:1.31.1
ssh:2.6.1
structs:1.20
timestamper:1.11.2
token-macro:2.11
trilead-api:1.0.5
variant:1.3
windows-azure-storage:1.1.5
windows-slaves:1.6
workflow-aggregator:2.6
workflow-api:2.39
workflow-basic-steps:2.19
workflow-cps-global-lib:2.15
workflow-cps:2.80
workflow-durable-task-step:2.35
workflow-job:2.36
workflow-multibranch:2.21
workflow-remote-loader:1.5
workflow-scm-step:2.10
workflow-step-api:2.22
workflow-support:3.4
ws-cleanup:0.38
