Have you tried this?
-Dhudson.slaves.NodeProvisioner.initialDelay=0 \
-Dhudson.slaves.NodeProvisioner.MARGIN=50 \
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 \
In the jenkins.log, is there anything interesting when you expect an agent
get provisioned but it didn't?
IMHO I would think the "concurrent limit" is way too high.
How about start with 100 then tune it high?
On Fri, Jun 5, 2020 at 1:24 PM Andrey Babushkin <[email protected]> wrote:
> Greetings Jenkins community!
>
> My team uses Jenkins to run long validation cycles. We use Kubernetes
> plugin (we run our own bare-metal cluster) and many quick single test case
> builds.
> Recently our number of test cases have grown drastically and we've noticed
> that scheduling of new builds is very slow despite we have enough compute
> resources in cluster.
>
> To summarize the problem:
>
> - Jenkins has over 1000 builds in queue
> - They weren't started at the same time - number of builds grows from
> zero to 1000 within 14 hours. We've added this delay to, well, prevent
> Jenkins from hanging :)
> - Those builds schedule very slowly (so queue size decreases not as
> fast as we desire)
> - There are enough resources in Kubernetes cluster - therefore
> slowness isn't caused by long waiting time. Other Jenkins instances we have
> can schedule builds quickly into the same Kubernetes (so Kubernetes is
> fine)
> - UI is responsive, but actions like configSubmit on jobs or nodes are
> slow
>
>
> This server runs only validation cycles, no builds are executed directly
> on master. We use Jenkins from official Docker image 2.222.3-jdk11 with
> Kubernetes plugin version 1.24.
> Hardware-wise we have server with two Intel(R) Xeon(R) CPU E5-2670 2.60GHz
> (8 cores each with hyper-threading enabled - 32 vCPU in total) and 128GB of
> RAM.
> Jenkins container uses --network=host option, inbound (JNLP) agents
> provisioned by Kubernetes plugin use random port.
> Settings of Kubernetes cloud we use:
>
> [image: Annotation 2020-06-05 210612.png]
> We've increased Max connections to Kubernetes API when the problem
> appeared (we thought we may have been bounded by number of connections).
> I've attached plugins.txt file also and here are JVM options we use:
> JAVA_OPTS=-Xms64G -Xmx64G
> -XX:+HeapDumpOnOutOfMemoryError
>
> -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=84000
> -Dpermissive-script-security.enabled=no_security # don't do this
> in production
> -Dorg.apache.commons.jelly.tags.fmt.timeZone=Europe/Moscow
> -Dhudson.model.DirectoryBrowserSupport.CSP=
> -Dfile.encoding=UTF-8
>
> -Djenkins.displayurl.provider=org.jenkinsci.plugins.displayurlapi.ClassicDisplayURLProvider
>
> -Xlog:gc*,gc+ref=debug,gc+heap=debug,gc+age=trace:file=/var/jenkins_home/gc-%p-%t.log:tags,uptime,time,level:filecount=10,filesize=500m
> -XX:-UseCompressedOops
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=9090
> -Dcom.sun.management.jmxremote.rmi.port=9090
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
>
>
> As you can see we've tried to profile Jenkins with VisualVM. I have thread
> dump and snapshot, but I don't want to share them publicly. I can send it
> to you personally, if you want to take a look.
> CPU was completely fine, around 20% and GC takes as little as 0.1%
>
> [image: peak_cpu.png]
>
>
> At this moment my team and me are out of guesses, so I'd appreciate any
> advice you have
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/7ea695e9-889f-486a-91b9-974a68e0f3d9o%40googlegroups.com
> <https://groups.google.com/d/msgid/jenkinsci-users/7ea695e9-889f-486a-91b9-974a68e0f3d9o%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Thanks,
Baibai
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAMRm5jbShWub2JUJbAT8Fc4JYxGUiY2ySQVCiYuLeSS8%2BWMZgA%40mail.gmail.com.
