[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title
Lance Johnson commented on JENKINS-52150
Re: High CPU sys time consumption after upgrade to 2.107.3
We are seeing the same behavior with v2.180
# time strace -c -f -p 25129
% time seconds usecs/call callserrors syscall
-- --- --- - -
64.24 325.7725072399135794 17935 futex
21.78 110.426436 800192 13861 restart_syscall
9.17 46.523219 35898 1296 epoll_wait
3.97 20.126873 972 2070529 recvfrom
0.522.624930 2763195 accept
0.130.642494 22 29615 getrusage
0.050.272781 24 11607 write
[pid 25189] futex(0x7fa0814d3a54, FUTEX_WAIT_BITSET_PRIVATE, 1, {372183, 895437760}, ) = -1 ETIMEDOUT (Connection timed out)
[pid 25189] futex(0x7fa0814d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 25189] futex(0x7fa0814d3a54, FUTEX_WAIT_BITSET_PRIVATE, 1, {372183, 945599747}, ) = -1 ETIMEDOUT (Connection timed out)
[pid 25189] futex(0x7fa0814d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 25189] futex(0x7fa0814d3a54, FUTEX_WAIT_BITSET_PRIVATE, 1, {372183, 995774240}, ) = -1 ETIMEDOUT (Connection timed out)
[pid 25189] futex(0x7fa0814d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 25189] futex(0x7fa0814d3a54, FUTEX_WAIT_BITSET_PRIVATE, 1, {372184, 45936909}, ) = -1 ETIMEDOUT (Connection timed out)
[pid 25189] futex(0x7fa0814d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
# java -jar /usr/lib/jenkins/jenkins.war --version
2.180
Add Comment
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title ASHOK MOHANTY commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 We are using Jenkins v.2.121.3 (upgraded couple of months back from v.2.73.3 ). We have around ~1000 MBranch + ~200 pipeline + ~200FreeStyle projects. Post upgrade we are seeing CPU spike in the master server. We are not running any builds in master (primarily only MB indexings runs in master box). Master Server - Linux 7.x (100+ cpu). We have gitLab* plugins for "SCM - GitLab" integration. GitLab hosted in separate Linux server. Can you please provide some tips to debug the issue !! Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Marco Jacob commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 I'm not quite sure if this is a related problem: 14 of 14 Build jobs in our Jenkins 2.107.3 are blocked in Fingerprint.add during collecting data from slaves. Both, master and two slaves running on Solaris. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Carsten Pfeiffer commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 One option would be to call FileChannel.force(false) instead of FileChannel.force(true) for the fingerprints in https://github.com/jenkinsci/jenkins/blame/jenkins-2.121.2/core/src/main/java/hudson/util/FileChannelWriter.java#L89 in order not to force-write all metadata. The manpage of fdatasync on Linux says: "The aim of fdatasync() is to reduce disk activity for applications that do not require all metadata to be synchronized with the disk.". Not sure how much this would help, though. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Denis Koryavov updated an issue Jenkins / JENKINS-52150 High CPU sys time consumption after upgrade to 2.107.3 Change By: Denis Koryavov Priority: Blocker Critical Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Denis Koryavov commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 We disabled automatic fingerprint archiving for all maven jobs and it worked. I thinks the priority can be changed to Critical, because we have a simply workaround. We'll try to use -Dhudson.util.AtomicFileWriter.DISABLE_FORCED_FLUSH=true, but I think the problem should be fixed, because this behaviour of Jenkins is real issue for a large instances. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Carsten Pfeiffer commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 So this is rather unrelated to the maven plugin and more of a Jenkins core issue. The maven plugin only brings up the issue because it creates fingerprints by default. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Carsten Pfeiffer updated an issue Jenkins / JENKINS-52150 High CPU sys time consumption after upgrade to 2.107.3 Change By: Carsten Pfeiffer Component/s: core Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title Carsten Pfeiffer commented on JENKINS-52150 Re: High CPU sys time consumption after upgrade to 2.107.3 The real cause is the use of AtomicFileWriter in https://github.com/jenkinsci/jenkins/blob/jenkins-2.121.2/core/src/main/java/hudson/model/Fingerprint.java#L1256 The backing FileChannelWriter has an option to force flush (FileChannel.force(true)) and AtomicFileWriter does this by default. Luckily there is a system property that you can set in order to revert to the previous less safe, but better performing behavior: -Dhudson.util.AtomicFileWriter.DISABLE_FORCED_FLUSH=true (See https://github.com/jenkinsci/jenkins/blob/jenkins-2.121.2/core/src/main/java/hudson/util/AtomicFileWriter.java#L57) Setting this brings back the previous performance. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52150) High CPU sys time consumption after upgrade to 2.107.3
Title: Message Title
Carsten Pfeiffer commented on JENKINS-52150
Re: High CPU sys time consumption after upgrade to 2.107.3
We're also suffering from this, since we updated one jenkins from 2.89.4 to 2.121.2. It looks like the I/O pattern for writing a single fingerprint could be improved quite a bit, but this is probably not the only cause for the performance degradation.
[pid 8789] stat("/var/jenkins_home/fingerprints/bf/e4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 8789] access("/var/jenkins_home/fingerprints/bf/e4", F_OK) = 0
[pid 8789] stat("/var/jenkins_home/fingerprints/bf/e4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 8789] lstat("/var/jenkins_home/fingerprints/bf/e4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 8789] mkdir("/var/jenkins_home/fingerprints/bf/e4", 0777) = -1 EEXIST (File exists)
[pid 8789] lstat("/var/jenkins_home/fingerprints/bf/e4", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 8789] stat("/var/jenkins_home/fingerprints/bf/e4/atomic8576544207056275611tmp", 0x7fe62a3e3f00) = -1 ENOENT (No such file or directory)
[pid 8789] open("/var/jenkins_home/fingerprints/bf/e4/atomic8576544207056275611tmp", O_RDWR|O_CREAT|O_EXCL, 0666) = 5
[pid 8789] open("/var/jenkins_home/fingerprints/bf/e4/atomic8576544207056275611tmp", O_WRONLY) = 5
[pid 8789] rename("/var/jenkins_home/fingerprints/bf/e4/atomic8576544207056275611tmp", "/var/jenkins_home/fingerprints/bf/e4/c7f5648f164d878a47abeebea260.xml") = 0
[pid 8789] lstat("/var/jenkins_home/fingerprints/bf/e4/atomic8576544207056275611tmp", 0x7fe62a3e4020) = -1 ENOENT (No such file or directory)
The problem really only shows up when you have lots of fingerprints. In our case, this is around 330k, stored in 6,5k directories in the fingerprints directory.
Add Comment
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)
