[JIRA] (JENKINS-58684) Linux max task resource exhaustion after excessive timer thread creation

[2blu...@gmail.com (JIRA)]

Title: Message Title

Kyle Manna created an issue   Jenkins / JENKINS-58684 Linux max task resource exhaustion after excessive timer thread creation Issue Type: Bug Assignee: vjuranek Attachments: d119bce8-b89a-4ece-9327-a7fbfc791ff9.png, Screenshot from 2019-07-22 15-35-03.png Components: groovy-plugin Created: 2019-07-26 17:59 Environment: Jenkins 2.176.2 using Ubuntu 18.04 package file from pkg.jenkins.io/debian-stable AMD Threadripper 32-core/64-core machine with 128GB of RAM Single master node with 20 executors Labels: groovy threads Priority: Major Reporter: Kyle Manna Jenkins is exhausting resources on my machine seemingly to create 10k + timer threads. This thread creation then hits the systemd imposed 15% of task limit set by the kernel (/proc/sys/kernel/pid_max = 131072) which is 19660 by default on my system. This particular java heap dump shows that 4675 of the total 4877 match the _expression_ Timer-.* and have the Context Class Loader -> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader. Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with Ubuntu 18.04 as of 2019.07.23.   To dig deeper, after the resource limit is hit, threads fail to create and Jenkins becomes unstable until restarted, logs contain many instances of the following: SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed java.lang.OutOfMemoryError: unable to create new native thread   NetData on the system shows that if it doesn't hit the thread limit it will spike and often recover correctly suggesting the threads are cleaned up at some point if it doesn't derail the process first. I've increased the systemd limit on the system and the problem hasn't derailed the jenkins process in a few days by adding the following to the systemd service file: TasksMax=32768 I have been capturing heaps with some automation tools and catch several heaps exceeding 20k threads in the last few days signifying that the process is still occurring and likely affecting other users, they probably aren't noticing though.   Not sure if this is related to a plugin (listed below with version) or Jenkins core code?   [ { "plugin": "Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api)", "version": "4.5.5-3.0" }, { "plugin": "Authentication Tokens API Plugin (authentication-tokens)", "version": "1.3" }, { "plugin": "Authorize Project (authorize-project)", "version": "1.3.0" }, { "plugin": "Autofavorite for Blue Ocean (blueocean-autofavorite)", "version": "1.2.4" }, { "plugin": "Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source)", "version": "2.4.5" }, { "plugin": "Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline)", "version": "1.17.0" }, { "plugin": "Bitbucket Plugin (bitbucket)", "version": "1.1.10" }, { "plugin": "Blue Ocean (blueocean)", "version": "1.17.0" }, { "plugin": "Blue Ocean Core JS (blueocean-core-js)", "version": "1.17.0" }, { "plugin": "Blue Ocean Executor Info (blueocean-executor-info)", "version": "1.17.0" }, { "plugin": "Blue Ocean Pipeline Editor (blueocean-pipeline-editor)", "version": "1.17.0" }, { "plugin": "Branch API Plugin (branch-api)", "version": "2.5.3" }, { "plugin": "Build Timeout (build-timeout)", "version": "1.19" }, { "plugin": "Command Agent Launcher Plugin (command-launcher)", "version": "1.3" }, { "plugin": "Common API for Blue Ocean (blueocean-commons)", "version": "1.17.0" }, { "plugin": "Conditional BuildStep (conditional-buildstep)", "version": "1.3.6" }, { "plugin": "Config API for Blue Ocean (blueocean-config)", "version": "1.17.0" }, { "plugin": "Copy Artifact Plugin (copyartifact)", "version": "1.42.1" }, { "plugin": "Credentials Binding Plugin (credentials-binding)", "version": "1.19" }, { "plugin": "Credentials Plugin (credentials)", "version": "2.2.0" }, { "plugin": "Dashboard for Blue Ocean (blueocean-dashboard)", "version": "1.17.0" }, { "plugin": "Design Language (jenkins-design-language)", "version": "1.17.0" }, { "plugin": "Display URL API (display-url-api)", "version": "2.3.1" }, { "plugin": "Display URL for Blue Ocean (blueocean-display-url)", "version": "2.3.0" }, { "plugin": "Docker Commons Plugin (docker-commons)", "version": "1.15" }, { "plugin": "Docker Pipeline (docker-workflow)", "version": "1.18" }, { "plugin": "Durable Task Plugin (durable-task)", "version": "1.30" }, { "plugin": "Email Extension Plugin (email-ext)", "version": "2.66" }, { "plugin": "Events API for Blue Ocean (blueocean-events)", "version": "1.17.0" }, { "plugin": "External Monitor Job Type Plugin (external-monitor-job)", "version": "1.7" }, { "plugin": "Favorite (favorite)", "version": "2.3.2" }, { "plugin": "Folders Plugin (cloudbees-folder)", "version": "6.9" }, { "plugin": "GIT server Plugin (git-server)", "version": "1.7" }, { "plugin": "Git Pipeline for Blue Ocean (blueocean-git-pipeline)", "version": "1.17.0" }, { "plugin": "Git client plugin (git-client)", "version": "3.0.0-rc" }, { "plugin": "Git plugin (git)", "version": "4.0.0-rc" }, { "plugin": "GitHub API Plugin (github-api)", "version": "1.95" }, { "plugin": "GitHub Branch Source Plugin (github-branch-source)", "version": "2.5.4" }, { "plugin": "GitHub Pipeline for Blue Ocean (blueocean-github-pipeline)", "version": "1.17.0" }, { "plugin": "GitHub plugin (github)", "version": "1.29.4" }, { "plugin": "Google Login Plugin (google-login)", "version": "1.6" }, { "plugin": "Gradle Plugin (gradle)", "version": "1.33" }, { "plugin": "HTML Publisher plugin (htmlpublisher)", "version": "1.18" }, { "plugin": "HTTP Request Plugin (http_request)", "version": "1.8.23" }, { "plugin": "Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api)", "version": "2.1.7-1.0" }, { "plugin": "Icon Shim Plugin (icon-shim)", "version": "2.0.3" }, { "plugin": "JIRA Integration for Blue Ocean (blueocean-jira)", "version": "1.17.0" }, { "plugin": "JIRA plugin (jira)", "version": "3.0.8" }, { "plugin": "JSch dependency plugin (jsch)", "version": "0.1.55" }, { "plugin": "JUnit Plugin (junit)", "version": "1.28" }, { "plugin": "JWT for Blue Ocean (blueocean-jwt)", "version": "1.17.0" }, { "plugin": "Jackson 2 API Plugin (jackson2-api)", "version": "2.9.9.1" }, { "plugin": "_javascript_ GUI Lib: ACE Editor bundle plugin (ace-editor)", "version": "1.1" }, { "plugin": "_javascript_ GUI Lib: Handlebars bundle plugin (handlebars)", "version": "1.1.1" }, { "plugin": "_javascript_ GUI Lib: Moment.js bundle plugin (momentjs)", "version": "1.1.1" }, { "plugin": "_javascript_ GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached)", "version": "1.2.1" }, { "plugin": "Javadoc Plugin (javadoc)", "version": "1.5" }, { "plugin": "LDAP Plugin (ldap)", "version": "1.20" }, { "plugin": "Lockable Resources plugin (lockable-resources)", "version": "2.5" }, { "plugin": "Mailer Plugin (mailer)", "version": "1.23" }, { "plugin": "MapDB API Plugin (mapdb-api)", "version": "1.0.9.0" }, { "plugin": "Matrix Authorization Strategy Plugin (matrix-auth)", "version": "2.4.2" }, { "plugin": "Matrix Project Plugin (matrix-project)", "version": "1.14" }, { "plugin": "Maven Integration plugin (maven-plugin)", "version": "3.3" }, { "plugin": "Mercurial plugin (mercurial)", "version": "2.7" }, { "plugin": "Metrics Plugin (metrics)", "version": "4.0.2.5" }, { "plugin": "OWASP Markup Formatter Plugin (antisamy-markup-formatter)", "version": "1.5" }, { "plugin": "Oracle Java SE Development Kit Installer Plugin (jdk-tool)", "version": "1.3" }, { "plugin": "PAM Authentication plugin (pam-auth)", "version": "1.5.1" }, { "plugin": "Personalization for Blue Ocean (blueocean-personalization)", "version": "1.17.0" }, { "plugin": "Pipeline (workflow-aggregator)", "version": "2.6" }, { "plugin": "Pipeline Graph Analysis Plugin (pipeline-graph-analysis)", "version": "1.10" }, { "plugin": "Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api)", "version": "1.17.0" }, { "plugin": "Pipeline Utility Steps (pipeline-utility-steps)", "version": "2.3.0" }, { "plugin": "Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl)", "version": "1.17.0" }, { "plugin": "Pipeline: API (workflow-api)", "version": "2.35" }, { "plugin": "Pipeline: Basic Steps (workflow-basic-steps)", "version": "2.18" }, { "plugin": "Pipeline: Build Step (pipeline-build-step)", "version": "2.9" }, { "plugin": "Pipeline: Declarative (pipeline-model-definition)", "version": "1.3.9" }, { "plugin": "Pipeline: Declarative Agent API (pipeline-model-declarative-agent)", "version": "1.1.1" }, { "plugin": "Pipeline: Declarative Extension Points API (pipeline-model-extensions)", "version": "1.3.9" }, { "plugin": "Pipeline: GitHub Groovy Libraries (pipeline-github-lib)", "version": "1.0" }, { "plugin": "Pipeline: Groovy (workflow-cps)", "version": "2.72" }, { "plugin": "Pipeline: Input Step (pipeline-input-step)", "version": "2.10" }, { "plugin": "Pipeline: Job (workflow-job)", "version": "2.33" }, { "plugin": "Pipeline: Milestone Step (pipeline-milestone-step)", "version": "1.3.1" }, { "plugin": "Pipeline: Model API (pipeline-model-api)", "version": "1.3.9" }, { "plugin": "Pipeline: Multibranch (workflow-multibranch)", "version": "2.21" }, { "plugin": "Pipeline: Nodes and Processes (workflow-durable-task-step)", "version": "2.32" }, { "plugin": "Pipeline: REST API Plugin (pipeline-rest-api)", "version": "2.11" }, { "plugin": "Pipeline: SCM Step (workflow-scm-step)", "version": "2.9" }, { "plugin": "Pipeline: Shared Groovy Libraries (workflow-cps-global-lib)", "version": "2.14" }, { "plugin": "Pipeline: Stage Step (pipeline-stage-step)", "version": "2.3" }, { "plugin": "Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata)", "version": "1.3.9" }, { "plugin": "Pipeline: Stage View Plugin (pipeline-stage-view)", "version": "2.11" }, { "plugin": "Pipeline: Step API (workflow-step-api)", "version": "2.20" }, { "plugin": "Pipeline: Supporting APIs (workflow-support)", "version": "3.3" }, { "plugin": "Plain Credentials Plugin (plain-credentials)", "version": "1.5" }, { "plugin": "Pub-Sub \"light\" Bus (pubsub-light)", "version": "1.12" }, { "plugin": "REST API for Blue Ocean (blueocean-rest)", "version": "1.17.0" }, { "plugin": "REST Implementation for Blue Ocean (blueocean-rest-impl)", "version": "1.17.0" }, { "plugin": "Resource Disposer Plugin (resource-disposer)", "version": "0.13" }, { "plugin": "Run Condition Plugin (run-condition)", "version": "1.2" }, { "plugin": "SCM API Plugin (scm-api)", "version": "2.6.3" }, { "plugin": "SSH Agent Plugin (ssh-agent)", "version": "1.17" }, { "plugin": "SSH Credentials Plugin (ssh-credentials)", "version": "1.17.1" }, { "plugin": "SSH Slaves plugin (ssh-slaves)", "version": "1.30.1" }, { "plugin": "Script Security Plugin (script-security)", "version": "1.61" }, { "plugin": "Server Sent Events (SSE) Gateway Plugin (sse-gateway)", "version": "1.18" }, { "plugin": "Skip Notifications Trait plugin (skip-notifications-trait)", "version": "1.0.3" }, { "plugin": "Structs Plugin (structs)", "version": "1.19" }, { "plugin": "Subversion Plug-in (subversion)", "version": "2.12.2" }, { "plugin": "Timestamper (timestamper)", "version": "1.10" }, { "plugin": "Token Macro Plugin (token-macro)", "version": "2.8" }, { "plugin": "Trilead API Plugin (trilead-api)", "version": "1.0.3" }, { "plugin": "Variant Plugin (variant)", "version": "1.2" }, { "plugin": "WMI Windows Agents Plugin (windows-slaves)", "version": "1.4" }, { "plugin": "Web for Blue Ocean (blueocean-web)", "version": "1.17.0" }, { "plugin": "Workspace Cleanup Plugin (ws-cleanup)", "version": "0.37" }, { "plugin": "bouncycastle API Plugin (bouncycastle-api)", "version": "2.17" }, { "plugin": "i18n for Blue Ocean (blueocean-i18n)", "version": "1.17.0" } ] Add Comment   This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200967.1564163970000.2039.1564164000341%40Atlassian.JIRA.