I would rollback to 1.13.5 for now then On Tue, Dec 11, 2018, 00:49 Simon Young <[email protected]> wrote:
> Hi Carlos, > > Thanks for the suggestion. I have tried changing the "Max connections to > Kubernetes API" and this does have an effect, but not in a way I'd > expect... I've found that if I change that value in any way at all (either > increase or decrease), then the next build always passes, and all > subsequent builds fail. > > So for example, I tried raising the value to 200 via the Jenkins UI. I hit > 'Apply' and re-ran the job and it passed, but it failed the next time it > ran, so I raised the value to 400. Again, the next job passed, but any > after that failed. At this point, I suspected that changing the value in > any way might have this effect, so I reduced the value back to 32. Same > thing happened - pass, then subsequent fails. I reduced the value all the > way down to 1, and observed the same symptoms. > > I tried changing other plugin configuration values such as "Connection > Timeout" and "Container Cap", but this did not have the same effect. > > It's also worth noting if I don't run the job for about an hour, the next > build passes and subsequent builds fail. Same happens if I restart Jenkins > (but not if I just reload the config from disk). > > So it seems like some internal counter on the Jenkins master is reaching a > limit, and the counter is reset when the config changes or after a period > of activity. Neither master nor slave appear to be running out of physical > resources, and this is the only job we're running on the master. > > Have you any idea what could be going on here? > > Simon. > > > > On Monday, December 10, 2018 at 4:10:27 PM UTC, Carlos Sanchez wrote: >> >> You may be hitting the limit of concurrent connections to k8s api, see >> https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136 >> >> On Mon, Dec 10, 2018 at 8:01 AM Simon Young < >> [email protected]> wrote: >> > Hi, >>> >>> We are trying to use the Kubernetes Plugin to run tests in a different >>> container - but *almost* every time we try and execute a command in the >>> second container, the build fails immediately. I say "almost" because the >>> build always succeeds after restarting the Jenkins master (or possibly >>> after it's been idle for a long time). >>> >>> Here's the Jenkinsfile we're using to test: >>> >>> def label = "k8s-test-${UUID.randomUUID().toString()}" >>> >>> podTemplate( >>> cloud: 'kubernetes-test.k8s.local', >>> namespace: 'mynamespace', >>> label: label, >>> yaml: """ >>> apiVersion: v1 >>> kind: Pod >>> spec: >>> containers: >>> - name: busybox >>> image: busybox >>> command: ['cat'] >>> tty: true >>> """ >>> ) { >>> >>> node (label) { >>> stage ('Get Agent Info') { >>> sh "java -version" >>> } >>> stage ('Run tests') { >>> container('busybox') { >>> sh "echo foo" >>> sh "netstat -tln" >>> sh "df -h" >>> } >>> } >>> stage ('Grab Logs') { >>> containerLog('jnlp') >>> } >>> } >>> } >>> >>> This fails at the "echo foo" step. The error reported by the pipeline is: >>> >>> Task okhttp3.RealCall$AsyncCall@b5277fa rejected from >>> java.util.concurrent.ThreadPoolExecutor@6c383ec7[Terminated, pool size >>> = 0, active threads = 0, queued tasks = 0, completed tasks = 3] >>> >>> What we know so far: >>> >>> * Both containers are successfully deployed to the same pod, as expected. >>> * If we don't try and run the shell commands on the 'busybox' container, >>> the build completes successfully. >>> * If Jenkins is restarted, the build passes. But subsequent builds fail. >>> * The Agent logs contain similar exceptions whether the builds pass or >>> fail, so it's hard to pinpoint a cause. >>> >>> The ThreadPool exception may imply that *something* has run out of >>> Executors, but it's not clear what, or how to rectify the situation. >>> >>> Has anyone seen anything like this? Any ideas how to get to the bottom >>> of it? As per the plugin's README, I've created Jenkins log recorders for >>> org.csanchez.jenkins.plugins.kubernetes and okhttp3, but there are no >>> obvious problems reported in the logs. >>> >>> Software Versions: >>> >>> Jenkins: v2.150.1 >>> Kubernetes Plugin: 1.13.7 >>> Kubernetes: 1.10.11 (same behaviour observed on 1.8.7) >>> >>> All suggestions appreciated! >>> >>> Thanks, >>> >>> Simon. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> >> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/141b526e-f00b-434d-9f43-ec97fdb23df7%40googlegroups.com >>> <https://groups.google.com/d/msgid/jenkinsci-users/141b526e-f00b-434d-9f43-ec97fdb23df7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/29389695-c3c2-415b-b945-e1b4a9099fbe%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/29389695-c3c2-415b-b945-e1b4a9099fbe%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6OXSoX0vzZxKhoYTNA0JRcAi-BzOsZvPTVO3gksBrGCxA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
