You may be hitting the limit of concurrent connections to k8s api, see https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136
On Mon, Dec 10, 2018 at 8:01 AM Simon Young < [email protected]> wrote: > Hi, > > We are trying to use the Kubernetes Plugin to run tests in a different > container - but *almost* every time we try and execute a command in the > second container, the build fails immediately. I say "almost" because the > build always succeeds after restarting the Jenkins master (or possibly > after it's been idle for a long time). > > Here's the Jenkinsfile we're using to test: > > def label = "k8s-test-${UUID.randomUUID().toString()}" > > podTemplate( > cloud: 'kubernetes-test.k8s.local', > namespace: 'mynamespace', > label: label, > yaml: """ > apiVersion: v1 > kind: Pod > spec: > containers: > - name: busybox > image: busybox > command: ['cat'] > tty: true > """ > ) { > > node (label) { > stage ('Get Agent Info') { > sh "java -version" > } > stage ('Run tests') { > container('busybox') { > sh "echo foo" > sh "netstat -tln" > sh "df -h" > } > } > stage ('Grab Logs') { > containerLog('jnlp') > } > } > } > > This fails at the "echo foo" step. The error reported by the pipeline is: > > Task okhttp3.RealCall$AsyncCall@b5277fa rejected from > java.util.concurrent.ThreadPoolExecutor@6c383ec7[Terminated, pool size = > 0, active threads = 0, queued tasks = 0, completed tasks = 3] > > What we know so far: > > * Both containers are successfully deployed to the same pod, as expected. > * If we don't try and run the shell commands on the 'busybox' container, > the build completes successfully. > * If Jenkins is restarted, the build passes. But subsequent builds fail. > * The Agent logs contain similar exceptions whether the builds pass or > fail, so it's hard to pinpoint a cause. > > The ThreadPool exception may imply that *something* has run out of > Executors, but it's not clear what, or how to rectify the situation. > > Has anyone seen anything like this? Any ideas how to get to the bottom of > it? As per the plugin's README, I've created Jenkins log recorders for > org.csanchez.jenkins.plugins.kubernetes and okhttp3, but there are no > obvious problems reported in the logs. > > Software Versions: > > Jenkins: v2.150.1 > Kubernetes Plugin: 1.13.7 > Kubernetes: 1.10.11 (same behaviour observed on 1.8.7) > > All suggestions appreciated! > > Thanks, > > Simon. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/141b526e-f00b-434d-9f43-ec97fdb23df7%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/141b526e-f00b-434d-9f43-ec97fdb23df7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6PM4RaDkYy-MZ8CJceoJxap-cxr3bMQZtR7oVzuJd2Tjg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
