[JIRA] (JENKINS-54683) Pods entering error state

grzegorz.zieba....@gigaset.com (JIRA) Wed, 10 Jul 2019 06:59:42 -0700

Title: Message Title

Got similar issue:

master:

 
                                                                Updating pod for jenkins/pod-meta-zebra-check-234-6mqk5-3qmwv : Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={buildUrl=https://jenkins.dev.devnet.pl/job/meta-zebra-check/234/}, clusterName=null, creationTimestamp=2019-07-10T13:44:06Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={app=jenkins, component=agent, jenkins=slave, jenkins/pod-meta-zebra-check-234=true}, name=pod-meta-zebra-check-234-6mqk5-3qmwv, namespace=jenkins, ownerReferences=[], resourceVersion=33830607, selfLink=/api/v1/namespaces/jenkins/pods/pod-meta-zebra-check-234-6mqk5-3qmwv, uid=c93a30f8-a318-11e9-a741-005056b81a4d, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[cat], env=[], envFrom=[], image=registry.dev.devnet.pl/jenkins/build:v2.4.0, imagePullPolicy=Always, lifecycle=null, livenessProbe=null, name=build, ports=[], readinessProbe=null, resources=ResourceRequirements(limits=null, requests={cpu=Quantity(amount=3, format=null, additionalProperties={}), memory=Quantity(amount=4Gi, format=null, additionalProperties={})}, additionalProperties={}), securityContext=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=true, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/home/jenkins, mountPropagation=null, name=workspace-volume, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/var/run/secrets/kubernetes.io/serviceaccount, mountPropagation=null, name=default-token-ffqzp, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={}), Container(args=[], command=[], env=[EnvVar(name=JENKINS_SECRET, value=5dacc741ca9193d3625fb68f3e459a063784921015241cd131dda26eded3788d, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_TUNNEL, value=jenkins.dev.devnet.pl:50000, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_AGENT_NAME, value=pod-meta-zebra-check-234-6mqk5-3qmwv, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_NAME, value=pod-meta-zebra-check-234-6mqk5-3qmwv, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_URL, value=https://jenkins.dev.devnet.pl/, valueFrom=null, additionalProperties={}), EnvVar(name=HOME, value=/home/jenkins, valueFrom=null, additionalProperties={})], envFrom=[], image=registry.dev.devnet.pl/jenkins/jnlp-slave:3.29-1-alpine, imagePullPolicy=IfNotPresent, lifecycle=null, livenessProbe=null, name=jnlp, ports=[], readinessProbe=null, resources=ResourceRequirements(limits=null, requests={cpu=Quantity(amount=1, format=null, additionalProperties={}), memory=Quantity(amount=512Mi, format=null, additionalProperties={})}, additionalProperties={}), securityContext=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=null, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/home/jenkins, mountPropagation=null, name=workspace-volume, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/var/run/secrets/kubernetes.io/serviceaccount, mountPropagation=null, name=default-token-ffqzp, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=ClusterFirst, enableServiceLinks=true, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=worker-101, nodeSelector={node-role.kubernetes.io/worker=}, priority=0, priorityClassName=null, readinessGates=[], restartPolicy=Never, runtimeClassName=null, schedulerName=default-scheduler, securityContext=PodSecurityContext(fsGroup=null, runAsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, supplementalGroups=[], sysctls=[], additionalProperties={}), serviceAccount=default, serviceAccountName=default, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=30, tolerations=[Toleration(effect=NoExecute, key=node.kubernetes.io/not-ready, operator=Exists, tolerationSeconds=300, value=null, additionalProperties={}), Toleration(effect=NoExecute, key=node.kubernetes.io/unreachable, operator=Exists, tolerationSeconds=300, value=null, additionalProperties={})], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=null, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=workspace-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=default-token-ffqzp, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=420, items=[], optional=null, secretName=default-token-ffqzp, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=PodStatus(conditions=[PodCondition(lastProbeTime=null, lastTransitionTime=2019-07-10T13:44:06Z, message=null, reason=null, status=True, type=Initialized, additionalProperties={}), PodCondition(lastProbeTime=null, lastTransitionTime=2019-07-10T13:44:06Z, message=containers with unready status: [jnlp], reason=ContainersNotReady, status=False, type=Ready, additionalProperties={}), PodCondition(lastProbeTime=null, lastTransitionTime=2019-07-10T13:44:06Z, message=containers with unready status: [jnlp], reason=ContainersNotReady, status=False, type=ContainersReady, additionalProperties={}), PodCondition(lastProbeTime=null, lastTransitionTime=2019-07-10T13:44:06Z, message=null, reason=null, status=True, type=PodScheduled, additionalProperties={})], containerStatuses=[ContainerStatus(containerID=docker://16085a368a76e9e27283bcba9c7e5abf5b559469501714c67e77051f0977bdb4, image=registry.dev.devnet.pl/jenkins/build:v2.4.0, imageID=docker-pullable://registry.dev.devnet.pl/jenkins/build@sha256:1f5b9b012ed42a533c59aeba8b0bc20eedad45253fd5e567f70981dbd4372561, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=build, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=2019-07-10T13:44:08Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={}), ContainerStatus(containerID=docker://0ee0b09c732d9dbbe0f69af003134e9a1eb2f00d0dedc51a6b327fa6b3f7de37, image=registry.dev.devnet.pl/jenkins/jnlp-slave:3.29-1-alpine, imageID=docker-pullable://registry.dev.devnet.pl/jenkins/jnlp-slave@sha256:8d13ac6c95c94a40ce1a76b5afacabcfd31c3362e78b0fa53e923e65a4ba37e0, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=jnlp, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://0ee0b09c732d9dbbe0f69af003134e9a1eb2f00d0dedc51a6b327fa6b3f7de37, exitCode=255, finishedAt=2019-07-10T13:46:32Z, message=null, reason=Error, signal=null, startedAt=2019-07-10T13:44:09Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})], hostIP=172.29.7.245, initContainerStatuses=[], message=null, nominatedNodeName=null, phase=Running, podIP=10.244.11.175, qosClass=Burstable, reason=null, startTime=2019-07-10T13:44:06Z, additionalProperties={}), additionalProperties={})
Jul 10, 2019 1:47:06 PM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launchError in provisioning; agent=KubernetesSlave name: pod-meta-zebra-check-234-6mqk5-3qmwv, template=PodTemplate{, name='pod-meta-zebra-check-234-6mqk5', namespace='jenkins', slaveConnectTimeout=300, label='pod-meta-zebra-check-234', nodeUsageMode=EXCLUSIVE, annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821], yamls=[apiVersion: v1
kind: Pod
metadata:
  namespace: jenkins
  labels:
    app: jenkins
    component: agent
spec:
  containers:
    - name: jnlp
      image: jenkins/jnlp-slave:3.29-1-alpine
      imagePullPolicy: IfNotPresent
      resources:
        requests:
          cpu: 1000m
          memory: 512Mi
    - name: build
      image: jenkins/build:v2.4.0
      imagePullPolicy: Always
      resources:
        requests:
          cpu: 3000m
          memory: 4096Mi
      command:
        - cat
      tty: true
  nodeSelector:
    node-role.kubernetes.io/worker: ""
]}
java.lang.IllegalStateException: Pod has terminated containers: jenkins/pod-meta-zebra-check-234-6mqk5-3qmwv (jnlp)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:166)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.periodicAwait(AllContainersRunningPodWatcher.java:187)
	at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.await(AllContainersRunningPodWatcher.java:127)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:128)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
 
                                                            

pod:

 
                                                                $ kubectl -n jenkins logs pod-meta-zebra-check-234-6mqk5-3qmwv jnlp                                                                                        Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behaviorJul 10, 2019 1:44:09 PM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: pod-meta-zebra-check-234-6mqk5-3qmwvJul 10, 2019 1:44:09 PM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Jul 10, 2019 1:44:09 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 3.29Jul 10, 2019 1:44:09 PM hudson.remoting.Engine startEngineWARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jarsJul 10, 2019 1:44:09 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://jenkins.dev.devnet.pl/]Jul 10, 2019 1:44:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Jul 10, 2019 1:44:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability checkJul 10, 2019 1:44:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Agent discovery successful  Agent address: jenkins.dev.devnet.pl  Agent port:    50000  Identity:      15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36Jul 10, 2019 1:44:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: HandshakingJul 10, 2019 1:44:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to jenkins.dev.devnet.pl:50000Jul 10, 2019 1:44:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Trying protocol: JNLP4-connectJul 10, 2019 1:44:16 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Remote identity confirmed: 15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36Jul 10, 2019 1:44:16 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: ConnectedJul 10, 2019 1:45:06 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warnWARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/Jul 10, 2019 1:46:19 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: TerminatedJul 10, 2019 1:46:30 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Performing onReconnect operation.Jul 10, 2019 1:46:30 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1 onReconnectINFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@7cf2bdcaJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: pod-meta-zebra-check-234-6mqk5-3qmwvJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Jul 10, 2019 1:46:31 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 3.29Jul 10, 2019 1:46:31 PM hudson.remoting.Engine startEngineWARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jarsJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://jenkins.dev.devnet.pl/]Jul 10, 2019 1:46:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Jul 10, 2019 1:46:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability checkJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Agent discovery successful  Agent address: jenkins.dev.devnet.pl  Agent port:    50000  Identity:      15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36Jul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: HandshakingJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to jenkins.dev.devnet.pl:50000Jul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Trying protocol: JNLP4-connectJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Remote identity confirmed: 15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36Jul 10, 2019 1:46:31 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecvINFO: [JNLP4-connect connection to jenkins.dev.devnet.pl/172.29.7.98:50000] Local headers refused by remote: pod-meta-zebra-check-234-6mqk5-3qmwv is already connected to this master. Rejecting this connection.Jul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Protocol JNLP4-connect encountered an unexpected exceptionjava.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: pod-meta-zebra-check-234-6mqk5-3qmwv is already connected to this master. Rejecting this connection. at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:614) at hudson.remoting.Engine.run(Engine.java:474)Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: pod-meta-zebra-check-234-6mqk5-3qmwv is already connected to this master. Rejecting this connection. at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: java.nio.channels.ClosedChannelException ... 7 more
Jul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to jenkins.dev.devnet.pl:50000Jul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Server reports protocol JNLP4-plaintext not supported, skippingJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Protocol JNLP3-connect is not enabled, skippingJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Server reports protocol JNLP2-connect not supported, skippingJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Server reports protocol JNLP-connect not supported, skippingJul 10, 2019 1:46:31 PM hudson.remoting.jnlp.Main$CuiListener errorSEVERE: The server rejected the connection: None of the protocols were acceptedjava.lang.Exception: The server rejected the connection: None of the protocols were accepted at hudson.remoting.Engine.onConnectionRejected(Engine.java:682) at hudson.remoting.Engine.innerRun(Engine.java:639) at hudson.remoting.Engine.run(Engine.java:474)

Add Comment

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.195475.1542484359000.6898.1562767140659%40Atlassian.JIRA.
For more options, visit https://groups.google.com/d/optout.

[JIRA] (JENKINS-54683) Pods entering error state

Reply via email to