This is the same issue as https://github.com/openshift/origin/issues/6934
I reproduced it and see in the kubelet logs
Error syncing pod c81fef8b-d0d0-11e5-a0c1-080027242396, skipping: failed to
"StartContainer" for "sample-phpinfo" with ErrImagePull: "Error: image
library/sample-phpinfo:latest not found"
If you want to workaround that issue, you will need to setup an ICT and
have the deployment run automatically once the build finishes.
triggers:
- imageChangeParams:
automatic: true
containerNames:
- sample-phpinfo
from:
kind: ImageStreamTag
name: sample-phpinfo:latest
type: ImageChange
instead of
triggers: []
On Thu, Feb 11, 2016 at 2:31 PM, François VILLAIN <[email protected]>
wrote:
> Hello Dan,
>
>
> Sorry for the lack of response.. I went on vacations and a crash made us
> reinstall the whole cluster...
>
>
> Anyway , I'm just getting back to the subject and still getting the same
> results and error messages.
>
> I recreated everything, so here are the definition files and various logs :
>
>
> - Definition of template objects : http://pastebin.com/6srmPJAS
> - Log level (9) of deployment on CLI : http://pastebin.com/Z4m0YfB8
> - Logs of the deploy pod : http://pastebin.com/6Uz3yhW1
> - Describe of the deploy pod : http://pastebin.com/2cZN04zG
> - Docker logs of containers : http://pastebin.com/ds9A4RgX
> - Deploy pod YAML output : http://pastebin.com/5BKtYbdK
>
> I hope this can help you
>
>
> Thanks for you help
> *---------------------*
> *François Villain*
>
> *DevOps - AAIO *
>
>
>
> ------------------------------
> *De :* Dan Mace <[email protected]>
> *Envoyé :* mercredi 6 janvier 2016 16:26
> *À :* François VILLAIN
> *Cc :* Michalis Kargakis; [email protected]
> *Objet :* Re: Pod not passing deployment tests
>
>
>
> On Wed, Jan 6, 2016 at 10:00 AM, François VILLAIN <[email protected]>
> wrote:
> > Here is all i got :
> >
> > CONTAINER ID IMAGE
>
> COMMAND CREATED STATUS
> PORTS NAMES
> > 9e332c821d82 openshift/origin-pod:v1.1.0.1
>
> "/pod" 31 minutes ago Exited (0) 21 minutes ago
>
>
> k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd
> > 55967e6c5c82
> 172.30.56.201:5000/jfr4/debian-80-lc-apache-wordpress@sha256:53bbfa0e4b8aa0f6d1c3ee709107066169d3507307014912726c8bebe5e9e9f0
> "/usr/sbin/runit_boot" 42 minutes ago Up 42 minutes
>
>
> k8s_test-apachewordpress.336aaeef_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_7d5b41e0
> > 422e4332abab openshift/origin-docker-registry:v1.1.0.1
>
> "/bin/sh -c 'REGISTRY" 58 minutes ago Up 58 minutes
>
>
> k8s_registry.5a0244b_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_e1f870e0
> > aca25d6b48c4 openshift/origin-pod:v1.1.0.1
>
> "/pod" 58 minutes ago Up 58 minutes
>
>
> k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_4bcf1754
> > 49e418f3c464 openshift/origin-haproxy-router:v1.1.0.1
>
> "/usr/bin/openshift-r" 58 minutes ago Up 58 minutes
>
>
> k8s_router.8fe4297d_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_5687da01
> > 4c019f72a1b6 openshift/origin-pod:v1.1.0.1
>
> "/pod" 58 minutes ago Created
>
>
> k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_5fe0fd41
> > 0e86cbfab7f0 openshift/origin-pod:v1.1.0.1
>
> "/pod" 58 minutes ago Up 58 minutes
>
>
> k8s_POD.bafffdb8_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_3af3542f
> > 519829fd6951 openshift/origin-pod:v1.1.0.1
>
> "/pod" 58 minutes ago Up 58 minutes
>
>
> k8s_POD.425ddcfc_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_a85a5ead
> > 3dcb164c7f8a
> 172.30.56.201:5000/jfr4/debian-80-lc-apache-wordpress@sha256:53bbfa0e4b8aa0f6d1c3ee709107066169d3507307014912726c8bebe5e9e9f0
> "/usr/sbin/runit_boot" 29 hours ago Exited (111) About an hour
> ago
> k8s_test-apachewordpress.336aaeef_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_b65b30ed
> > 01a3b9501f7f openshift/origin-pod:v1.1.0.1
>
> "/pod" 29 hours ago Exited (0) About an hour ago
>
> k8s_POD.bafffdb8_apachewordpress.test.com-2-q2lg5_jfr4_d2e58621-b390-11e5-9910-0a57f8bdd6b3_a61e7d00
> > 6156365c321a openshift/origin-docker-registry:v1.1.0.1
>
> "/bin/sh -c 'REGISTRY" 8 days ago Exited (2) About an hour ago
>
> k8s_registry.5a0244b_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_221bd690
> > c7fdabda7172 openshift/origin-haproxy-router:v1.1.0.1
>
> "/usr/bin/openshift-r" 8 days ago Exited (2) About an hour ago
>
> k8s_router.8fe4297d_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_22b62a48
> > e75a4750e79b openshift/origin-pod:v1.1.0.1
>
> "/pod" 8 days ago Exited (0) About an hour ago
>
> k8s_POD.7c1fe15_docker-registry-12-elryb_default_0ca0efc2-ad70-11e5-a609-0a57f8bdd6b3_196dba88
> > 482fb0703f7e openshift/origin-pod:v1.1.0.1
>
> “/
> pod” 8 days ago Exited (0) About an hour ago
>
> k8s_POD.425ddcfc_router-4-m068y_default_0cd882c5-ad70-11e5-a609-0a57f8bdd6b3_851a9591
>
>
> François
> ,
>
> Can you also show the output of `oc get pod -o yams
> test/phpfpm-app-1
> `? If you have OpenShift master logs (preferably running at
> —
> loglevel=4) that would also be helpful.
>
>
> > ________________________________________
> > De : Dan Mace <[email protected]>
> > Envoyé : mercredi 6 janvier 2016 15:53
> > À : François VILLAIN
> > Cc : Michalis Kargakis; [email protected]
> > Objet : Re: Pod not passing deployment tests
> >
> > On Wed, Jan 6, 2016 at 9:35 AM, François VILLAIN
> > <[email protected]> wrote:
> >>
> >> This is what i get (I just started another deploy manually):
> >>
> >> # sudo docker ps
> >> CONTAINER ID IMAGE
> COMMAND CREATED STATUS
> PORTS NAMES
> >> 9e332c821d82 openshift/origin-pod:v1.1.0.1
> "/pod" 4 minutes ago Up 4 minutes
>
>
> k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd
> >>
> >> # sudo docker inspect 9e332c821d82
> >> [
> >> {
> >> "Id":
> "9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a",
> >> "Created": "2016-01-06T14:27:28.296408249Z",
> >> "Path": "/pod",
> >> "Args": [],
> >> "State": {
> >> "Running": true,
> >> "Paused": false,
> >> "Restarting": false,
> >> "OOMKilled": false,
> >> "Dead": false,
> >> "Pid": 7739,
> >> "ExitCode": 0,
> >> "Error": "",
> >> "StartedAt": "2016-01-06T14:27:29.339842008Z",
> >> "FinishedAt": "0001-01-01T00:00:00Z"
> >> },
> >> "Image":
> "04d505e60cd48657230be57fcaa8a490b3c48b0d0217a632f3a5bf7e3ffc4302",
> >> "NetworkSettings": {
> >> "Bridge": "",
> >> "EndpointID":
> "15137677eb35d24abf9840b60ab68aeb824aec91b90dd77b48d41bd78737131a",
> >> "Gateway": "10.1.1.1",
> >> "GlobalIPv6Address": "",
> >> "GlobalIPv6PrefixLen": 0,
> >> "HairpinMode": false,
> >> "IPAddress": "10.1.1.4",
> >> "IPPrefixLen": 24,
> >> "IPv6Gateway": "",
> >> "LinkLocalIPv6Address": "",
> >> "LinkLocalIPv6PrefixLen": 0,
> >> "MacAddress": "02:42:0a:01:01:04",
> >> "NetworkID":
> "777c1fc8f94a1b06445092cf2441d0423f5f2f1b51fe500a27c76f4121953369",
> >> "PortMapping": null,
> >> "Ports": {},
> >> "SandboxKey": "/var/run/docker/netns/9e332c821d82",
> >> "SecondaryIPAddresses": null,
> >> "SecondaryIPv6Addresses": null
> >> },
> >> "ResolvConfPath":
> "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/resolv.conf",
> >> "HostnamePath":
> "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/hostname",
> >> "HostsPath":
> "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/hosts",
> >> "LogPath":
> "/var/lib/docker/containers/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a/9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a-json.log",
> >> "Name":
> "/k8s_POD.f61fe19_phpfpm-app-2-8g07w_test_9cfb7546-b481-11e5-8adb-0a57f8bdd6b3_0fa51bbd",
> >> "RestartCount": 0,
> >> "Driver": "devicemapper",
> >> "ExecDriver": "native-0.2",
> >> "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c6,c0",
> >> "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c6,c0",
> >> "AppArmorProfile": "",
> >> "ExecIDs": null,
> >> "HostConfig": {
> >> "Binds": null,
> >> "ContainerIDFile": "",
> >> "LxcConf": null,
> >> "Memory": 0,
> >> "MemorySwap": -1,
> >> "CpuShares": 2,
> >> "CpuPeriod": 0,
> >> "CpusetCpus": "",
> >> "CpusetMems": "",
> >> "CpuQuota": 0,
> >> "BlkioWeight": 0,
> >> "OomKillDisable": false,
> >> "MemorySwappiness": null,
> >> "Privileged": false,
> >> "PortBindings": null,
> >> "Links": null,
> >> "PublishAllPorts": false,
> >> "Dns": [
> >> "10.100.134.27",
> >> "10.100.0.2"
> >> ],
> >> "DnsSearch": [
> >> "test.svc.cluster.local",
> >> "svc.cluster.local",
> >> "cluster.local",
> >> "eu-west-1.compute.internal"
> >> ],
> >> "ExtraHosts": null,
> >> "VolumesFrom": null,
> >> "Devices": null,
> >> "NetworkMode": "default",
> >> "IpcMode": "",
> >> "PidMode": "",
> >> "UTSMode": "",
> >> "CapAdd": null,
> >> "CapDrop": null,
> >> "GroupAdd": null,
> >> "RestartPolicy": {
> >> "Name": "",
> >> "MaximumRetryCount": 0
> >> },
> >> "SecurityOpt": [
> >> "label:level:s0:c6,c0"
> >> ],
> >> "ReadonlyRootfs": false,
> >> "Ulimits": null,
> >> "LogConfig": {
> >> "Type": "json-file",
> >> "Config": {}
> >> },
> >> "CgroupParent": "",
> >> "ConsoleSize": [
> >> 0,
> >> 0
> >> ]
> >> },
> >> "GraphDriver": {
> >> "Name": "devicemapper",
> >> "Data": {
> >> "DeviceId": "4090",
> >> "DeviceName":
> "docker-202:1-50348421-9e332c821d828d16202dbaca8511936ffb6dfd86a13c4595b4f9511b30c6a14a",
> >> "DeviceSize": "107374182400"
> >> }
> >> },
> >> "Mounts": [],
> >> "Config": {
> >> "Hostname": "phpfpm-app-2-8g07w",
> >> "Domainname": "",
> >> "User": "",
> >> "AttachStdin": false,
> >> "AttachStdout": false,
> >> "AttachStderr": false,
> >> "ExposedPorts": null,
> >> "PublishService": "",
> >> "Tty": false,
> >> "OpenStdin": false,
> >> "StdinOnce": false,
> >> "Env": [
> >> "NGINX_APP_SERVICE_PORT=80",
> >> "PHPFPM_APP_PORT_9000_TCP=tcp://172.30.164.254:9000",
> >> "KUBERNETES_PORT_53_UDP_PROTO=udp",
> >> "KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53",
> >> "KUBERNETES_PORT_53_TCP_PROTO=tcp",
> >> "NGINX_APP_SERVICE_HOST=172.30.152.11",
> >> "NGINX_APP_PORT_80_TCP_ADDR=172.30.152.11",
> >> "KUBERNETES_SERVICE_HOST=172.30.0.1",
> >> "KUBERNETES_SERVICE_PORT_HTTPS=443",
> >> "KUBERNETES_PORT_443_TCP_PROTO=tcp",
> >> "PHPFPM_APP_PORT_9000_TCP_PORT=9000",
> >> "KUBERNETES_PORT_53_UDP_ADDR=172.30.0.1",
> >> "NGINX_APP_PORT_80_TCP_PROTO=tcp",
> >> "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
> >> "KUBERNETES_PORT_443_TCP_PORT=443",
> >> "KUBERNETES_PORT_53_UDP_PORT=53",
> >> "NGINX_APP_PORT=tcp://172.30.152.11:80",
> >> "PHPFPM_APP_SERVICE_HOST=172.30.164.254",
> >> "KUBERNETES_SERVICE_PORT=443",
> >> "KUBERNETES_PORT_53_UDP=udp://172.30.0.1:53",
> >> "KUBERNETES_PORT_53_TCP_PORT=53",
> >> "NGINX_APP_PORT_80_TCP_PORT=80",
> >> "PHPFPM_APP_PORT=tcp://172.30.164.254:9000",
> >> "PHPFPM_APP_PORT_9000_TCP_PROTO=tcp",
> >> "NGINX_APP_PORT_80_TCP=tcp://172.30.152.11:80",
> >> "PHPFPM_APP_SERVICE_PORT_PHPFPM=9000",
> >> "PHPFPM_APP_PORT_9000_TCP_ADDR=172.30.164.254",
> >> "KUBERNETES_SERVICE_PORT_DNS=53",
> >> "KUBERNETES_SERVICE_PORT_DNS_TCP=53",
> >> "NGINX_APP_SERVICE_PORT_HTTP=80",
> >> "PHPFPM_APP_SERVICE_PORT=9000",
> >> "KUBERNETES_PORT=tcp://172.30.0.1:443",
> >> "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
> >> "KUBERNETES_PORT_53_TCP_ADDR=172.30.0.1"
> >> ],
> >> "Cmd": null,
> >> "Image": "openshift/origin-pod:v1.1.0.1",
> >> "Volumes": null,
> >> "VolumeDriver": "",
> >> "WorkingDir": "",
> >> "Entrypoint": [
> >> "/pod"
> >> ],
> >> "NetworkDisabled": false,
> >> "MacAddress": "",
> >> "OnBuild": null,
> >> "Labels": {
> >> "io.kubernetes.pod.name": "test/phpfpm-app-2-8g07w",
> >> "io.kubernetes.pod.terminationGracePeriod": "30"
> >> }
> >> }
> >> }
> >> ]
> >>
> >> ---------------------
> >> François Villain
> >> DevOps - AAIO
> >
> > François,
> >
> > Is this the only container you see? The one you are looking at is the
> > Kube networking container which isn’t the same as the app container. I
> > notice that you didn’t use “docker ps -a” which will also show dead
> > containers. Could you try that and see if there’s another failed
> > container?
> >
> >> ________________________________________
> >> De : Dan Mace <[email protected]>
> >> Envoyé : mercredi 6 janvier 2016 14:46
> >> À : François VILLAIN
> >> Cc : Michalis Kargakis; [email protected]
> >> Objet : Re: Pod not passing deployment tests
> >>
> >> François,
> >>
> >> Do you have access to the docker daemon in this environment? I’d be
> >> interested to see the output of `docker ps -a` and then a `docker
> >> inspect` of your app container which may have failed to start or be
> >> created.
> >>
> >> Michalis, regarding the recreate strategy: even when using the rolling
> >> strategy, the deployer will always start with recreate for the initial
> >> deployment since the “legacy” Kubernetes rolling updater doesn’t
> >> support initial rollouts (only transitions between two existing RCs).
> >>
> >> On Wed, Jan 6, 2016 at 6:15 AM, François VILLAIN
> >> <[email protected]> wrote:
> >>> I first tried without the probes , but wasn't any more successful
> >>>
> >>> I have another DC in my template indeed.
> >>>
> >>>
> >>> Here is the full template file i created :
> http://pastebin.com/EZJaCeQj
> >>>
> >>>
> >>>
> >>> ---------------------
> >>> François Villain
> >>> DevOps - AAIO
> >>>
> >>>
> >>>
> >>> ________________________________
> >>> De : Michalis Kargakis <[email protected]>
> >>> Envoyé : mercredi 6 janvier 2016 12:10
> >>> À : François VILLAIN
> >>> Cc : [email protected]
> >>> Objet : Re: Pod not passing deployment tests
> >>>
> >>> Hi François,
> >>>
> >>> can you deploy the app if you remove the probes?
> >>> Also it's really strange that you are specifying a Rolling strategy
> but in
> >>> the logs it seems that Recreate is running.
> >>> Do you have any other DCs in your template?
> >>>
> >>> On Wed, Jan 6, 2016 at 11:11 AM, François VILLAIN <
> [email protected]>
> >>> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>> I'm having trouble deploying a pod running a php-fpm daemon (on port
> >>>> 9000).
> >>>> I created a DeploymentConfig in my template for that:
> >>>>
> >>>>
> >>>> - kind: DeploymentConfig
> >>>> apiVersion: v1
> >>>> metadata:
> >>>> name: "phpfpm-${SERVICE_NAME}"
> >>>> annotations:
> >>>> description: How to deploy the PHP App
> >>>> spec:
> >>>> strategy:
> >>>> type: Rolling
> >>>> replicas: 1
> >>>> selector:
> >>>> name: "phpfpm-${SERVICE_NAME}"
> >>>> template:
> >>>> metadata:
> >>>> name: "phpfpm-${SERVICE_NAME}"
> >>>> labels:
> >>>> name: "phpfpm-${SERVICE_NAME}"
> >>>> spec:
> >>>> volumes: []
> >>>> containers:
> >>>> - name: "phpfpm-${SERVICE_NAME}"
> >>>> imagePullPolicy: 'Always'
> >>>> image: "test/sample-phpinfo:latest"
> >>>> ports:
> >>>> - containerPort: 9000
> >>>> env: []
> >>>> readinessProbe:
> >>>> timeoutSeconds: 1
> >>>> initialDelaySeconds: 5
> >>>> tcpSocket:
> >>>> port: 9000
> >>>> livenessProbe:
> >>>> timeoutSeconds: 1
> >>>> initialDelaySeconds: 5
> >>>> tcpSocket:
> >>>> port: 9000
> >>>>
> >>>>
> >>>> And here is what i get in the deployment step:
> >>>>
> >>>> 1 I0104 14:57:09.139422 1 deployer.go:196] Deploying
> >>>> test/phpfpm-app-1 for the first time (replicas: 1)
> >>>> 2 I0104 14:57:09.143539 1 recreate.go:105] Scaling
> test/phpfpm-app-1
> >>>> to 1 before performing acceptance check
> >>>> 3 I0104 14:57:11.194413 1 recreate.go:110] Performing acceptance
> >>>> check of test/phpfpm-app-1
> >>>> 4 I0104 14:57:11.195425 1 lifecycle.go:379] Waiting 600 seconds
> for
> >>>> pods owned by deployment "test/phpfpm-app-1" to become ready
> (checking every
> >>>> 1 seconds; 0 pods previously accepted)
> >>>> 5 F0104 15:07:11.195704 1 deployer.go:65] update acceptor
> rejected
> >>>> test/phpfpm-app-1: pods for deployment "test/phpfpm-app-1" took
> longer than
> >>>> 600 seconds to become ready
> >>>>
> >>>> Ca someone please tell me what's wrong ?
> >>>>
> >>>>
> >>>> I got a same pod with a nginx instance perfectly working btw, so i
> guess
> >>>> something is bad but i can't figure ou what
> >>>>
> >>>> exactly...
> >>>>
> >>>> ---------------------
> >>>> François Villain
> >>>> DevOps - AAIO
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> users mailing list
> >>>> [email protected]
> >>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> [email protected]
> >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> >>>
>
>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
