docker run hello-world works. On Tue., 4 Oct. 2016 at 8:06 pm, Michail Kargakis <mkarg...@redhat.com> wrote:
> Can you run docker containers directly via the docker command? > > On Tue, Oct 4, 2016 at 10:18 AM, Lionel Orellana <lione...@gmail.com> > wrote: > > The deployment of the router hangs in Pending status. If I cancel the > deployment, wait a little and try to deploy again I get > > "Deployment of version 3 awaiting cancellation of older running > deployments" > > This shows in the logs: > > Oct 04 18:54:58 poc-docker02.aipo.gov.au origin-node[1773]: E1004 > 18:54:58.805480 1815 kubelet.go:2684] *Failed cleaning pod*s: [remove > /var/lib/origin/openshift.local.volumes/pods/0d0c9fa0-8624-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/default-token-36qqf: device or resource busy, remove > /var/lib/origin/openshift.local.volumes/pods/10493062-89ce-11e6-827b-005056915814/volumes/ > kubernetes.io~secret/server-certificate: device or resource busy, remove > /var/lib/origin/openshift.local.volumes/pods/104a80c3-89ce-11e6-827b-005056915814/volumes/ > kubernetes.io~nfs/pv-registry/.snapshot/hourly.2016-10-04_1505: *device > or resource busy*, remove > /var/lib/origin/openshift.local.volumes/pods/1204b6d2-8556-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/deployer-token-ygldd: *device or resource busy*, > remove > /var/lib/origin/openshift.local.volumes/pods/19f736b2-8619-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/deployer-token-lhbl7: device or resource busy, > remove > /var/lib/origin/openshift.local.volumes/pods/1a16e853-8460-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-token-33845: device or resource busy, remove > /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/ > kubernetes.io~secret/router-token-5qacw: *device or resource busy*, > remove > /var/lib/origin/openshift.local.volumes/pods/29857e2d-8554-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-token-ftgao: device or resource busy, remove > /var/lib/origin/openshift.local.volumes/pods/2f33e60b-854d-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-token-ftgao: device or resource busy, remove > /var/lib/origin/openshift.local.volumes/pods/3810b449-89d5-11e6-827b-005056915814/volumes/ > kubernetes.io~secret/deployer-token-1ehih: device or resource busy, > remove > /var/lib/origin/openshift.local.volumes/pods/3e5e918b-85e1-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-dockercfg-4kyom-push: *device or resource > busy*, remove > /var/lib/origin/openshift.local.volumes/pods/3e8cc25e-85ee-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-dockercfg-4kyom- > > If I then force-delete the old deployment pod by doing > > oc delete pod router-2-deploy --grace-period=0 > > then the next deployment starts and hangs in Pending again. > > A similar but smaller output appears in the logs > > Oct 04 18:55:14 poc-docker02.aipo.gov.au origin-node[1773]: I1004 > 18:55:14.421502 1815 kubelet.go:2117] *Failed to remove orphaned pod* > "1a997bdd-7ed8-11e6-adb7-005056915814" dir; err: remove > /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/ > kubernetes.io~secret/router-token-5qacw: *device or resource busy* > > This device is a tmpfs mount. > > -bash-4.2$ sudo df -h | grep 1a997bdd > tmpfs > 1.9G 0 1.9G 0% > /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/ > kubernetes.io~secret/server-certificate > tmpfs > 1.9G 0 1.9G 0% > /var/lib/origin/openshift.local.volumes/pods/1a997bdd-7ed8-11e6-adb7-005056915814/volumes/ > kubernetes.io~secret/router-token-5qacw > -bash-4.2$ > > Restarting the docker daemon doesn't get rid of them. I'm well and truly > stuck. > > > On 4 October 2016 at 17:40, Lionel Orellana <lione...@gmail.com> wrote: > > All the "device or resource busy" errors seem related to tmpfs mounts for > secret volumes. > > > > On 4 October 2016 at 17:32, Lionel Orellana <lione...@gmail.com> wrote: > > Hi All, > > I had a v1.3 cluster with a master and a node going. Both servers were > rebooted over the weekend and all hell broke loose. > > The registry, the router and all apps I had running have stopped working. > > I see quite a few of these errors in the logs: > > Oct 04 17:14:07 poc-docker02.aipo.gov.au origin-node[1773]: I1004 > 17:14:07.510615 1815 kubelet.go:2117] Failed to remove orphaned pod > "c449d37f-8549-11e6-83c1-005056915814" dir; err: remove > /var/lib/origin/openshift.local.volumes/pods/c449d37f-8549-11e6-83c1-005056915814/volumes/ > kubernetes.io~secret/builder-token-ftgao:* device or resource busy* > > I don't really know what happened, how I got into this state. The registry > is stuck in "Container Creating". If I start a new deployment of the > router the deployment pod doesn't get past "Pending". > > Seems like I can't delete pods either. They get stuck in "Terminating". > > Not sure how to narrow this down. Any help greatly appreciated. > > Thanks > > Lionel. > > > > > _______________________________________________ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
