This morning our cluster started experiencing an odd error on multiple nodes. Pods are stuck in the terminating phase. In our node log I see the following:
Sep 5 19:17:22 ip-10-0-1-184 origin-node: E0905 19:17:22.043257 112306 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf-default-token-f18hx\" (\"182285ee-9267-11e7-b7be-06415eb17bbf\")" failed. No retries permitted until 2017-09-05 19:17:22.543230782 +0000 UTC (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf-default-token-f18hx" (volume.spec.Name: "default-token-f18hx") pod "182285ee-9267-11e7-b7be-06415eb17bbf" (UID: "182285ee-9267-11e7-b7be-06415eb17bbf") with: remove /var/lib/origin/openshift.local.volumes/pods/182285ee-9267-11e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx: device or resource busy That path is not mounted (running mount does not list it) and running fuser -v on that directory does not show anything. Trying to rmdir results in a similar error: sudo rmdir var/lib/origin/openshift.local.volumes/pods/182285ee-9267-11e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx rmdir: failed to remove ‘var/lib/origin/openshift.local.volumes/pods/182285ee-9267-11e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx’: No such file or directory Is anyone else getting this error?
_______________________________________________ dev mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
