Please open a bug in openshift/origin and we'll triage it there. On Tue, Sep 5, 2017 at 5:14 PM, Patrick Tescher <[email protected]> wrote:
> The pods are still “terminating” and have been stuck in that state. New > pods have come and gone since then but the stuck ones are still stuck. > > > On Sep 5, 2017, at 2:13 PM, Clayton Coleman <[email protected]> wrote: > > So the errors recur continuously for a given pod once they start happening? > > On Tue, Sep 5, 2017 at 5:07 PM, Patrick Tescher <[email protected]> > wrote: > >> No patches have been applied since we upgraded to 3.6.0 over a week ago. >> The errors just popped up for a few different pods in different namespaces. >> The only thing we did today was launch a stateful set in a new namespace. >> Those pods were not the ones throwing this error. >> >> >> On Sep 5, 2017, at 1:19 PM, Clayton Coleman <[email protected]> wrote: >> >> Were any patches applied to the system? Some of these are normal if they >> happen for a brief period of time. Are you seeing these errors >> continuously for the same pod over and over? >> >> On Tue, Sep 5, 2017 at 3:23 PM, Patrick Tescher <[email protected] >> > wrote: >> >>> This morning our cluster started experiencing an odd error on multiple >>> nodes. Pods are stuck in the terminating phase. In our node log I see the >>> following: >>> >>> Sep 5 19:17:22 ip-10-0-1-184 origin-node: E0905 19:17:22.043257 112306 >>> nestedpendingoperations.go:262] Operation for "\" >>> kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf >>> -default-token-f18hx\ >>> <http://kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf-default-token-f18hx%5C>" >>> (\"182285ee-9267-11e7-b7be-06415eb17bbf\")" failed. No retries >>> permitted until 2017-09-05 19:17:22.543230782 +0000 UTC >>> (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for >>> volume "kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf-d >>> efault-token-f18hx" (volume.spec.Name <http://volume.spec.name/>: >>> "default-token-f18hx") pod "182285ee-9267-11e7-b7be-06415eb17bbf" (UID: >>> "182285ee-9267-11e7-b7be-06415eb17bbf") with: remove >>> /var/lib/origin/openshift.local.volumes/pods/182285ee-9267-1 >>> 1e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx: >>> device or resource busy >>> >>> That path is not mounted (running mount does not list it) and running >>> fuser -v on that directory does not show anything. Trying to rmdir results >>> in a similar error: >>> >>> sudo rmdir var/lib/origin/openshift.local.volumes/pods/182285ee-9267-11 >>> e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx >>> rmdir: failed to remove ‘var/lib/origin/openshift.loca >>> l.volumes/pods/182285ee-9267-11e7-b7be-06415eb17bbf/volumes/ >>> kubernetes.io~secret/default-token-f18hx’: No such file or directory >>> >>> Is anyone else getting this error? >>> >>> >>> _______________________________________________ >>> dev mailing list >>> [email protected] >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev >>> >>> >> >> > >
_______________________________________________ dev mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
