Re: Pods stuck on Terminating status
I had this at one point, but it was before I cared about the data in that cluster so I just rebuilt it, so you could just rebuild your cluster ;) But in all seriousness sounds like you need to do some etcd surgery, but I have no idea how that works. On Tue, 20 Mar 2018 at 4:00 am, bahhooo <bah...@gmail.com> wrote: > Hi Rodrigo, > > > Restarting master services did not help either.. > I tried afterwards again with grace period and force delete. Stil no luck. > > > > > Bahho > > On 16 March 2018 at 21:29, Rodrigo Bersa <rbe...@redhat.com> wrote: > >> Bahhoo, >> >> I believe that the namespace will get stuck also. 'Cause it will only be >> deleted after all of it's objects got deleted. >> >> I would try to restart the Masters services before. >> >> >> Regards, >> >> >> Rodrigo Bersa >> >> Cloud Consultant, RHCVA, RHCE >> >> Red Hat Brasil <https://www.redhat.com> >> >> rbe...@redhat.comM: +55-11-99557-5841 >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil >> pelo *Great Place to Work*. >> >> On Fri, Mar 16, 2018 at 5:25 PM, Bahhoo <bah...@gmail.com> wrote: >> >>> Hi Rodrigo, >>> >>> No PVs are used. One of the pods is a build pod, the other one's a >>> normal pod without storage. >>> I'll try deleting the namespace. I didn't want to do that,since I had >>> running pods in the namespace. >>> >>> Best, >>> Bahho >>> -- >>> Kimden: Rodrigo Bersa <rbe...@redhat.com> >>> Gönderme tarihi: 16.3.2018 16:12 >>> Kime: Bahhoo <bah...@gmail.com> >>> Bilgi: rahul334...@gmail.com; users <users@lists.openshift.redhat.com> >>> >>> Konu: Re: Pods stuck on Terminating status >>> >>> Hi Bahhoo, >>> >>> Are you using PVs on the "Terminating" POD? I heard about some issues >>> with PODs bounded to PV/PVCs provided by dynamic storage, where you have to >>> first remove the volume form POD, then the PVPVC. Just after that remove >>> the POD or the DeplymentConfig. >>> >>> If it's not the case, maybe restarting the atomic-openshift-master-* >>> services can work removing the inconsistent POD. >>> >>> >>> Regards, >>> >>> >>> Rodrigo Bersa >>> >>> Cloud Consultant, RHCVA, RHCE >>> >>> Red Hat Brasil <https://www.redhat.com> >>> >>> rbe...@redhat.comM: +55-11-99557-5841 >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> Red Hat é reconhecida entre as melhores empresas para trabalhar no >>> Brasil pelo *Great Place to Work*. >>> >>> On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: >>> >>>> Hi Rahul, >>>> >>>> That won't do it either. >>>> >>>> Thanks >>>> Bahho >>>> -- >>>> Kimden: Rahul Agarwal <rahul334...@gmail.com> >>>> Gönderme tarihi: 15.3.2018 22:26 >>>> Kime: bahhooo <bah...@gmail.com> >>>> Bilgi: users <users@lists.openshift.redhat.com> >>>> Konu: Re: Pods stuck on Terminating status >>>> >>>> Hi Bahho >>>> >>>> Try: oc delete all -l app= >>>> >>>> Thanks, >>>> Rahul >>>> >>>> On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have some zombie pods stuck on Terminating status on a OCP 3.7 >>>>> HA-cluster. >>>>> >>>>> oc delete with --grace-period=0 --force etc. won't work. >>>>> Docker restart. server reboot won't help either. >>>>> >>>>> I tried to find the pod key in etcd either in order to delete it >>>>> manually. I couldn't find it. >>>>> >>>>> Is there a way to delete these pods? >>>>> >>>>> >>>>> >>>>> >>>>> Bahho >>>>> >>>>> ___ >>>>> users mailing list >>>>> users@lists.openshift.redhat.com >>>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>>> >>>>> >>>> >>>> ___ >>>> users mailing list >>>> users@lists.openshift.redhat.com >>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>> >>>> >>> >> > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Hi Rodrigo, Restarting master services did not help either.. I tried afterwards again with grace period and force delete. Stil no luck. Bahho On 16 March 2018 at 21:29, Rodrigo Bersa <rbe...@redhat.com> wrote: > Bahhoo, > > I believe that the namespace will get stuck also. 'Cause it will only be > deleted after all of it's objects got deleted. > > I would try to restart the Masters services before. > > > Regards, > > > Rodrigo Bersa > > Cloud Consultant, RHCVA, RHCE > > Red Hat Brasil <https://www.redhat.com> > > rbe...@redhat.comM: +55-11-99557-5841 > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil > pelo *Great Place to Work*. > > On Fri, Mar 16, 2018 at 5:25 PM, Bahhoo <bah...@gmail.com> wrote: > >> Hi Rodrigo, >> >> No PVs are used. One of the pods is a build pod, the other one's a normal >> pod without storage. >> I'll try deleting the namespace. I didn't want to do that,since I had >> running pods in the namespace. >> >> Best, >> Bahho >> -- >> Kimden: Rodrigo Bersa <rbe...@redhat.com> >> Gönderme tarihi: 16.3.2018 16:12 >> Kime: Bahhoo <bah...@gmail.com> >> Bilgi: rahul334...@gmail.com; users <users@lists.openshift.redhat.com> >> >> Konu: Re: Pods stuck on Terminating status >> >> Hi Bahhoo, >> >> Are you using PVs on the "Terminating" POD? I heard about some issues >> with PODs bounded to PV/PVCs provided by dynamic storage, where you have to >> first remove the volume form POD, then the PVPVC. Just after that remove >> the POD or the DeplymentConfig. >> >> If it's not the case, maybe restarting the atomic-openshift-master-* >> services can work removing the inconsistent POD. >> >> >> Regards, >> >> >> Rodrigo Bersa >> >> Cloud Consultant, RHCVA, RHCE >> >> Red Hat Brasil <https://www.redhat.com> >> >> rbe...@redhat.comM: +55-11-99557-5841 >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil >> pelo *Great Place to Work*. >> >> On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: >> >>> Hi Rahul, >>> >>> That won't do it either. >>> >>> Thanks >>> Bahho >>> -- >>> Kimden: Rahul Agarwal <rahul334...@gmail.com> >>> Gönderme tarihi: 15.3.2018 22:26 >>> Kime: bahhooo <bah...@gmail.com> >>> Bilgi: users <users@lists.openshift.redhat.com> >>> Konu: Re: Pods stuck on Terminating status >>> >>> Hi Bahho >>> >>> Try: oc delete all -l app= >>> >>> Thanks, >>> Rahul >>> >>> On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I have some zombie pods stuck on Terminating status on a OCP 3.7 >>>> HA-cluster. >>>> >>>> oc delete with --grace-period=0 --force etc. won't work. >>>> Docker restart. server reboot won't help either. >>>> >>>> I tried to find the pod key in etcd either in order to delete it >>>> manually. I couldn't find it. >>>> >>>> Is there a way to delete these pods? >>>> >>>> >>>> >>>> >>>> Bahho >>>> >>>> ___ >>>> users mailing list >>>> users@lists.openshift.redhat.com >>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>> >>>> >>> >>> ___ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >>> >> > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Bahhoo, I believe that the namespace will get stuck also. 'Cause it will only be deleted after all of it's objects got deleted. I would try to restart the Masters services before. Regards, Rodrigo Bersa Cloud Consultant, RHCVA, RHCE Red Hat Brasil <https://www.redhat.com> rbe...@redhat.comM: +55-11-99557-5841 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo *Great Place to Work*. On Fri, Mar 16, 2018 at 5:25 PM, Bahhoo <bah...@gmail.com> wrote: > Hi Rodrigo, > > No PVs are used. One of the pods is a build pod, the other one's a normal > pod without storage. > I'll try deleting the namespace. I didn't want to do that,since I had > running pods in the namespace. > > Best, > Bahho > -- > Kimden: Rodrigo Bersa <rbe...@redhat.com> > Gönderme tarihi: 16.3.2018 16:12 > Kime: Bahhoo <bah...@gmail.com> > Bilgi: rahul334...@gmail.com; users <users@lists.openshift.redhat.com> > > Konu: Re: Pods stuck on Terminating status > > Hi Bahhoo, > > Are you using PVs on the "Terminating" POD? I heard about some issues with > PODs bounded to PV/PVCs provided by dynamic storage, where you have to > first remove the volume form POD, then the PVPVC. Just after that remove > the POD or the DeplymentConfig. > > If it's not the case, maybe restarting the atomic-openshift-master-* > services can work removing the inconsistent POD. > > > Regards, > > > Rodrigo Bersa > > Cloud Consultant, RHCVA, RHCE > > Red Hat Brasil <https://www.redhat.com> > > rbe...@redhat.comM: +55-11-99557-5841 > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil > pelo *Great Place to Work*. > > On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: > >> Hi Rahul, >> >> That won't do it either. >> >> Thanks >> Bahho >> -- >> Kimden: Rahul Agarwal <rahul334...@gmail.com> >> Gönderme tarihi: 15.3.2018 22:26 >> Kime: bahhooo <bah...@gmail.com> >> Bilgi: users <users@lists.openshift.redhat.com> >> Konu: Re: Pods stuck on Terminating status >> >> Hi Bahho >> >> Try: oc delete all -l app= >> >> Thanks, >> Rahul >> >> On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: >> >>> Hi all, >>> >>> I have some zombie pods stuck on Terminating status on a OCP 3.7 >>> HA-cluster. >>> >>> oc delete with --grace-period=0 --force etc. won't work. >>> Docker restart. server reboot won't help either. >>> >>> I tried to find the pod key in etcd either in order to delete it >>> manually. I couldn't find it. >>> >>> Is there a way to delete these pods? >>> >>> >>> >>> >>> Bahho >>> >>> ___ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >>> >> >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Hi Rodrigo, No PVs are used. One of the pods is a build pod, the other one's a normal pod without storage. I'll try deleting the namespace. I didn't want to do that,since I had running pods in the namespace. Best, Bahho -Orijinal ileti- Kimden: "Rodrigo Bersa" <rbe...@redhat.com> Gönderme tarihi: 16.3.2018 16:12 Kime: "Bahhoo" <bah...@gmail.com> Bilgi: "rahul334...@gmail.com" <rahul334...@gmail.com>; "users" <users@lists.openshift.redhat.com> Konu: Re: Pods stuck on Terminating status Hi Bahhoo, Are you using PVs on the "Terminating" POD? I heard about some issues with PODs bounded to PV/PVCs provided by dynamic storage, where you have to first remove the volume form POD, then the PVPVC. Just after that remove the POD or the DeplymentConfig. If it's not the case, maybe restarting the atomic-openshift-master-* services can work removing the inconsistent POD. Regards, Rodrigo Bersa Cloud Consultant, RHCVA, RHCE Red Hat Brasil rbe...@redhat.comM: +55-11-99557-5841 TRIED. TESTED. TRUSTED. Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo Great Place to Work. On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: Hi Rahul, That won't do it either. Thanks Bahho Kimden: Rahul Agarwal Gönderme tarihi: 15.3.2018 22:26 Kime: bahhooo Bilgi: users Konu: Re: Pods stuck on Terminating status Hi Bahho Try: oc delete all -l app= Thanks, Rahul On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: Hi all, I have some zombie pods stuck on Terminating status on a OCP 3.7 HA-cluster. oc delete with --grace-period=0 --force etc. won't work. Docker restart. server reboot won't help either. I tried to find the pod key in etcd either in order to delete it manually. I couldn't find it. Is there a way to delete these pods? Bahho ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Which PV types are in use here? --Humble On Fri, Mar 16, 2018 at 8:42 PM, Rodrigo Bersa <rbe...@redhat.com> wrote: > Hi Bahhoo, > > Are you using PVs on the "Terminating" POD? I heard about some issues with > PODs bounded to PV/PVCs provided by dynamic storage, where you have to > first remove the volume form POD, then the PVPVC. Just after that remove > the POD or the DeplymentConfig. > > If it's not the case, maybe restarting the atomic-openshift-master-* > services can work removing the inconsistent POD. > > > Regards, > > > Rodrigo Bersa > > Cloud Consultant, RHCVA, RHCE > > Red Hat Brasil <https://www.redhat.com> > > rbe...@redhat.comM: +55-11-99557-5841 > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil > pelo *Great Place to Work*. > > On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: > >> Hi Rahul, >> >> That won't do it either. >> >> Thanks >> Bahho >> -- >> Kimden: Rahul Agarwal <rahul334...@gmail.com> >> Gönderme tarihi: 15.3.2018 22:26 >> Kime: bahhooo <bah...@gmail.com> >> Bilgi: users <users@lists.openshift.redhat.com> >> Konu: Re: Pods stuck on Terminating status >> >> Hi Bahho >> >> Try: oc delete all -l app= >> >> Thanks, >> Rahul >> >> On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: >> >>> Hi all, >>> >>> I have some zombie pods stuck on Terminating status on a OCP 3.7 >>> HA-cluster. >>> >>> oc delete with --grace-period=0 --force etc. won't work. >>> Docker restart. server reboot won't help either. >>> >>> I tried to find the pod key in etcd either in order to delete it >>> manually. I couldn't find it. >>> >>> Is there a way to delete these pods? >>> >>> >>> >>> >>> Bahho >>> >>> ___ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >>> >> >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> > > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Hi Bahhoo, Are you using PVs on the "Terminating" POD? I heard about some issues with PODs bounded to PV/PVCs provided by dynamic storage, where you have to first remove the volume form POD, then the PVPVC. Just after that remove the POD or the DeplymentConfig. If it's not the case, maybe restarting the atomic-openshift-master-* services can work removing the inconsistent POD. Regards, Rodrigo Bersa Cloud Consultant, RHCVA, RHCE Red Hat Brasil <https://www.redhat.com> rbe...@redhat.comM: +55-11-99557-5841 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil pelo *Great Place to Work*. On Thu, Mar 15, 2018 at 7:28 PM, Bahhoo <bah...@gmail.com> wrote: > Hi Rahul, > > That won't do it either. > > Thanks > Bahho > -- > Kimden: Rahul Agarwal <rahul334...@gmail.com> > Gönderme tarihi: 15.3.2018 22:26 > Kime: bahhooo <bah...@gmail.com> > Bilgi: users <users@lists.openshift.redhat.com> > Konu: Re: Pods stuck on Terminating status > > Hi Bahho > > Try: oc delete all -l app= > > Thanks, > Rahul > > On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: > >> Hi all, >> >> I have some zombie pods stuck on Terminating status on a OCP 3.7 >> HA-cluster. >> >> oc delete with --grace-period=0 --force etc. won't work. >> Docker restart. server reboot won't help either. >> >> I tried to find the pod key in etcd either in order to delete it >> manually. I couldn't find it. >> >> Is there a way to delete these pods? >> >> >> >> >> Bahho >> >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> > > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
YNT: Pods stuck on Terminating status
Hi Rahul, That won't do it either. Thanks Bahho -Orijinal ileti- Kimden: "Rahul Agarwal" <rahul334...@gmail.com> Gönderme tarihi: 15.3.2018 22:26 Kime: "bahhooo" <bah...@gmail.com> Bilgi: "users" <users@lists.openshift.redhat.com> Konu: Re: Pods stuck on Terminating status Hi Bahho Try: oc delete all -l app= Thanks, Rahul On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: Hi all, I have some zombie pods stuck on Terminating status on a OCP 3.7 HA-cluster. oc delete with --grace-period=0 --force etc. won't work. Docker restart. server reboot won't help either. I tried to find the pod key in etcd either in order to delete it manually. I couldn't find it. Is there a way to delete these pods? Bahho ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Pods stuck on Terminating status
Hi Bahho Try: oc delete all -l app= Thanks, Rahul On Thu, Mar 15, 2018 at 5:19 PM, bahhooo <bah...@gmail.com> wrote: > Hi all, > > I have some zombie pods stuck on Terminating status on a OCP 3.7 > HA-cluster. > > oc delete with --grace-period=0 --force etc. won't work. > Docker restart. server reboot won't help either. > > I tried to find the pod key in etcd either in order to delete it manually. > I couldn't find it. > > Is there a way to delete these pods? > > > > > Bahho > > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Pods stuck on Terminating status
Hi all, I have some zombie pods stuck on Terminating status on a OCP 3.7 HA-cluster. oc delete with --grace-period=0 --force etc. won't work. Docker restart. server reboot won't help either. I tried to find the pod key in etcd either in order to delete it manually. I couldn't find it. Is there a way to delete these pods? Bahho ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
