Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...

2016-08-05 Thread Ilya Dryomov 
On Thu, Aug 4, 2016 at 10:44 PM, K.C. Wong  wrote:
> Thank you, Jason.
>
> While I can't find the culprit for the watcher (the watcher never expired,
> and survived a reboot. udev, maybe?), blacklisting the host did allow me
> to remove the device.

It survived a reboot because watch state is persisted on the OSDs.
This sounds like an OSD bug where the watcher wasn't cleaned up or
timed out after 30 seconds.

Was that image unmapped gracefully?

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...

2016-08-04 Thread K.C. Wong 
Thank you, Jason.

While I can't find the culprit for the watcher (the watcher never expired,
and survived a reboot. udev, maybe?), blacklisting the host did allow me
to remove the device.

Much appreciated,

-kc

> On Aug 4, 2016, at 4:50 AM, Jason Dillaman  wrote:
> 
> If the client is no longer running the watch should expire within 30
> seconds. If you are still experiencing this issue, you can blacklist
> the mystery client via "ceph osd blacklist add".
> 
> On Wed, Aug 3, 2016 at 6:06 PM, K.C. Wong  wrote:
>> I'm having a hard time removing an RBD that I no longer need.
>> 
>> # rbd rm /
>> 2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not 
>> removing
>> Removing image: 0% complete...failed.
>> rbd: error: image still has watchers
>> This means the image is still open or the client using it crashed. Try again 
>> after closing/unmapping it or waiting 30s for the crashed client to timeout.
>> 
>> So, I use `rbd status` to identify the watcher:
>> 
>> # rbd status /
>> Watchers:
>>watcher=:0/705293879 client.1076985 cookie=1
>> 
>> I log onto that host, and did
>> 
>> # rbd showmapped
>> 
>> which returns nothing
>> 
>> I don't use snapshot and I don't use cloning, so, there shouldn't
>> be any image sharing. I ended up rebooting that host and the
>> watcher is still around, and my problem persist: I can't remove
>> the RBD.
>> 
>> At this point, I'm all out of ideas on how to troubleshoot this
>> problem. I'm running infernalis:
>> 
>> # ceph --version
>> ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>> 
>> in my set up, on CentOS 7.2 hosts
>> 
>> # uname -r
>> 3.10.0-327.22.2.el7.x86_64
>> 
>> I appreciate any assistance,
>> 
>> -kc
>> 
>> K.C. Wong
>> kcw...@verseon.com
>> 4096R/B8995EDE  E527 CBE8 023E 79EA 8BBB  5C77 23A6 92E9 B899 5EDE
>> hkps://hkps.pool.sks-keyservers.net
>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
> 
> 
> 
> --
> Jason

K.C. Wong
kcw...@verseon.com
4096R/B8995EDE  E527 CBE8 023E 79EA 8BBB  5C77 23A6 92E9 B899 5EDE
hkps://hkps.pool.sks-keyservers.net



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...

2016-08-04 Thread Jason Dillaman 
If the client is no longer running the watch should expire within 30
seconds. If you are still experiencing this issue, you can blacklist
the mystery client via "ceph osd blacklist add".

On Wed, Aug 3, 2016 at 6:06 PM, K.C. Wong  wrote:
> I'm having a hard time removing an RBD that I no longer need.
>
> # rbd rm /
> 2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not 
> removing
> Removing image: 0% complete...failed.
> rbd: error: image still has watchers
> This means the image is still open or the client using it crashed. Try again 
> after closing/unmapping it or waiting 30s for the crashed client to timeout.
>
> So, I use `rbd status` to identify the watcher:
>
> # rbd status /
> Watchers:
> watcher=:0/705293879 client.1076985 cookie=1
>
> I log onto that host, and did
>
> # rbd showmapped
>
> which returns nothing
>
> I don't use snapshot and I don't use cloning, so, there shouldn't
> be any image sharing. I ended up rebooting that host and the
> watcher is still around, and my problem persist: I can't remove
> the RBD.
>
> At this point, I'm all out of ideas on how to troubleshoot this
> problem. I'm running infernalis:
>
> # ceph --version
> ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>
> in my set up, on CentOS 7.2 hosts
>
> # uname -r
> 3.10.0-327.22.2.el7.x86_64
>
> I appreciate any assistance,
>
> -kc
>
> K.C. Wong
> kcw...@verseon.com
> 4096R/B8995EDE  E527 CBE8 023E 79EA 8BBB  5C77 23A6 92E9 B899 5EDE
> hkps://hkps.pool.sks-keyservers.net
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [Troubleshooting] I have a watcher I can't get rid of...

2016-08-03 Thread K.C. Wong 
I'm having a hard time removing an RBD that I no longer need.

# rbd rm /
2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not 
removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again 
after closing/unmapping it or waiting 30s for the crashed client to timeout.

So, I use `rbd status` to identify the watcher:

# rbd status /
Watchers:
watcher=:0/705293879 client.1076985 cookie=1

I log onto that host, and did

# rbd showmapped

which returns nothing

I don't use snapshot and I don't use cloning, so, there shouldn't
be any image sharing. I ended up rebooting that host and the
watcher is still around, and my problem persist: I can't remove
the RBD.

At this point, I'm all out of ideas on how to troubleshoot this
problem. I'm running infernalis:

# ceph --version
ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)

in my set up, on CentOS 7.2 hosts

# uname -r
3.10.0-327.22.2.el7.x86_64

I appreciate any assistance,

-kc

K.C. Wong
kcw...@verseon.com
4096R/B8995EDE  E527 CBE8 023E 79EA 8BBB  5C77 23A6 92E9 B899 5EDE
hkps://hkps.pool.sks-keyservers.net



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com