Re: [ceph-users] Error Creating OSD

2018-04-16 Thread Alfredo Deza 
On Sat, Apr 14, 2018 at 5:17 PM, Rhian Resnick <rresn...@fau.edu> wrote:

> Afternoon,
>
>
> Happily, I resolved this issue.
>
>
> Running vgdisplay showed that ceph-volume tried to create a disk on failed
> disk. (We didn't know we had a bad did so this is information that was new
> to us) and when the command failed it left three bad volume groups. Since
> you cannot rename them you need to use the following command to delete
> them.
>

We actually tried to guard against this problem, but we failed in this
specific situation where multiple VGs exist. I've created
http://tracker.ceph.com/issues/23757 to follow up with this, thanks for
reporting it!


>
> vgdisplay to find the bad volume groups
>
> vgremove --select vg_uuid=your uuid -f # -f forces it to be removed
>
>
> Rhian Resnick
>
> Associate Director Middleware and HPC
>
> Office of Information Technology
>
>
> Florida Atlantic University
>
> 777 Glades Road
> <https://maps.google.com/?q=777+Glades+Road=gmail=g>, CM22,
> Rm 173B
>
> Boca Raton, FL 33431
>
> Phone 561.297.2647
>
> Fax 561.297.0222
>
>  [image: image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>
>
>
> --
> *From:* Rhian Resnick
> *Sent:* Saturday, April 14, 2018 12:47 PM
> *To:* Alfredo Deza
>
> *Cc:* ceph-users@lists.ceph.com
> *Subject:* Re: [ceph-users] Error Creating OSD
>
>
> Thanks all,
>
>
> Here is a link to our our command being executed: https://pastebin.
> com/iy8iSaKH
>
>
> Here are the results from the command
>
>
> Executed with debug enabled (after a zap with destroy)
>
>
> [root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data
> /dev/sdu
> Running command: ceph-authtool --gen-print-key
> Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring
> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> 664894a8-530a-4557-b2f4-1af5b391f2b7
> --> Was unable to complete a new OSD, will rollback changes
> --> OSD will be fully purged from the cluster, because the ID was generated
> Running command: ceph osd purge osd.140 --yes-i-really-mean-it
>  stderr: purged osd.140
> Traceback (most recent call last):
>   File "/sbin/ceph-volume", line 6, in 
> main.Volume()
>   File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37,
> in __init__
> self.main(self.argv)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
> 59, in newfunc
> return f(*a, **kw)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153,
> in main
> terminal.dispatch(self.mapper, subcommand_args)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> 182, in dispatch
> instance.main()
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py",
> line 38, in main
> terminal.dispatch(self.mapper, self.argv)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> 182, in dispatch
> instance.main()
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py",
> line 74, in main
> self.create(args)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
> 16, in is_root
> return func(*a, **kw)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py",
> line 26, in create
> prepare_step.safe_prepare(args)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py",
> line 217, in safe_prepare
> self.prepare(args)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
> 16, in is_root
> return func(*a, **kw)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py",
> line 283, in prepare
> block_lv = self.prepare_device(args.data, 'block', cluster_fsid,
> osd_fsid)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py",
> line 193, in prepare_device
> if api.get_vg(vg_name=vg_name):
>   File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line
> 334, in get_vg
> return vgs.get(vg_name=vg_name, vg_tags=vg_tags)
>   File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line
> 429, in get
> raise MultipleVGsError(vg_name)
> ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking
> for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc
>
>
>
> Rhian Resnick
>
> Associate Director Middleware and HPC
>
> Office of Informat

Re: [ceph-users] Error Creating OSD

2018-04-14 Thread Rhian Resnick 
Afternoon,


Happily, I resolved this issue.


Running vgdisplay showed that ceph-volume tried to create a disk on failed 
disk. (We didn't know we had a bad did so this is information that was new to 
us) and when the command failed it left three bad volume groups. Since you 
cannot rename them you need to use the following command to delete them.


vgdisplay to find the bad volume groups

vgremove --select vg_uuid=your uuid -f # -f forces it to be removed


Rhian Resnick

Associate Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>



From: Rhian Resnick
Sent: Saturday, April 14, 2018 12:47 PM
To: Alfredo Deza
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Error Creating OSD


Thanks all,


Here is a link to our our command being executed: https://pastebin.com/iy8iSaKH



Here are the results from the command


Executed with debug enabled (after a zap with destroy)


[root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
664894a8-530a-4557-b2f4-1af5b391f2b7
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.140 --yes-i-really-mean-it
 stderr: purged osd.140
Traceback (most recent call last):
  File "/sbin/ceph-volume", line 6, in 
main.Volume()
  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in 
__init__
self.main(self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, 
in newfunc
return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main
terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch
instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 
38, in main
terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch
instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", 
line 74, in main
self.create(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, 
in is_root
return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", 
line 26, in create
prepare_step.safe_prepare(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 217, in safe_prepare
self.prepare(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, 
in is_root
return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 283, in prepare
block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 193, in prepare_device
if api.get_vg(vg_name=vg_name):
  File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in 
get_vg
return vgs.get(vg_name=vg_name, vg_tags=vg_tags)
  File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in 
get
raise MultipleVGsError(vg_name)
ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for 
volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc




Rhian Resnick

Associate Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>


________
From: Alfredo Deza <ad...@redhat.com>
Sent: Saturday, April 14, 2018 8:45 AM
To: Rhian Resnick
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Error Creating OSD



On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick 
<rresn...@fau.edu<mailto:rresn...@fau.edu>> wrote:

Evening,

When attempting to create an OSD we receive the following error.

[ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data 
/dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
c8cb8cff-dad9-48b8-8d77-6f130a4b629d
--> Was unable to complete a new OSD, will rollback changes

Re: [ceph-users] Error Creating OSD

2018-04-14 Thread Rhian Resnick 
Thanks all,


Here is a link to our our command being executed: https://pastebin.com/iy8iSaKH



Here are the results from the command


Executed with debug enabled (after a zap with destroy)


[root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
664894a8-530a-4557-b2f4-1af5b391f2b7
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.140 --yes-i-really-mean-it
 stderr: purged osd.140
Traceback (most recent call last):
  File "/sbin/ceph-volume", line 6, in 
main.Volume()
  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in 
__init__
self.main(self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, 
in newfunc
return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main
terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch
instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 
38, in main
terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch
instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", 
line 74, in main
self.create(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, 
in is_root
return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", 
line 26, in create
prepare_step.safe_prepare(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 217, in safe_prepare
self.prepare(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, 
in is_root
return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 283, in prepare
block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", 
line 193, in prepare_device
if api.get_vg(vg_name=vg_name):
  File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in 
get_vg
return vgs.get(vg_name=vg_name, vg_tags=vg_tags)
  File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in 
get
raise MultipleVGsError(vg_name)
ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for 
volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc




Rhian Resnick

Associate Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>


____________
From: Alfredo Deza <ad...@redhat.com>
Sent: Saturday, April 14, 2018 8:45 AM
To: Rhian Resnick
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Error Creating OSD



On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick 
<rresn...@fau.edu<mailto:rresn...@fau.edu>> wrote:

Evening,

When attempting to create an OSD we receive the following error.

[ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data 
/dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
c8cb8cff-dad9-48b8-8d77-6f130a4b629d
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.140 --yes-i-really-mean-it
 stderr: purged osd.140
-->  MultipleVGsError: Got more than 1 result looking for volume group: 
ceph-6a2e8f21-bca2-492b-8869-eecc995216cc

Any hints on what to do? This occurs when we attempt to create osd's on this 
node.

Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? 
Also, if you could try the same command but with:

CEPH_VOLUME_DEBUG=1

I think you are hitting two issues here:

1) Somehow `osd new` is not completing and failing
2) The `purge` command to wipe out the LV is getting multiple LV's and cannot 
make sure to match the one it used.

#2 definitely looks like something we are doing wrong, and #1 can have a lot of 
different causes. The logs would be tremendously helpful!


Rhian Resnick

Associate Director Middleware and HPC

Office of Inf

Re: [ceph-users] Error Creating OSD

2018-04-14 Thread Alfredo Deza 
On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick  wrote:

> Evening,
>
> When attempting to create an OSD we receive the following error.
>
> [ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore
> --data /dev/sdu
> Running command: ceph-authtool --gen-print-key
> Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring
> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
> c8cb8cff-dad9-48b8-8d77-6f130a4b629d
> --> Was unable to complete a new OSD, will rollback changes
> --> OSD will be fully purged from the cluster, because the ID was generated
> Running command: ceph osd purge osd.140 --yes-i-really-mean-it
>  stderr: purged osd.140
> -->  MultipleVGsError: Got more than 1 result looking for volume group:
> ceph-6a2e8f21-bca2-492b-8869-eecc995216cc
>
> Any hints on what to do? This occurs when we attempt to create osd's on
> this node.
>

Can you use a paste site and get the /var/log/ceph/ceph-volume.log
contents? Also, if you could try the same command but with:

CEPH_VOLUME_DEBUG=1

I think you are hitting two issues here:

1) Somehow `osd new` is not completing and failing
2) The `purge` command to wipe out the LV is getting multiple LV's and
cannot make sure to match the one it used.

#2 definitely looks like something we are doing wrong, and #1 can have a
lot of different causes. The logs would be tremendously helpful!

>
> Rhian Resnick
>
> Associate Director Middleware and HPC
>
> Office of Information Technology
>
>
> Florida Atlantic University
>
> 777 Glades Road, CM22, Rm 173B
>
> Boca Raton, FL 33431
>
> Phone 561.297.2647
>
> Fax 561.297.0222
>
>  [image: image] 
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error Creating OSD

2018-04-13 Thread Rhian Resnick 
Evening,

When attempting to create an OSD we receive the following error.

[ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data 
/dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
c8cb8cff-dad9-48b8-8d77-6f130a4b629d
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.140 --yes-i-really-mean-it
 stderr: purged osd.140
-->  MultipleVGsError: Got more than 1 result looking for volume group: 
ceph-6a2e8f21-bca2-492b-8869-eecc995216cc

Any hints on what to do? This occurs when we attempt to create osd's on this 
node.


Rhian Resnick

Associate Director Middleware and HPC

Office of Information Technology


Florida Atlantic University

777 Glades Road, CM22, Rm 173B

Boca Raton, FL 33431

Phone 561.297.2647

Fax 561.297.0222

 [image] 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com