Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-20 Thread Pelletier, Robert 
Thank you gentlemen. I will give this a shot and reply with what worked.

On Jul 19, 2019, at 11:11 AM, Tarek Zegar 
mailto:tze...@us.ibm.com>> wrote:


On the host with the osd run:


 ceph-volume lvm list





"☣Adam" ---07/18/2019 03:25:05 PM---The block device can be found 
in /var/lib/ceph/osd/ceph-$ID/block # ls -l /var/lib/ceph/osd/ceph-9/b

From: "☣Adam" mailto:a...@dc949.org>>
To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
Date: 07/18/2019 03:25 PM
Subject: [EXTERNAL] Re: [ceph-users] Need to replace OSD. How do I find 
physical disk
Sent by: "ceph-users" 
mailto:ceph-users-boun...@lists.ceph.com>>





The block device can be found in /var/lib/ceph/osd/ceph-$ID/block
# ls -l /var/lib/ceph/osd/ceph-9/block

In my case it links to /dev/sdbvg/sdb which makes is pretty obvious
which drive this is, but the Volume Group and Logical volume could be
named anything.  To see what physical disk(s) make up this volume group
use lvblk (as Reed suggested)
# lvblk

If that drive needs to be located in a computer with many drives,
smartctl should be able to be used to pull the make, model, and serial
number
# smartctl -i /dev/sdb


I was not aware of ceph-volume, or `ceph-disk list` (which is apparently
now deprecated in favor of ceph-volume), so thank you to all in this
thread for teaching about alternative (arguably more proper) ways of
doing this. :-)

On 7/18/19 12:58 PM, Pelletier, Robert wrote:
> How do I find the physical disk in a Ceph luminous cluster in order to
> replace it. Osd.9 is down in my cluster which resides on ceph-osd1 host.
>
>
>
> If I run lsblk -io KNAME,TYPE,SIZE,MODEL,SERIAL I can get the serial
> numbers of all the physical disks for example
>
> sdbdisk  1.8T ST2000DM001-1CH1 Z1E5VLRG
>
>
>
> But how do I find out which osd is mapped to sdb and so on?
>
> When I run df –h I get this
>
>
>
> [root@ceph-osd1 ~]# df -h
>
> Filesystem   Size  Used Avail Use% Mounted on
>
> /dev/mapper/ceph--osd1-root   19G  1.9G   17G  10% /
>
> devtmpfs  48G 0   48G   0% /dev
>
> tmpfs 48G 0   48G   0% /dev/shm
>
> tmpfs 48G  9.3M   48G   1% /run
>
> tmpfs 48G 0   48G   0% /sys/fs/cgroup
>
> /dev/sda3947M  232M  716M  25% /boot
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-2
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-5
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-0
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-8
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-7
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-33
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-10
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-1
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-38
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-4
>
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-6
>
> tmpfs9.5G 0  9.5G   0% /run/user/0
>
>
>
>
>
> *Robert Pelletier, **IT and Security Specialist***
>
> Eastern Maine Community College
> (207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-19 Thread Tarek Zegar 
On the host with the osd run:


 ceph-volume lvm list







From:   "☣Adam" 
To: ceph-users@lists.ceph.com
Date:   07/18/2019 03:25 PM
Subject:[EXTERNAL] Re: [ceph-users] Need to replace OSD. How do I find
    physical disk
Sent by:"ceph-users" 



The block device can be found in /var/lib/ceph/osd/ceph-$ID/block
# ls -l /var/lib/ceph/osd/ceph-9/block

In my case it links to /dev/sdbvg/sdb which makes is pretty obvious
which drive this is, but the Volume Group and Logical volume could be
named anything.  To see what physical disk(s) make up this volume group
use lvblk (as Reed suggested)
# lvblk

If that drive needs to be located in a computer with many drives,
smartctl should be able to be used to pull the make, model, and serial
number
# smartctl -i /dev/sdb


I was not aware of ceph-volume, or `ceph-disk list` (which is apparently
now deprecated in favor of ceph-volume), so thank you to all in this
thread for teaching about alternative (arguably more proper) ways of
doing this. :-)

On 7/18/19 12:58 PM, Pelletier, Robert wrote:
> How do I find the physical disk in a Ceph luminous cluster in order to
> replace it. Osd.9 is down in my cluster which resides on ceph-osd1 host.
>
>
>
> If I run lsblk -io KNAME,TYPE,SIZE,MODEL,SERIAL I can get the serial
> numbers of all the physical disks for example
>
> sdb    disk  1.8T ST2000DM001-1CH1 Z1E5VLRG
>
>
>
> But how do I find out which osd is mapped to sdb and so on?
>
> When I run df –h I get this
>
>
>
> [root@ceph-osd1 ~]# df -h
>
> Filesystem   Size  Used Avail Use% Mounted on
>
> /dev/mapper/ceph--osd1-root   19G  1.9G   17G  10% /
>
> devtmpfs  48G 0   48G   0% /dev
>
> tmpfs 48G 0   48G   0% /dev/shm
>
> tmpfs 48G  9.3M   48G   1% /run
>
> tmpfs 48G 0   48G   0% /sys/fs/cgroup
>
> /dev/sda3    947M  232M  716M  25% /boot
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-2
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-5
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-0
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-8
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-7
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-33
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-10
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-1
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-38
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-4
>
> tmpfs 48G   24K   48G
1% /var/lib/ceph/osd/ceph-6
>
> tmpfs    9.5G 0  9.5G   0% /run/user/0
>
>
>
>
>
> *Robert Pelletier, **IT and Security Specialist***
>
> Eastern Maine Community College
> (207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com=DwIF-g=jf_iaSHvJObTbx-siA1ZOg=3V1n-r1W__Mu-wEAwzq7jDpopOSMrfRfomn1f5bgT28=TXW65vJi4jZZ8MBtN2cjvq0bG2nV1-y_EM43NonJWFs=a-SpJzGVKsv4FRPY4Q84J3RrM3-FRsTVJz3f0825pOc=

>
___
ceph-users mailing list
ceph-users@lists.ceph.com
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dusers-2Dceph.com=DwIF-g=jf_iaSHvJObTbx-siA1ZOg=3V1n-r1W__Mu-wEAwzq7jDpopOSMrfRfomn1f5bgT28=TXW65vJi4jZZ8MBtN2cjvq0bG2nV1-y_EM43NonJWFs=a-SpJzGVKsv4FRPY4Q84J3RrM3-FRsTVJz3f0825pOc=




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-18 Thread ☣Adam 
The block device can be found in /var/lib/ceph/osd/ceph-$ID/block
# ls -l /var/lib/ceph/osd/ceph-9/block

In my case it links to /dev/sdbvg/sdb which makes is pretty obvious
which drive this is, but the Volume Group and Logical volume could be
named anything.  To see what physical disk(s) make up this volume group
use lvblk (as Reed suggested)
# lvblk

If that drive needs to be located in a computer with many drives,
smartctl should be able to be used to pull the make, model, and serial
number
# smartctl -i /dev/sdb


I was not aware of ceph-volume, or `ceph-disk list` (which is apparently
now deprecated in favor of ceph-volume), so thank you to all in this
thread for teaching about alternative (arguably more proper) ways of
doing this. :-)

On 7/18/19 12:58 PM, Pelletier, Robert wrote:
> How do I find the physical disk in a Ceph luminous cluster in order to
> replace it. Osd.9 is down in my cluster which resides on ceph-osd1 host.
> 
>  
> 
> If I run lsblk -io KNAME,TYPE,SIZE,MODEL,SERIAL I can get the serial
> numbers of all the physical disks for example
> 
> sdb    disk  1.8T ST2000DM001-1CH1 Z1E5VLRG
> 
>  
> 
> But how do I find out which osd is mapped to sdb and so on?
> 
> When I run df –h I get this
> 
>  
> 
> [root@ceph-osd1 ~]# df -h
> 
> Filesystem   Size  Used Avail Use% Mounted on
> 
> /dev/mapper/ceph--osd1-root   19G  1.9G   17G  10% /
> 
> devtmpfs  48G 0   48G   0% /dev
> 
> tmpfs 48G 0   48G   0% /dev/shm
> 
> tmpfs 48G  9.3M   48G   1% /run
> 
> tmpfs 48G 0   48G   0% /sys/fs/cgroup
> 
> /dev/sda3    947M  232M  716M  25% /boot
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-2
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-5
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-0
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-8
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-7
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-33
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-10
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-1
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-38
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-4
> 
> tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-6
> 
> tmpfs    9.5G 0  9.5G   0% /run/user/0
> 
>  
> 
>  
> 
> *Robert Pelletier, **IT and Security Specialist***
> 
> Eastern Maine Community College
> (207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
> 
>  
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-18 Thread Reed Dier 
You can use ceph-volume to get the LV ID

> # ceph-volume lvm list
> 
> == osd.24 ==
> 
>   [block]
> /dev/ceph-edeb727e-c6d3-4347-bfbb-b9ce7f60514b/osd-block-1da5910e-136a-48a7-8cf1-1c265b7b612a
> 
>   type  block
>   osd id24
>   osd fsid  1da5910e-136a-48a7-8cf1-1c265b7b612a
>   db device /dev/nvme0n1p4
>   db uuid   c4939e17-c787-4630-9ec7-b44565ecf845
>   block uuidn8mCnv-PW4n-43R6-I4uN-P1E0-7qDh-I5dslh
>   block device  
> /dev/ceph-edeb727e-c6d3-4347-bfbb-b9ce7f60514b/osd-block-1da5910e-136a-48a7-8cf1-1c265b7b612a
>   devices   /dev/sda
> 
>   [  db]/dev/nvme0n1p4
> 
>   PARTUUID  c4939e17-c787-4630-9ec7-b44565ecf845

And you can then match this against lsblk which should give you the LV

> $ lsblk -a
> NAME  
> MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda   
>   8:00   1.8T  0 disk
> └─ceph--edeb727e--c6d3--4347--bfbb--b9ce7f60514b-osd--block--1da5910e--136a--48a7--8cf1--1c265b7b612a
>  253:60   1.8T  0 lvm
> nvme0n1   
> 259:00 372.6G  0 disk
> ├─nvme0n1p4   
> 259:40  14.9G  0 part

And if the device has just dropped off, which I have seen before, you should be 
able to see that in dmesg

> [Sat May 11 22:56:27 2019] sd 1:0:17:0: attempting task abort! 
> scmd(2d043ad6)
> [Sat May 11 22:56:27 2019] sd 1:0:17:0: [sdr] tag#0 CDB: Inquiry 12 00 00 00 
> 24 00
> [Sat May 11 22:56:27 2019] scsi target1:0:17: handle(0x001b), 
> sas_address(0x500304801f12eca1), phy(33)
> [Sat May 11 22:56:27 2019] scsi target1:0:17: enclosure logical 
> id(0x500304801f12ecbf), slot(17)
> [Sat May 11 22:56:27 2019] scsi target1:0:17: enclosure level(0x), 
> connector name( )
> [Sat May 11 22:56:28 2019] sd 1:0:17:0: device_block, handle(0x001b)
> [Sat May 11 22:56:30 2019] sd 1:0:17:0: device_unblock and setting to 
> running, handle(0x001b)
> [Sat May 11 22:56:30 2019] sd 1:0:17:0: [sdr] Synchronizing SCSI cache
> [Sat May 11 22:56:30 2019] sd 1:0:17:0: [sdr] Synchronize Cache(10) failed: 
> Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> [Sat May 11 22:56:31 2019] scsi 1:0:17:0: task abort: SUCCESS 
> scmd(2d043ad6)
> [Sat May 11 22:56:31 2019] mpt3sas_cm0: removing handle(0x001b), 
> sas_addr(0x500304801f12eca1)
> [Sat May 11 22:56:31 2019] mpt3sas_cm0: enclosure logical 
> id(0x500304801f12ecbf), slot(17)
> [Sat May 11 22:56:31 2019] mpt3sas_cm0: enclosure level(0x), connector 
> name( )
> [Sat May 11 23:00:57 2019] Buffer I/O error on dev dm-20, logical block 
> 488378352, async page read
> [Sat May 11 23:00:57 2019] Buffer I/O error on dev dm-20, logical block 1, 
> async page read
> [Sat May 11 23:00:58 2019] Buffer I/O error on dev dm-20, logical block 
> 488378352, async page read
> [Sat May 11 23:00:58 2019] Buffer I/O error on dev dm-20, logical block 1, 
> async page read
> 
> # smartctl -a /dev/sdr
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-46-generic] (local build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
> 
> Smartctl open device: /dev/sdr failed: No such device
> Hopefully that helps.

Reed

> On Jul 18, 2019, at 1:11 PM, Paul Emmerich  wrote:
> 
> 
> 
> On Thu, Jul 18, 2019 at 8:10 PM John Petrini  > wrote:
> Try ceph-disk list
> 
> no, this system is running ceph-volume not ceph-disk because the mountpoints 
> are in tmpfs
> 
> ceph-volume lvm list
> 
> But it looks like the disk is just completely broken and disappeared from the 
> system.
> 
> 
> -- 
> Paul Emmerich
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io 
> 
> 
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io 
> Tel: +49 89 1896585 90
>  
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-18 Thread Paul Emmerich 
On Thu, Jul 18, 2019 at 8:10 PM John Petrini  wrote:

> Try ceph-disk list
>

no, this system is running ceph-volume not ceph-disk because the
mountpoints are in tmpfs

ceph-volume lvm list

But it looks like the disk is just completely broken and disappeared from
the system.


-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Need to replace OSD. How do I find physical disk

2019-07-18 Thread John Petrini 
Try ceph-disk list
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Need to replace OSD. How do I find physical disk

2019-07-18 Thread Pelletier, Robert 
How do I find the physical disk in a Ceph luminous cluster in order to replace 
it. Osd.9 is down in my cluster which resides on ceph-osd1 host.

If I run lsblk -io KNAME,TYPE,SIZE,MODEL,SERIAL I can get the serial numbers of 
all the physical disks for example
sdbdisk  1.8T ST2000DM001-1CH1 Z1E5VLRG

But how do I find out which osd is mapped to sdb and so on?
When I run df -h I get this

[root@ceph-osd1 ~]# df -h
Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/ceph--osd1-root   19G  1.9G   17G  10% /
devtmpfs  48G 0   48G   0% /dev
tmpfs 48G 0   48G   0% /dev/shm
tmpfs 48G  9.3M   48G   1% /run
tmpfs 48G 0   48G   0% /sys/fs/cgroup
/dev/sda3947M  232M  716M  25% /boot
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-2
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-5
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-0
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-8
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-7
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-33
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-10
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-1
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-38
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-4
tmpfs 48G   24K   48G   1% /var/lib/ceph/osd/ceph-6
tmpfs9.5G 0  9.5G   0% /run/user/0


Robert Pelletier, IT and Security Specialist
Eastern Maine Community College
(207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com