On Thu, Jul 25, 2019 at 7:00 PM Bob R <[email protected]> wrote: > I would try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume simple scan' > again. We had some problems upgrading due to OSDs (perhaps initially > installed as firefly?) missing the 'type' attribute and iirc the > 'ceph-volume simple scan' command refused to overwrite existing json files > after I made some changes to ceph-volume. >
Ooof. I could swear that this issue was fixed already and it took me a while to find out that it wasn't at all. We saw this a few months ago in our Long Running Cluster used for dogfooding. I've created a ticket to track this work at http://tracker.ceph.com/issues/40987 But what you've done is exactly why we chose to persist the JSON files in /etc/ceph/osd/*.json, so that an admin could tell if anything is missing (or incorrect like in this case) and make the changes needed. > Bob > > On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza <[email protected]> wrote: > >> >> >> On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch <[email protected]> >> wrote: >> >>> Hi, >>> >>> >>> >>> I appreciate the insistency that the directions be followed. I wholly >>> agree. The only liberty I took was to do a ‘yum update’ instead of just >>> ‘yum update ceph-osd’ and then reboot. (Also my MDS runs on the MON hosts, >>> so it got update a step early.) >>> >>> >>> >>> As for the logs: >>> >>> >>> >>> [2019-07-24 15:07:22,713][ceph_volume.main][INFO ] Running command: >>> ceph-volume simple scan >>> >>> [2019-07-24 15:07:22,714][ceph_volume.process][INFO ] Running command: >>> /bin/systemctl show --no-pager --property=Id --state=running ceph-osd@* >>> >>> [2019-07-24 15:07:27,574][ceph_volume.main][INFO ] Running command: >>> ceph-volume simple activate --all >>> >>> [2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO ] >>> activating OSD specified in >>> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json >>> >>> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] >>> Required devices (block and data) not present for bluestore >>> >>> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] >>> bluestore devices found: [u'data'] >>> >>> [2019-07-24 15:07:27,576][ceph_volume][ERROR ] exception caught by >>> decorator >>> >>> Traceback (most recent call last): >>> >>> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", >>> line 59, in newfunc >>> >>> return f(*a, **kw) >>> >>> File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, >>> in main >>> >>> terminal.dispatch(self.mapper, subcommand_args) >>> >>> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line >>> 182, in dispatch >>> >>> instance.main() >>> >>> File >>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line >>> 33, in main >>> >>> terminal.dispatch(self.mapper, self.argv) >>> >>> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line >>> 182, in dispatch >>> >>> instance.main() >>> >>> File >>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", >>> line 272, in main >>> >>> self.activate(args) >>> >>> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", >>> line 16, in is_root >>> >>> return func(*a, **kw) >>> >>> File >>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", >>> line 131, in activate >>> >>> self.validate_devices(osd_metadata) >>> >>> File >>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", >>> line 62, in validate_devices >>> >>> raise RuntimeError('Unable to activate bluestore OSD due to missing >>> devices') >>> >>> RuntimeError: Unable to activate bluestore OSD due to missing devices >>> >>> >>> >>> (this is repeated for each of the 16 drives) >>> >>> >>> >>> Any other thoughts? (I’ll delete/create the OSDs with ceph-deply >>> otherwise.) >>> >> >> Try using `ceph-volume simple scan --stdout` so that it doesn't persist >> data onto /etc/ceph/osd/ and inspect that the JSON produced is capturing >> all the necessary details for OSDs. >> >> Alternatively, I would look into the JSON files already produced in >> /etc/ceph/osd/ and check if the details are correct. The `scan` sub-command >> does a tremendous effort to cover all cases where ceph-disk >> created an OSD (filestore, bluestore, dmcrypt, etc...) but it is possible >> that it may be hitting a problem. This is why the tool made these JSON >> files available, so that they could be inspected and corrected if anything. >> >> The details of the scan sub-command can be found at >> http://docs.ceph.com/docs/master/ceph-volume/simple/scan/ and the JSON >> structure is described in detail below at >> http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents >> >> In this particular case the tool is refusing to activate what seems to be >> a bluestore OSD. Is it really a bluestore OSD? if so, then it can't find >> where is the data partition. What does that partition look like (for any of >> the failing OSDs) ? Does it use dmcrypt, how was it created? (hopefully >> with ceph-disk!) >> >> If you know the data partition for a given OSD, try and pass it onto >> 'scan'. For example if it is /dev/sda1 you could do `ceph-volume simple >> scan /dev/sda1` and check its output. >> >> >> >>> >>> peter >>> >>> >>> >>> >>> Peter Eisch >>> Senior Site Reliability Engineer >>> T *1.612.659.3228* <1.612.659.3228> >>> [image: Facebook] <https://www.facebook.com/VirginPulse> >>> [image: LinkedIn] <https://www.linkedin.com/company/virgin-pulse> >>> [image: Twitter] <https://twitter.com/virginpulse> >>> *virginpulse.com* <https://www.virginpulse.com/> >>> | *virginpulse.com/global-challenge* >>> <https://www.virginpulse.com/en-gb/global-challenge/> >>> >>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | >>> Switzerland | United Kingdom | USA >>> Confidentiality Notice: The information contained in this e-mail, >>> including any attachment(s), is intended solely for use by the designated >>> recipient(s). Unauthorized use, dissemination, distribution, or >>> reproduction of this message by anyone other than the intended >>> recipient(s), or a person designated as responsible for delivering such >>> messages to the intended recipient, is strictly prohibited and may be >>> unlawful. This e-mail may contain proprietary, confidential or privileged >>> information. Any views or opinions expressed are solely those of the author >>> and do not necessarily represent those of Virgin Pulse, Inc. If you have >>> received this message in error, or are not the named recipient(s), please >>> immediately notify the sender and delete this e-mail message. >>> v2.59 >>> >>> *From: *Alfredo Deza <[email protected]> >>> *Date: *Wednesday, July 24, 2019 at 3:02 PM >>> *To: *Peter Eisch <[email protected]> >>> *Cc: *Paul Emmerich <[email protected]>, "[email protected]" >>> <[email protected]> >>> *Subject: *Re: [ceph-users] Upgrading and lost OSDs >>> >>> >>> >>> >>> >>> On Wed, Jul 24, 2019 at 3:49 PM Peter Eisch <[email protected]> >>> wrote: >>> >>> >>> >>> I’m at step 6. I updated/rebooted the host to complete “installing the >>> new packages and restarting the ceph-osd daemon” on the first OSD host. >>> All the systemctl definitions to start the OSDs were deleted, all the >>> properties in /var/lib/ceph/osd/ceph-* directories were deleted. All the >>> files in /var/lib/ceph/osd-lockbox, for comparison, were untouched and >>> still present. >>> >>> >>> >>> Peeking into step 7 I can run ceph-volume: >>> >>> >>> >>> # ceph-volume simple scan /dev/sda1 >>> >>> Running command: /usr/sbin/cryptsetup status /dev/sda1 >>> >>> Running command: /usr/sbin/cryptsetup status >>> 93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> Running command: /bin/mount -v /dev/sda5 /tmp/tmpF5F8t2 >>> >>> stdout: mount: /dev/sda5 mounted on /tmp/tmpF5F8t2. >>> >>> Running command: /usr/sbin/cryptsetup status /dev/sda5 >>> >>> Running command: /bin/ceph --cluster ceph --name >>> client.osd-lockbox.93fb5f2f-0273-4c87-a718-886d7e6db983 --keyring >>> /tmp/tmpF5F8t2/keyring config-key get >>> dm-crypt/osd/93fb5f2f-0273-4c87-a718-886d7e6db983/luks >>> >>> Running command: /bin/umount -v /tmp/tmpF5F8t2 >>> >>> stderr: umount: /tmp/tmpF5F8t2 (/dev/sda5) unmounted >>> >>> Running command: /usr/sbin/cryptsetup --key-file - --allow-discards >>> luksOpen /dev/sda1 93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> Running command: /bin/mount -v >>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 /tmp/tmpYK0WEV >>> >>> stdout: mount: /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 mounted >>> on /tmp/tmpYK0WEV. >>> >>> --> broken symlink found /tmp/tmpYK0WEV/block -> >>> /dev/mapper/a05b447c-c901-4690-a249-cc1a2d62a110 >>> >>> Running command: /usr/sbin/cryptsetup status /tmp/tmpYK0WEV/block_dmcrypt >>> >>> Running command: /usr/sbin/cryptsetup status >>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> Running command: /bin/umount -v /tmp/tmpYK0WEV >>> >>> stderr: umount: /tmp/tmpYK0WEV >>> (/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983) unmounted >>> >>> Running command: /usr/sbin/cryptsetup remove >>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> --> OSD 0 got scanned and metadata persisted to file: >>> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json >>> >>> --> To take over management of this scanned OSD, and disable ceph-disk >>> and udev, run: >>> >>> --> ceph-volume simple activate 0 >>> 93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> # >>> >>> # >>> >>> # ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983 >>> >>> --> Required devices (block and data) not present for bluestore >>> >>> --> bluestore devices found: [u'data'] >>> >>> --> RuntimeError: Unable to activate bluestore OSD due to missing >>> devices >>> >>> # >>> >>> >>> >>> The tool detected bluestore, or rather, it failed to find a journal >>> associated with /dev/sda1. Scanning a single partition can cause that. >>> There is a flag to spit out the findings to STDOUT instead of persisting >>> them in /etc/ceph/osd/ >>> >>> >>> >>> Since this is a "whole system" upgrade, then the upgrade documentation >>> instructions need to be followed: >>> >>> >>> >>> ceph-volume simple scan >>> ceph-volume simple activate --all >>> >>> >>> >>> If the `scan` command doesn't display any information (not even with the >>> --stdout flag) then the logs at /var/log/ceph/ceph-volume.log need to be >>> inspected. It would be useful to check any findings in there >>> >>> >>> >>> >>> Okay, this created /etc/ceph/osd/*.json. This is cool. Is there a >>> command or option which will read these files and mount the devices? >>> >>> >>> >>> peter >>> >>> >>> >>> >>> >>> >>> >>> *Peter Eisch* >>> >>> Senior Site Reliability Engineer >>> >>> *T* >>> >>> 1.612.659.3228 >>> >>> [image: Facebook] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227707952&sdata=B2JiNp12z7gsfF2i5T2l%2FSjfg6Fhg8E85OpdyGpEMHg%3D&reserved=0> >>> >>> [image: LinkedIn] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=z3Ii%2BGgPKe7fCOhNGXw%2BlD9j28YCY4gH81is%2BJoiSJU%3D&reserved=0> >>> >>> [image: Twitter] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=zUaGW2Fm16sdyJdHUPtDN6CzaMXtxMOHvmNDi9VshCw%3D&reserved=0> >>> >>> virginpulse.com >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=n3m9S%2Bt8fYzGY%2BqPw2wT433TQhf2oPXp9wAum9s9%2BUk%3D&reserved=0> >>> >>> | >>> >>> virginpulse.com/global-challenge >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=5wn%2F%2FkY1IL0d4BxXNpqCJUHG09gUFRTr2S9KWv1mVG4%3D&reserved=0> >>> >>> >>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | >>> Switzerland | United Kingdom | USA >>> >>> Confidentiality Notice: The information contained in this e-mail, >>> including any attachment(s), is intended solely for use by the designated >>> recipient(s). Unauthorized use, dissemination, distribution, or >>> reproduction of this message by anyone other than the intended >>> recipient(s), or a person designated as responsible for delivering such >>> messages to the intended recipient, is strictly prohibited and may be >>> unlawful. This e-mail may contain proprietary, confidential or privileged >>> information. Any views or opinions expressed are solely those of the author >>> and do not necessarily represent those of Virgin Pulse, Inc. If you have >>> received this message in error, or are not the named recipient(s), please >>> immediately notify the sender and delete this e-mail message. >>> >>> v2.59 >>> >>> *From: *Alfredo Deza <[email protected]> >>> *Date: *Wednesday, July 24, 2019 at 2:20 PM >>> *To: *Peter Eisch <[email protected]> >>> *Cc: *Paul Emmerich <[email protected]>, "[email protected]" >>> <[email protected]> >>> *Subject: *Re: [ceph-users] Upgrading and lost OSDs >>> >>> >>> >>> On Wed, Jul 24, 2019 at 2:56 PM Peter Eisch <[email protected]> >>> wrote: >>> >>> Hi Paul, >>> >>> To do better to answer you question, I'm following: >>> http://docs.ceph.com/docs/nautilus/releases/nautilus/ >>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fnautilus%2Freleases%2Fnautilus%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=D1Q0Hrg9mq2tTtfTt1kp4Ts3vBc7mertKvYy8vBWNF8%3D&reserved=0> >>> >>> At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum >>> update' for patching the host and rebooted to pick up the current centos >>> kernel. >>> >>> >>> >>> If you are at Step 6 then it is *crucial* to understand that the tooling >>> used to create the OSDs is no longer available and Step 7 *is absolutely >>> required*. >>> >>> >>> >>> ceph-volume has to scan the system and give you the output of all OSDs >>> found so that it can persist them in /etc/ceph/osd/*.json files and then >>> can later be >>> >>> "activated". >>> >>> >>> >>> >>> I didn't do anything to specific commands for just updating the ceph >>> RPMs in this process. >>> >>> >>> >>> It is not clear if you are at Step 6 and wondering why OSDs are not up, >>> or you are past that and ceph-volume wasn't able to detect anything. >>> >>> >>> >>> peter >>> >>> *Peter Eisch* >>> >>> Senior Site Reliability Engineer >>> >>> *T* >>> >>> 1.612.659.3228 >>> >>> [image: Facebook] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=uncHzIOSXt25%2F0NydXSJaLAf6E3Ad05N%2BJLBKYYJQ%2Fw%3D&reserved=0> >>> >>> [image: LinkedIn] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=NxQ3BmhgWo93uoQfJ1W7lLcDdSUQHgoXu1I49vzibwE%3D&reserved=0> >>> >>> [image: Twitter] >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=sYUapxFqHq0LVyxO4I7kkwN1y9PG5ZLHd83gseRIxvM%3D&reserved=0> >>> >>> virginpulse.com >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=aoXH94QOjdngJXkAPcz9kmJAK5BA6c9rR5BP01lX0bw%3D&reserved=0> >>> >>> | >>> >>> virginpulse.com/global-challenge >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=aUbx8uIlqldr5JtKT1PL05nbzcNPHONpouVkj1qXRdM%3D&reserved=0> >>> >>> >>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | >>> Switzerland | United Kingdom | USA >>> >>> Confidentiality Notice: The information contained in this e-mail, >>> including any attachment(s), is intended solely for use by the designated >>> recipient(s). Unauthorized use, dissemination, distribution, or >>> reproduction of this message by anyone other than the intended >>> recipient(s), or a person designated as responsible for delivering such >>> messages to the intended recipient, is strictly prohibited and may be >>> unlawful. This e-mail may contain proprietary, confidential or privileged >>> information. Any views or opinions expressed are solely those of the author >>> and do not necessarily represent those of Virgin Pulse, Inc. If you have >>> received this message in error, or are not the named recipient(s), please >>> immediately notify the sender and delete this e-mail message. >>> >>> v2.59 >>> >>> >>> From: Paul Emmerich <[email protected]> >>> Date: Wednesday, July 24, 2019 at 1:39 PM >>> To: Peter Eisch <[email protected]> >>> Cc: Xavier Trilla <[email protected]>, " >>> [email protected]" <[email protected]> >>> Subject: Re: [ceph-users] Upgrading and lost OSDs >>> >>> On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto: >>> [email protected]> wrote: >>> # lsblk >>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >>> sda 8:0 0 1.7T 0 disk >>> ├─sda1 8:1 0 100M 0 part >>> ├─sda2 8:2 0 1.7T 0 part >>> └─sda5 8:5 0 10M 0 part >>> sdb 8:16 0 1.7T 0 disk >>> ├─sdb1 8:17 0 100M 0 part >>> ├─sdb2 8:18 0 1.7T 0 part >>> └─sdb5 8:21 0 10M 0 part >>> sdc 8:32 0 1.7T 0 disk >>> ├─sdc1 8:33 0 100M 0 part >>> >>> That's ceph-disk which was removed, run "ceph-volume simple scan" >>> >>> >>> -- >>> Paul Emmerich >>> >>> Looking for help with your Ceph cluster? Contact us at >>> https://nam02.safelinks.protection.outlook.com/?url=https://croit.io&data=02|01|[email protected]|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0 >>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=gAFSElTqBRhu1zDIQYvWQ2WZtqqHoW%2FLa3stBfqXXHQ%3D&reserved=0> >>> >>> croit GmbH >>> Freseniusstr. 31h >>> 81247 München >>> >>> https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.io&data=02|01|[email protected]|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0 >>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.croit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=LWJKCZ4VxuwoHMJGSWiEWAcawFPw7pDGC48%2B6bnXk6A%3D&reserved=0> >>> Tel: +49 89 1896585 90 >>> >>> >>> ... >>> I'm thinking the OSD would start (I can recreate the .service >>> definitions in systemctl) if the above were mounted in a way like they are >>> on another of my hosts: >>> # lsblk >>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >>> sda 8:0 0 1.7T 0 disk >>> ├─sda1 8:1 0 100M 0 part >>> │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt >>> /var/lib/ceph/osd/ceph-16 >>> ├─sda2 8:2 0 1.7T 0 part >>> │ └─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt >>> └─sda5 8:5 0 10M 0 part >>> /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5 >>> sdb 8:16 0 1.7T 0 disk >>> ├─sdb1 8:17 0 100M 0 part >>> │ └─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt >>> /var/lib/ceph/osd/ceph-17 >>> ├─sdb2 8:18 0 1.7T 0 part >>> │ └─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt >>> └─sdb5 8:21 0 10M 0 part >>> /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2 >>> sdc 8:32 0 1.7T 0 disk >>> ├─sdc1 8:33 0 100M 0 part >>> │ └─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt >>> /var/lib/ceph/osd/ceph-18 >>> ├─sdc2 8:34 0 1.7T 0 part >>> │ └─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt >>> ... >>> >>> Thank you for your time on this, >>> >>> peter >>> >>> From: Xavier Trilla <mailto:[email protected]> >>> Date: Wednesday, July 24, 2019 at 1:25 PM >>> To: Peter Eisch <mailto:[email protected]> >>> Cc: "mailto:[email protected]"; <mailto:[email protected] >>> > >>> Subject: Re: [ceph-users] Upgrading and lost OSDs >>> >>> Hi Peter, >>> >>> Im not sure but maybe after some changes the OSDs are not being >>> recongnized by ceph scripts. >>> >>> Ceph used to use udev to detect the OSDs and then moved to lvm, which >>> kind of OSDs are you running? Blustore or filestore? Which version did you >>> use to create them? >>> >>> Cheers! >>> >>> El 24 jul 2019, a les 20:04, Peter Eisch <mailto:mailto: >>> [email protected]> va escriure: >>> Hi, >>> >>> I’m working through updating from 12.2.12/luminious to 14.2.2/nautilus >>> on centos 7.6. The managers are updated alright: >>> >>> # ceph -s >>> cluster: >>> id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec >>> health: HEALTH_WARN >>> Degraded data redundancy: 24177/9555955 objects degraded >>> (0.253%), 7 pgs degraded, 1285 pgs undersized >>> 3 monitors have not enabled msgr2 >>> ... >>> >>> I updated ceph on a OSD host with 'yum update' and then rebooted to grab >>> the current kernel. Along the way, the contents of all the directories in >>> /var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this. >>> I can manage the undersized but I'd like to get these drives working again >>> without deleting each OSD and recreating them. >>> >>> So far I've pulled the respective cephx key into the 'keyring' file and >>> populated 'bluestore' into the 'type' files but I'm unsure how to get the >>> lockboxes mounted to where I can get the OSDs running. The osd-lockbox >>> directory is otherwise untouched from when the OSDs were deployed. >>> >>> Is there a way to run ceph-deploy or some other tool to rebuild the >>> mounts for the drives? >>> >>> peter >>> >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com%2Flistinfo.cgi%2Fceph-users-ceph.com&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=jeAj%2FfzN%2BOG1NPFeZwqYnQiB4mgbgLGqq85tz99xBz8%3D&reserved=0> >>> >>> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
