I would try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume simple scan'
again. We had some problems upgrading due to OSDs (perhaps initially
installed as firefly?) missing the 'type' attribute and iirc the
'ceph-volume simple scan' command refused to overwrite existing json files
after I made some changes to ceph-volume.
Bob
On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza <ad...@redhat.com> wrote:
>
>
> On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch <peter.ei...@virginpulse.com>
> wrote:
>
>> Hi,
>>
>>
>>
>> I appreciate the insistency that the directions be followed. I wholly
>> agree. The only liberty I took was to do a ‘yum update’ instead of just
>> ‘yum update ceph-osd’ and then reboot. (Also my MDS runs on the MON hosts,
>> so it got update a step early.)
>>
>>
>>
>> As for the logs:
>>
>>
>>
>> [2019-07-24 15:07:22,713][ceph_volume.main][INFO ] Running command:
>> ceph-volume simple scan
>>
>> [2019-07-24 15:07:22,714][ceph_volume.process][INFO ] Running command:
>> /bin/systemctl show --no-pager --property=Id --state=running ceph-osd@*
>>
>> [2019-07-24 15:07:27,574][ceph_volume.main][INFO ] Running command:
>> ceph-volume simple activate --all
>>
>> [2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO ]
>> activating OSD specified in
>> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
>>
>> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
>> Required devices (block and data) not present for bluestore
>>
>> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
>> bluestore devices found: [u'data']
>>
>> [2019-07-24 15:07:27,576][ceph_volume][ERROR ] exception caught by
>> decorator
>>
>> Traceback (most recent call last):
>>
>> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
>> 59, in newfunc
>>
>> return f(*a, **kw)
>>
>> File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148,
>> in main
>>
>> terminal.dispatch(self.mapper, subcommand_args)
>>
>> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
>> 182, in dispatch
>>
>> instance.main()
>>
>> File
>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line
>> 33, in main
>>
>> terminal.dispatch(self.mapper, self.argv)
>>
>> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
>> 182, in dispatch
>>
>> instance.main()
>>
>> File
>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
>> line 272, in main
>>
>> self.activate(args)
>>
>> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
>> 16, in is_root
>>
>> return func(*a, **kw)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
>> line 131, in activate
>>
>> self.validate_devices(osd_metadata)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
>> line 62, in validate_devices
>>
>> raise RuntimeError('Unable to activate bluestore OSD due to missing
>> devices')
>>
>> RuntimeError: Unable to activate bluestore OSD due to missing devices
>>
>>
>>
>> (this is repeated for each of the 16 drives)
>>
>>
>>
>> Any other thoughts? (I’ll delete/create the OSDs with ceph-deply
>> otherwise.)
>>
>
> Try using `ceph-volume simple scan --stdout` so that it doesn't persist
> data onto /etc/ceph/osd/ and inspect that the JSON produced is capturing
> all the necessary details for OSDs.
>
> Alternatively, I would look into the JSON files already produced in
> /etc/ceph/osd/ and check if the details are correct. The `scan` sub-command
> does a tremendous effort to cover all cases where ceph-disk
> created an OSD (filestore, bluestore, dmcrypt, etc...) but it is possible
> that it may be hitting a problem. This is why the tool made these JSON
> files available, so that they could be inspected and corrected if anything.
>
> The details of the scan sub-command can be found at
> http://docs.ceph.com/docs/master/ceph-volume/simple/scan/ and the JSON
> structure is described in detail below at
> http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents
>
> In this particular case the tool is refusing to activate what seems to be
> a bluestore OSD. Is it really a bluestore OSD? if so, then it can't find
> where is the data partition. What does that partition look like (for any of
> the failing OSDs) ? Does it use dmcrypt, how was it created? (hopefully
> with ceph-disk!)
>
> If you know the data partition for a given OSD, try and pass it onto
> 'scan'. For example if it is /dev/sda1 you could do `ceph-volume simple
> scan /dev/sda1` and check its output.
>
>
>
>>
>> peter
>>
>>
>>
>>
>> Peter Eisch
>> Senior Site Reliability Engineer
>> T *1.612.659.3228* <1.612.659.3228>
>> [image: Facebook] <https://www.facebook.com/VirginPulse>
>> [image: LinkedIn] <https://www.linkedin.com/company/virgin-pulse>
>> [image: Twitter] <https://twitter.com/virginpulse>
>> *virginpulse.com* <https://www.virginpulse.com/>
>> | *virginpulse.com/global-challenge*
>> <https://www.virginpulse.com/en-gb/global-challenge/>
>>
>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
>> Switzerland | United Kingdom | USA
>> Confidentiality Notice: The information contained in this e-mail,
>> including any attachment(s), is intended solely for use by the designated
>> recipient(s). Unauthorized use, dissemination, distribution, or
>> reproduction of this message by anyone other than the intended
>> recipient(s), or a person designated as responsible for delivering such
>> messages to the intended recipient, is strictly prohibited and may be
>> unlawful. This e-mail may contain proprietary, confidential or privileged
>> information. Any views or opinions expressed are solely those of the author
>> and do not necessarily represent those of Virgin Pulse, Inc. If you have
>> received this message in error, or are not the named recipient(s), please
>> immediately notify the sender and delete this e-mail message.
>> v2.59
>>
>> *From: *Alfredo Deza <ad...@redhat.com>
>> *Date: *Wednesday, July 24, 2019 at 3:02 PM
>> *To: *Peter Eisch <peter.ei...@virginpulse.com>
>> *Cc: *Paul Emmerich <paul.emmer...@croit.io>, "ceph-users@lists.ceph.com"
>> <ceph-users@lists.ceph.com>
>> *Subject: *Re: [ceph-users] Upgrading and lost OSDs
>>
>>
>>
>>
>>
>> On Wed, Jul 24, 2019 at 3:49 PM Peter Eisch <peter.ei...@virginpulse.com>
>> wrote:
>>
>>
>>
>> I’m at step 6. I updated/rebooted the host to complete “installing the
>> new packages and restarting the ceph-osd daemon” on the first OSD host.
>> All the systemctl definitions to start the OSDs were deleted, all the
>> properties in /var/lib/ceph/osd/ceph-* directories were deleted. All the
>> files in /var/lib/ceph/osd-lockbox, for comparison, were untouched and
>> still present.
>>
>>
>>
>> Peeking into step 7 I can run ceph-volume:
>>
>>
>>
>> # ceph-volume simple scan /dev/sda1
>>
>> Running command: /usr/sbin/cryptsetup status /dev/sda1
>>
>> Running command: /usr/sbin/cryptsetup status
>> 93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> Running command: /bin/mount -v /dev/sda5 /tmp/tmpF5F8t2
>>
>> stdout: mount: /dev/sda5 mounted on /tmp/tmpF5F8t2.
>>
>> Running command: /usr/sbin/cryptsetup status /dev/sda5
>>
>> Running command: /bin/ceph --cluster ceph --name
>> client.osd-lockbox.93fb5f2f-0273-4c87-a718-886d7e6db983 --keyring
>> /tmp/tmpF5F8t2/keyring config-key get
>> dm-crypt/osd/93fb5f2f-0273-4c87-a718-886d7e6db983/luks
>>
>> Running command: /bin/umount -v /tmp/tmpF5F8t2
>>
>> stderr: umount: /tmp/tmpF5F8t2 (/dev/sda5) unmounted
>>
>> Running command: /usr/sbin/cryptsetup --key-file - --allow-discards
>> luksOpen /dev/sda1 93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> Running command: /bin/mount -v
>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 /tmp/tmpYK0WEV
>>
>> stdout: mount: /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 mounted
>> on /tmp/tmpYK0WEV.
>>
>> --> broken symlink found /tmp/tmpYK0WEV/block ->
>> /dev/mapper/a05b447c-c901-4690-a249-cc1a2d62a110
>>
>> Running command: /usr/sbin/cryptsetup status /tmp/tmpYK0WEV/block_dmcrypt
>>
>> Running command: /usr/sbin/cryptsetup status
>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> Running command: /bin/umount -v /tmp/tmpYK0WEV
>>
>> stderr: umount: /tmp/tmpYK0WEV
>> (/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983) unmounted
>>
>> Running command: /usr/sbin/cryptsetup remove
>> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> --> OSD 0 got scanned and metadata persisted to file:
>> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
>>
>> --> To take over management of this scanned OSD, and disable ceph-disk
>> and udev, run:
>>
>> --> ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> #
>>
>> #
>>
>> # ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983
>>
>> --> Required devices (block and data) not present for bluestore
>>
>> --> bluestore devices found: [u'data']
>>
>> --> RuntimeError: Unable to activate bluestore OSD due to missing devices
>>
>> #
>>
>>
>>
>> The tool detected bluestore, or rather, it failed to find a journal
>> associated with /dev/sda1. Scanning a single partition can cause that.
>> There is a flag to spit out the findings to STDOUT instead of persisting
>> them in /etc/ceph/osd/
>>
>>
>>
>> Since this is a "whole system" upgrade, then the upgrade documentation
>> instructions need to be followed:
>>
>>
>>
>> ceph-volume simple scan
>> ceph-volume simple activate --all
>>
>>
>>
>> If the `scan` command doesn't display any information (not even with the
>> --stdout flag) then the logs at /var/log/ceph/ceph-volume.log need to be
>> inspected. It would be useful to check any findings in there
>>
>>
>>
>>
>> Okay, this created /etc/ceph/osd/*.json. This is cool. Is there a
>> command or option which will read these files and mount the devices?
>>
>>
>>
>> peter
>>
>>
>>
>>
>>
>>
>>
>> *Peter Eisch*
>>
>> Senior Site Reliability Engineer
>>
>> *T*
>>
>> 1.612.659.3228
>>
>> [image: Facebook]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227707952&sdata=B2JiNp12z7gsfF2i5T2l%2FSjfg6Fhg8E85OpdyGpEMHg%3D&reserved=0>
>>
>> [image: LinkedIn]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=z3Ii%2BGgPKe7fCOhNGXw%2BlD9j28YCY4gH81is%2BJoiSJU%3D&reserved=0>
>>
>> [image: Twitter]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=zUaGW2Fm16sdyJdHUPtDN6CzaMXtxMOHvmNDi9VshCw%3D&reserved=0>
>>
>> virginpulse.com
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=n3m9S%2Bt8fYzGY%2BqPw2wT433TQhf2oPXp9wAum9s9%2BUk%3D&reserved=0>
>>
>> |
>>
>> virginpulse.com/global-challenge
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=5wn%2F%2FkY1IL0d4BxXNpqCJUHG09gUFRTr2S9KWv1mVG4%3D&reserved=0>
>>
>>
>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
>> Switzerland | United Kingdom | USA
>>
>> Confidentiality Notice: The information contained in this e-mail,
>> including any attachment(s), is intended solely for use by the designated
>> recipient(s). Unauthorized use, dissemination, distribution, or
>> reproduction of this message by anyone other than the intended
>> recipient(s), or a person designated as responsible for delivering such
>> messages to the intended recipient, is strictly prohibited and may be
>> unlawful. This e-mail may contain proprietary, confidential or privileged
>> information. Any views or opinions expressed are solely those of the author
>> and do not necessarily represent those of Virgin Pulse, Inc. If you have
>> received this message in error, or are not the named recipient(s), please
>> immediately notify the sender and delete this e-mail message.
>>
>> v2.59
>>
>> *From: *Alfredo Deza <ad...@redhat.com>
>> *Date: *Wednesday, July 24, 2019 at 2:20 PM
>> *To: *Peter Eisch <peter.ei...@virginpulse.com>
>> *Cc: *Paul Emmerich <paul.emmer...@croit.io>, "ceph-users@lists.ceph.com"
>> <ceph-users@lists.ceph.com>
>> *Subject: *Re: [ceph-users] Upgrading and lost OSDs
>>
>>
>>
>> On Wed, Jul 24, 2019 at 2:56 PM Peter Eisch <peter.ei...@virginpulse.com>
>> wrote:
>>
>> Hi Paul,
>>
>> To do better to answer you question, I'm following:
>> http://docs.ceph.com/docs/nautilus/releases/nautilus/
>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fnautilus%2Freleases%2Fnautilus%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=D1Q0Hrg9mq2tTtfTt1kp4Ts3vBc7mertKvYy8vBWNF8%3D&reserved=0>
>>
>> At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum
>> update' for patching the host and rebooted to pick up the current centos
>> kernel.
>>
>>
>>
>> If you are at Step 6 then it is *crucial* to understand that the tooling
>> used to create the OSDs is no longer available and Step 7 *is absolutely
>> required*.
>>
>>
>>
>> ceph-volume has to scan the system and give you the output of all OSDs
>> found so that it can persist them in /etc/ceph/osd/*.json files and then
>> can later be
>>
>> "activated".
>>
>>
>>
>>
>> I didn't do anything to specific commands for just updating the ceph RPMs
>> in this process.
>>
>>
>>
>> It is not clear if you are at Step 6 and wondering why OSDs are not up,
>> or you are past that and ceph-volume wasn't able to detect anything.
>>
>>
>>
>> peter
>>
>> *Peter Eisch*
>>
>> Senior Site Reliability Engineer
>>
>> *T*
>>
>> 1.612.659.3228
>>
>> [image: Facebook]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=uncHzIOSXt25%2F0NydXSJaLAf6E3Ad05N%2BJLBKYYJQ%2Fw%3D&reserved=0>
>>
>> [image: LinkedIn]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=NxQ3BmhgWo93uoQfJ1W7lLcDdSUQHgoXu1I49vzibwE%3D&reserved=0>
>>
>> [image: Twitter]
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=sYUapxFqHq0LVyxO4I7kkwN1y9PG5ZLHd83gseRIxvM%3D&reserved=0>
>>
>> virginpulse.com
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=aoXH94QOjdngJXkAPcz9kmJAK5BA6c9rR5BP01lX0bw%3D&reserved=0>
>>
>> |
>>
>> virginpulse.com/global-challenge
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=aUbx8uIlqldr5JtKT1PL05nbzcNPHONpouVkj1qXRdM%3D&reserved=0>
>>
>>
>> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
>> Switzerland | United Kingdom | USA
>>
>> Confidentiality Notice: The information contained in this e-mail,
>> including any attachment(s), is intended solely for use by the designated
>> recipient(s). Unauthorized use, dissemination, distribution, or
>> reproduction of this message by anyone other than the intended
>> recipient(s), or a person designated as responsible for delivering such
>> messages to the intended recipient, is strictly prohibited and may be
>> unlawful. This e-mail may contain proprietary, confidential or privileged
>> information. Any views or opinions expressed are solely those of the author
>> and do not necessarily represent those of Virgin Pulse, Inc. If you have
>> received this message in error, or are not the named recipient(s), please
>> immediately notify the sender and delete this e-mail message.
>>
>> v2.59
>>
>>
>> From: Paul Emmerich <paul.emmer...@croit.io>
>> Date: Wednesday, July 24, 2019 at 1:39 PM
>> To: Peter Eisch <peter.ei...@virginpulse.com>
>> Cc: Xavier Trilla <xavier.tri...@clouding.io>, "ceph-users@lists.ceph.com"
>> <ceph-users@lists.ceph.com>
>> Subject: Re: [ceph-users] Upgrading and lost OSDs
>>
>> On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto:
>> peter.ei...@virginpulse.com> wrote:
>> # lsblk
>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>> sda 8:0 0 1.7T 0 disk
>> ├─sda1 8:1 0 100M 0 part
>> ├─sda2 8:2 0 1.7T 0 part
>> └─sda5 8:5 0 10M 0 part
>> sdb 8:16 0 1.7T 0 disk
>> ├─sdb1 8:17 0 100M 0 part
>> ├─sdb2 8:18 0 1.7T 0 part
>> └─sdb5 8:21 0 10M 0 part
>> sdc 8:32 0 1.7T 0 disk
>> ├─sdc1 8:33 0 100M 0 part
>>
>> That's ceph-disk which was removed, run "ceph-volume simple scan"
>>
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at
>> https://nam02.safelinks.protection.outlook.com/?url=https://croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0
>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=gAFSElTqBRhu1zDIQYvWQ2WZtqqHoW%2FLa3stBfqXXHQ%3D&reserved=0>
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>>
>> https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0
>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.croit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=LWJKCZ4VxuwoHMJGSWiEWAcawFPw7pDGC48%2B6bnXk6A%3D&reserved=0>
>> Tel: +49 89 1896585 90
>>
>>
>> ...
>> I'm thinking the OSD would start (I can recreate the .service definitions
>> in systemctl) if the above were mounted in a way like they are on another
>> of my hosts:
>> # lsblk
>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>> sda 8:0 0 1.7T 0 disk
>> ├─sda1 8:1 0 100M 0 part
>> │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt
>> /var/lib/ceph/osd/ceph-16
>> ├─sda2 8:2 0 1.7T 0 part
>> │ └─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt
>> └─sda5 8:5 0 10M 0 part
>> /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5
>> sdb 8:16 0 1.7T 0 disk
>> ├─sdb1 8:17 0 100M 0 part
>> │ └─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt
>> /var/lib/ceph/osd/ceph-17
>> ├─sdb2 8:18 0 1.7T 0 part
>> │ └─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt
>> └─sdb5 8:21 0 10M 0 part
>> /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2
>> sdc 8:32 0 1.7T 0 disk
>> ├─sdc1 8:33 0 100M 0 part
>> │ └─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt
>> /var/lib/ceph/osd/ceph-18
>> ├─sdc2 8:34 0 1.7T 0 part
>> │ └─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt
>> ...
>>
>> Thank you for your time on this,
>>
>> peter
>>
>> From: Xavier Trilla <mailto:xavier.tri...@clouding.io>
>> Date: Wednesday, July 24, 2019 at 1:25 PM
>> To: Peter Eisch <mailto:peter.ei...@virginpulse.com>
>> Cc: "mailto:ceph-users@lists.ceph.com"; <mailto:ceph-users@lists.ceph.com>
>> Subject: Re: [ceph-users] Upgrading and lost OSDs
>>
>> Hi Peter,
>>
>> Im not sure but maybe after some changes the OSDs are not being
>> recongnized by ceph scripts.
>>
>> Ceph used to use udev to detect the OSDs and then moved to lvm, which
>> kind of OSDs are you running? Blustore or filestore? Which version did you
>> use to create them?
>>
>> Cheers!
>>
>> El 24 jul 2019, a les 20:04, Peter Eisch <mailto:mailto:
>> peter.ei...@virginpulse.com> va escriure:
>> Hi,
>>
>> I’m working through updating from 12.2.12/luminious to 14.2.2/nautilus on
>> centos 7.6. The managers are updated alright:
>>
>> # ceph -s
>> cluster:
>> id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec
>> health: HEALTH_WARN
>> Degraded data redundancy: 24177/9555955 objects degraded
>> (0.253%), 7 pgs degraded, 1285 pgs undersized
>> 3 monitors have not enabled msgr2
>> ...
>>
>> I updated ceph on a OSD host with 'yum update' and then rebooted to grab
>> the current kernel. Along the way, the contents of all the directories in
>> /var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this.
>> I can manage the undersized but I'd like to get these drives working again
>> without deleting each OSD and recreating them.
>>
>> So far I've pulled the respective cephx key into the 'keyring' file and
>> populated 'bluestore' into the 'type' files but I'm unsure how to get the
>> lockboxes mounted to where I can get the OSDs running. The osd-lockbox
>> directory is otherwise untouched from when the OSDs were deployed.
>>
>> Is there a way to run ceph-deploy or some other tool to rebuild the
>> mounts for the drives?
>>
>> peter
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com%2Flistinfo.cgi%2Fceph-users-ceph.com&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=jeAj%2FfzN%2BOG1NPFeZwqYnQiB4mgbgLGqq85tz99xBz8%3D&reserved=0>
>>
>> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
