I can raise a tracker for this issue since it looks like an intermittent issue and mostly dependent on specific hardware or it would be better if you add all the hardware/os details in tracker.ceph.com, also from your logs it looks like you have Resource busy issue: Error: Failed to add partition 2 (Device or resource busy)
From my test run logs on centos 7.2 , 10.0.5 ( http://qa-proxy.ceph.com/teuthology/vasu-2016-03-15_15:34:41-selinux-master---basic-mira/62626/teuthology.log ) 2016-03-15T18:49:56.305 INFO:teuthology.orchestra.run.mira041.stderr:[ceph_deploy.osd][DEBUG ] Preparing host mira041 disk /dev/sdb journal None activate True 2016-03-15T18:49:56.305 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] find the location of an executable 2016-03-15T18:49:56.309 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][INFO ] Running command: sudo /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/sdb 2016-03-15T18:49:56.546 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid 2016-03-15T18:49:56.611 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster ceph 2016-03-15T18:49:56.643 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph 2016-03-15T18:49:56.708 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph 2016-03-15T18:49:56.708 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.709 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] set_type: Will colocate journal with data on /dev/sdb 2016-03-15T18:49:56.709 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size 2016-03-15T18:49:56.774 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.774 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.775 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.775 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs 2016-03-15T18:49:56.777 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs 2016-03-15T18:49:56.809 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs 2016-03-15T18:49:56.841 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs 2016-03-15T18:49:56.857 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.858 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.858 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] ptype_tobe_for_name: name = journal 2016-03-15T18:49:56.859 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:56.859 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] create_partition: Creating journal partition num 2 size 5120 on /dev/sdb 2016-03-15T18:49:56.859 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb 2016-03-15T18:49:57.927 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] The operation has completed successfully. 2016-03-15T18:49:57.927 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] update_partition: Calling partprobe on created device /dev/sdb 2016-03-15T18:49:57.928 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command_check_call: Running command: /usr/bin/udevadm settle --timeout=600 2016-03-15T18:49:58.393 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: Running command: /sbin/partprobe /dev/sdb 2016-03-15T18:49:58.393 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command_check_call: Running command: /usr/bin/udevadm settle --timeout=600 2016-03-15T18:49:59.109 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:59.203 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:59.203 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid 2016-03-15T18:49:59.204 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] prepare_device: Journal is GPT partition /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 2016-03-15T18:49:59.204 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] prepare_device: Journal is GPT partition /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 2016-03-15T18:49:59.204 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:59.205 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] set_data_partition: Creating osd partition on /dev/sdb 2016-03-15T18:49:59.205 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:59.205 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] ptype_tobe_for_name: name = data 2016-03-15T18:49:59.206 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid: get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid 2016-03-15T18:49:59.206 INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] create_partition: Creating data partition num 1 size 0 on /dev/sdb On Thu, Mar 17, 2016 at 8:06 AM, Dan van der Ster <[email protected]> wrote: > Hi, > > Is there a tracker for this? We just hit the same problem on 10.0.5. > > Cheers, Dan > > # rpm -q ceph > ceph-10.0.5-0.el7.x86_64 > > # cat /etc/redhat-release > CentOS Linux release 7.2.1511 (Core) > > # ceph-disk -v prepare /dev/sdc > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph > --show-config-value=fsid > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_mkfs_type > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_mkfs_options_xfs > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_fs_mkfs_options_xfs > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_mount_options_xfs > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph > --show-config-value=osd_journal_size > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_cryptsetup_parameters > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_dmcrypt_key_size > INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > --name=osd. --lookup osd_dmcrypt_type > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > INFO:ceph-disk:Will colocate journal with data on /dev/sdc > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > /sys/dev/block/8:32/dm/uuid > DEBUG:ceph-disk:Creating journal partition num 2 size 20480 on /dev/sdc > INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:20480M > --change-name=2:ceph journal > --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d > --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- > /dev/sdc > The operation has completed successfully. > DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdc > INFO:ceph-disk:Running command: /usr/bin/udevadm settle > INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/sdc > Error: Error informing the kernel about modifications to partition > /dev/sdc2 -- Device or resource busy. This means Linux won't know > about any changes you made to /dev/sdc2 until you reboot -- so you > shouldn't mount it or use it in any way before rebooting. > Error: Failed to add partition 2 (Device or resource busy) > Traceback (most recent call last): > File "/usr/sbin/ceph-disk", line 3528, in <module> > main(sys.argv[1:]) > File "/usr/sbin/ceph-disk", line 3482, in main > args.func(args) > File "/usr/sbin/ceph-disk", line 1817, in main_prepare > luks=luks > File "/usr/sbin/ceph-disk", line 1447, in prepare_journal > return prepare_journal_dev(data, journal, journal_size, > journal_uuid, journal_dm_keypath, cryptsetup_parameters, luks) > File "/usr/sbin/ceph-disk", line 1401, in prepare_journal_dev > raise Error(e) > __main__.Error: Error: Command '['/usr/sbin/partprobe', '/dev/sdc']' > returned non-zero exit status 1 > > On Tue, Mar 15, 2016 at 8:38 PM, Vasu Kulkarni <[email protected]> > wrote: > > Thanks for the steps that should be enough to test it out, I hope you got > > the latest ceph-deploy either from pip or throught github. > > > > On Tue, Mar 15, 2016 at 12:29 PM, Stephen Lord <[email protected]> > > wrote: > >> > >> I would have to nuke my cluster right now, and I do not have a spare > one.. > >> > >> The procedure though is literally this, given a 3 node redhat 7.2 > cluster, > >> ceph00, ceph01 and ceph02 > >> > >> ceph-deploy install --testing ceph00 ceph01 ceph02 > >> ceph-deploy new ceph00 ceph01 ceph02 > >> > >> ceph-deploy mon create ceph00 ceph01 ceph02 > >> ceph-deploy gatherkeys ceph00 > >> > >> ceph-deploy osd create ceph00:sdb:/dev/sdi > >> ceph-deploy osd create ceph00:sdc:/dev/sdi > >> > >> All devices have their partition tables wiped before this. They are all > >> just SATA devices, no special devices in the way. > >> > >> sdi is an ssd and it is being carved up for journals. The first osd > create > >> works, the second one gets stuck in a loop in the update_partition call > in > >> ceph_disk for the 5 iterations before it gives up. When I look in > >> /sys/block/sdi the partition for the first osd is visible, the one for > the > >> second is not. However looking at /proc/partitions it sees the correct > >> thing. So something about partprobe is not kicking udev into doing the > right > >> thing when the second partition is added I suspect. > >> > >> If I do not use the separate journal device then it usually works, but > >> occasionally I see a single retry in that same loop. > >> > >> There is code in ceph_deploy which uses partprobe or partx depending on > >> which distro it detects, that is how I worked out what to change here. > >> > >> If I have to tear things down again I will reproduce and post here. > >> > >> Steve > >> > >> > On Mar 15, 2016, at 2:12 PM, Vasu Kulkarni <[email protected]> > wrote: > >> > > >> > Do you mind giving the full failed logs somewhere in fpaste.org along > >> > with some os version details? > >> > There are some known issues on RHEL, If you use 'osd prepare' and > 'osd > >> > activate'(specifying just the journal partition here) it might work > better. > >> > > >> > On Tue, Mar 15, 2016 at 12:05 PM, Stephen Lord < > [email protected]> > >> > wrote: > >> > Not multipath if you mean using the multipath driver, just trying to > >> > setup OSDs which use a data disk and a journal ssd. If I run just a > disk > >> > based OSD and only specify one device to ceph-deploy then it usually > works > >> > although sometimes has to retry. In the case where I am using it to > carve an > >> > SSD into several partitions for journals it fails on the second one. > >> > > >> > Steve > >> > > >> > >> > >> ---------------------------------------------------------------------- > >> The information contained in this transmission may be confidential. Any > >> disclosure, copying, or further distribution of confidential > information is > >> not permitted unless such privilege is explicitly granted in writing by > >> Quantum. Quantum reserves the right to have electronic communications, > >> including email and attachments, sent across its networks filtered > through > >> anti virus and spam software programs and retain such messages in order > to > >> comply with applicable data security and retention requirements. > Quantum is > >> not responsible for the proper and complete transmission of the > substance of > >> this communication or for any delay in its receipt. > > > > > > > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
