Greetings ceph-users. I have been trying to build a test cluster in a KVM
environment - something I have done before successfully before but this time
I'm running into an issue I can't seem to get past. My Internet searches have
shown instances of this by other users that involved either ownership problems
with the OSD devices, or partition UID's needing to be set. Neither of these
problems seem to be in play here.
The cluster is on centos 7, running Ceph 10.2.10. I have configured one mon,
and 3 OSD servers with 4 disks each, and each is set to journal on a separate
partition of an SSD, one SSD per VM. I have built this VM environment several
times now, and recently I always have the same issue on at least one of my VM
OSD's and I cannot seem to get any hints of where the problem lies from the
sparse information printed to the console during the failure.
In addition to setting partition ownerships to ceph:ceph and UIDs to one of the
the values "set_data_partition" says it expects, I also zeroed out the entire
contents of both drives and re-partioned, but I still get the same results. The
problem at present only occurs on one virtual server, the other 8 drives split
between the other 2 VM OSD's had no issue with prepare or activate. I see no
difference between this server or drive configuration vs the other two that run
fine.
Hopefully someone can at least point me to some more fruitful log information,
"Failed to activate" isn't very helpful by itself. There is nothing in messages
other than clean mount/unmount messages for the OSD data device being processed
(in this case /dev/vdb1). BTW, I have also tried to repeat the same process
without a separate journal device ( just using prepare/activate osd3:/dev/vdb1)
and I got the same "Failed to activate" result.
[cephuser@groot cephcluster]$ ceph-deploy osd prepare osd3:/dev/vdb1:/dev/vdf1
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/cephuser/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd prepare
osd3:/dev/vdb1:/dev/vdf1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] block_db : None
[ceph_deploy.cli][INFO ] disk : [('osd3',
'/dev/vdb1', '/dev/vdf1')]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] block_wal : None
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : prepare
[ceph_deploy.cli][INFO ] dmcrypt_key_dir :
/etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at 0x2a7bdd0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] filestore : None
[ceph_deploy.cli][INFO ] func : <function osd at
0x2a6f1b8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks osd3:/dev/vdb1:/dev/vdf1
[osd3][DEBUG ] connection detected need for sudo
[osd3][DEBUG ] connected to host: osd3
[osd3][DEBUG ] detect platform information from remote host
[osd3][DEBUG ] detect machine type
[osd3][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to osd3
[osd3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.osd][DEBUG ] Preparing host osd3 disk /dev/vdb1 journal /dev/vdf1
activate False
[osd3][DEBUG ] find the location of an executable
[osd3][INFO ] Running command: sudo /usr/sbin/ceph-disk -v prepare --cluster
ceph --fs-type xfs -- /dev/vdb1 /dev/vdf1
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
--show-config-value=fsid
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd
--check-allows-journal -i 0 --log-file $run_dir/$cluster-osd-check.log
--cluster ceph --setuser ceph --setgroup ceph
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd
--check-wants-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster
ceph --setuser ceph --setgroup ceph
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd
--check-needs-journal -i 0 --log-file $run_dir/$cluster-osd-check.log --cluster
ceph --setuser ceph --setgroup ceph
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
--show-config-value=osd_journal_size
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_mkfs_options_xfs
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_fs_mkfs_options_xfs
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_mount_options_xfs
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_fs_mount_options_xfs
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdf1 uuid path is
/sys/dev/block/252:81/dm/uuid
[osd3][WARNIN] prepare_device: Journal /dev/vdf1 is a partition
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdf1 uuid path is
/sys/dev/block/252:81/dm/uuid
[osd3][WARNIN] prepare_device: OSD will not be hot-swappable if journal is not
the same device as the osd data
[osd3][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/vdf1
[osd3][WARNIN] prepare_device: Journal /dev/vdf1 was not prepared with
ceph-disk. Symlinking directly.
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] set_data_partition: OSD data device /dev/vdb1 is a partition
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/vdb1
[osd3][WARNIN] populate_data_path_device: Creating xfs fs on /dev/vdb1
[osd3][WARNIN] command_check_call: Running command: /sbin/mkfs -t xfs -f -i
size=2048 -- /dev/vdb1
[osd3][DEBUG ] meta-data=/dev/vdb1 isize=2048 agcount=4,
agsize=8920960 blks
[osd3][DEBUG ] = sectsz=512 attr=2,
projid32bit=1
[osd3][DEBUG ] = crc=1 finobt=0, sparse=0
[osd3][DEBUG ] data = bsize=4096 blocks=35683840,
imaxpct=25
[osd3][DEBUG ] = sunit=0 swidth=0 blks
[osd3][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 ftype=1
[osd3][DEBUG ] log =internal log bsize=4096 blocks=17423,
version=2
[osd3][DEBUG ] = sectsz=512 sunit=0 blks,
lazy-count=1
[osd3][DEBUG ] realtime =none extsz=4096 blocks=0,
rtextents=0
[osd3][WARNIN] mount: Mounting /dev/vdb1 on /var/lib/ceph/tmp/mnt.EWuVuW with
options noatime,inode64
[osd3][WARNIN] command_check_call: Running command: /usr/bin/mount -t xfs -o
noatime,inode64 -- /dev/vdb1 /var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] command: Running command: /sbin/restorecon
/var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] populate_data_path: Preparing osd data dir
/var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] command: Running command: /sbin/restorecon -R
/var/lib/ceph/tmp/mnt.EWuVuW/ceph_fsid.7378.tmp
[osd3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.EWuVuW/ceph_fsid.7378.tmp
[osd3][WARNIN] command: Running command: /sbin/restorecon -R
/var/lib/ceph/tmp/mnt.EWuVuW/fsid.7378.tmp
[osd3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.EWuVuW/fsid.7378.tmp
[osd3][WARNIN] command: Running command: /sbin/restorecon -R
/var/lib/ceph/tmp/mnt.EWuVuW/magic.7378.tmp
[osd3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.EWuVuW/magic.7378.tmp
[osd3][WARNIN] command: Running command: /sbin/restorecon -R
/var/lib/ceph/tmp/mnt.EWuVuW/journal_uuid.7378.tmp
[osd3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.EWuVuW/journal_uuid.7378.tmp
[osd3][WARNIN] adjust_symlink: Creating symlink
/var/lib/ceph/tmp/mnt.EWuVuW/journal -> /dev/vdf1
[osd3][WARNIN] command: Running command: /sbin/restorecon -R
/var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] command: Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] command_check_call: Running command: /bin/umount --
/var/lib/ceph/tmp/mnt.EWuVuW
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][INFO ] checking OSD status...
[osd3][DEBUG ] find the location of an executable
[osd3][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat
--format=json
[ceph_deploy.osd][DEBUG ] Host osd3 is now ready for osd use.
[cephuser@groot cephcluster]$ ceph-deploy osd activate osd3:/dev/vdb1:/dev/vdf1
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/cephuser/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd activate
osd3:/dev/vdb1:/dev/vdf1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : activate
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at 0x20f9dd0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at
0x20ed1b8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('osd3',
'/dev/vdb1', '/dev/vdf1')]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks osd3:/dev/vdb1:/dev/vdf1
[osd3][DEBUG ] connection detected need for sudo
[osd3][DEBUG ] connected to host: osd3
[osd3][DEBUG ] detect platform information from remote host
[osd3][DEBUG ] detect machine type
[osd3][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.osd][DEBUG ] activating host osd3 disk /dev/vdb1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[osd3][DEBUG ] find the location of an executable
[osd3][INFO ] Running command: sudo /usr/sbin/ceph-disk -v activate
--mark-init systemd --mount /dev/vdb1
[osd3][WARNIN] main_activate: path = /dev/vdb1
[osd3][WARNIN] get_dm_uuid: get_dm_uuid /dev/vdb1 uuid path is
/sys/dev/block/252:17/dm/uuid
[osd3][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/vdb1
[osd3][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value --
/dev/vdb1
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_mount_options_xfs
[osd3][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_fs_mount_options_xfs
[osd3][WARNIN] mount: Mounting /dev/vdb1 on /var/lib/ceph/tmp/mnt.G7uifc with
options noatime,inode64
[osd3][WARNIN] command_check_call: Running command: /usr/bin/mount -t xfs -o
noatime,inode64 -- /dev/vdb1 /var/lib/ceph/tmp/mnt.G7uifc
[osd3][WARNIN] command: Running command: /sbin/restorecon
/var/lib/ceph/tmp/mnt.G7uifc
[osd3][WARNIN] activate: Cluster uuid is 83d61520-5a38-4f50-9b54-bef4f6bef08c
[osd3][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
--show-config-value=fsid
[osd3][WARNIN] activate: Cluster name is ceph
[osd3][WARNIN] activate: OSD uuid is 4627c861-71b7-485e-a402-30bff54a963c
[osd3][WARNIN] allocate_osd_id: Allocating OSD id...
[osd3][WARNIN] command: Running command: /usr/bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd
create --concise 4627c861-71b7-485e-a402-30bff54a963c
[osd3][WARNIN] mount_activate: Failed to activate
[osd3][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.G7uifc
[osd3][WARNIN] command_check_call: Running command: /bin/umount --
/var/lib/ceph/tmp/mnt.G7uifc
[osd3][WARNIN] Traceback (most recent call last):
[osd3][WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module>
[osd3][WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts',
'ceph-disk')()
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 5371, in run
[osd3][WARNIN] main(sys.argv[1:])
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 5322, in main
[osd3][WARNIN] args.func(args)
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 3445, in main_activate
[osd3][WARNIN] reactivate=args.reactivate,
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 3202, in mount_activate
[osd3][WARNIN] (osd_id, cluster) = activate(path, activate_key_template,
init)
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 3365, in activate
[osd3][WARNIN] keyring=keyring,
[osd3][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
line 1013, in allocate_osd_id
[osd3][WARNIN] raise Error('ceph osd create failed', e, e.output)
[osd3][WARNIN] ceph_disk.main.Error: Error: ceph osd create failed: Command
'/usr/bin/ceph' returned non-zero exit status 1: 2018-02-07 09:38:40.104098
7fa479cf2700 0 librados: client.bootstrap-osd authentication error (1)
Operation not permitted
[osd3][WARNIN] Error connecting to cluster: PermissionError
[osd3][WARNIN]
[osd3][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command:
/usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/vdb1
[cephuser@groot cephcluster]$
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
