On Mon, Jun 30, 2014 at 11:22 AM, Iban Cabrillo <[email protected]> wrote: > Hi Alfredo and folk, > Could you have a look at this? > Someone else has any idea why i am getting this error? > > Thanks in advance, I > > > > 2014-06-27 16:37 GMT+02:00 Iban Cabrillo <[email protected]>: > >> Hi Alfredo, >> This is the complete procedure: >> >> >> On OSD node: >> >> [ceph@ceph02 ~]$ sudo parted /dev/xvdb >> >> GNU Parted 2.1 >> Using /dev/xvdb >> Welcome to GNU Parted! Type 'help' to view a list of commands. >> (parted) p >> Model: Xen Virtual Block Device (xvd) >> Disk /dev/xvdb: 107GB >> Sector size (logical/physical): 512B/512B >> Partition Table: gpt >> >> Number Start End Size File system Name Flags >> >> [ceph@ceph02 ~]$ sudo ls -la /var/lib/ceph/tmp/ >> total 8 >> drwxr-xr-x 2 root root 4096 Jun 27 16:30 . >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 .. >> [ceph@ceph02 ~]$ sudo ls -la /var/lib/ceph/osd/ >> total 8 >> drwxr-xr-x 2 root root 4096 Jun 27 12:14 . >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 .. >> >> On ceph admin node: >> >> [ceph@cephadm ~]$ sudo ceph osd tree >> # id weight type name up/down reweight >> -1 0.14 root default >> -2 0.009995 host ceph02 >> 1 0.009995 osd.1 DNE >> -3 0.03999 host ceph04 >> 4 0.03999 osd.4 up 1 >> -4 0.09 host ceph03 >> 6 0.09 osd.6 up 1 >> >> >> [ceph@cephadm ceph-cloud]$ ceph-deploy osd prepare ceph02:xvdb >> [ceph_deploy.conf][DEBUG ] found configuration file at: >> /home/ceph/.cephdeploy.conf >> [ceph_deploy.cli][INFO ] Invoked (1.5.5): /usr/bin/ceph-deploy osd >> prepare ceph02:xvdb >> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph02:/dev/xvdb: >> [ceph02][DEBUG ] connected to host: ceph02 >> [ceph02][DEBUG ] detect platform information from remote host >> [ceph02][DEBUG ] detect machine type >> [ceph_deploy.osd][INFO ] Distro info: Scientific Linux 6.2 Carbon >> [ceph_deploy.osd][DEBUG ] Deploying osd to ceph02 >> [ceph02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf >> [ceph02][INFO ] Running command: sudo udevadm trigger >> --subsystem-match=block --action=add >> [ceph_deploy.osd][DEBUG ] Preparing host ceph02 disk /dev/xvdb journal >> None activate False >> [ceph02][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs >> --cluster ceph -- /dev/xvdb >> [ceph02][DEBUG ] Setting name! >> [ceph02][DEBUG ] partNum is 1 >> [ceph02][DEBUG ] REALLY setting name! >> [ceph02][DEBUG ] The operation has completed successfully. >> [ceph02][DEBUG ] Setting name! >> [ceph02][DEBUG ] partNum is 0 >> [ceph02][DEBUG ] REALLY setting name! >> [ceph02][DEBUG ] The operation has completed successfully. >> [ceph02][DEBUG ] meta-data=/dev/xvdb1 isize=2048 agcount=4, >> agsize=5897919 blks >> [ceph02][DEBUG ] = sectsz=512 attr=2 >> [ceph02][DEBUG ] data = bsize=4096 >> blocks=23591675, imaxpct=25 >> [ceph02][DEBUG ] = sunit=0 swidth=0 >> blks >> [ceph02][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 >> [ceph02][DEBUG ] log =internal log bsize=4096 >> blocks=11519, version=2 >> [ceph02][DEBUG ] = sectsz=512 sunit=0 >> blks, lazy-count=1 >> [ceph02][DEBUG ] realtime =none extsz=4096 blocks=0, >> rtextents=0 >> [ceph02][DEBUG ] The operation has completed successfully. >> [ceph02][WARNIN] INFO:ceph-disk:Will colocate journal with data on >> /dev/xvdb >> [ceph02][INFO ] checking OSD status... >> [ceph02][INFO ] Running command: sudo ceph --cluster=ceph osd stat >> --format=json >> [ceph_deploy.osd][DEBUG ] Host ceph02 is now ready for osd use. >> >> >> >> If i make create instead of prepare do the same (create do not make the >> trick prepare+activate ) >> >> >> In the OSD: >> >> [ceph@ceph02 ~]$ sudo parted /dev/xvdb >> >> GNU Parted 2.1 >> Using /dev/xvdb >> Welcome to GNU Parted! Type 'help' to view a list of commands. >> (parted) p >> Model: Xen Virtual Block Device (xvd) >> Disk /dev/xvdb: 107GB >> Sector size (logical/physical): 512B/512B >> Partition Table: gpt >> >> Number Start End Size File system Name Flags >> 2 1049kB 10.7GB 10.7GB xfs ceph journal >> 1 10.7GB 107GB 96.6GB xfs ceph data >> >> (parted) q >> >> [ceph@ceph02 ~]$ sudo ls -la /var/lib/ceph/osd/ >> total 8 >> drwxr-xr-x 2 root root 4096 Jun 27 12:14 . >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 .. >> [ceph@ceph02 ~]$ sudo ls -la /var/lib/ceph/tmp/ >> total 8 >> drwxr-xr-x 2 root root 4096 Jun 27 16:32 . >> drwxr-xr-x 7 root root 4096 Jun 26 22:30 .. >> -rw-r--r-- 1 root root 0 Jun 27 16:32 ceph-disk.prepare.lock >> >> [ceph@ceph02 ~]$ sudo ceph-disk list >> /dev/xvda1 other, ext4, mounted on / >> /dev/xvdb : >> /dev/xvdb1 ceph data, prepared, cluster ceph, journal /dev/xvdb2 >> /dev/xvdb2 ceph journal, for /dev/xvdb1 >> >> >> In the cephadm: >> >> [ceph@cephadm ceph-cloud]$ ceph-deploy osd activate ceph02:xvdb1 >> [ceph_deploy.conf][DEBUG ] found configuration file at: >> /home/ceph/.cephdeploy.conf >> [ceph_deploy.cli][INFO ] Invoked (1.5.5): /usr/bin/ceph-deploy osd >> activate ceph02:xvdb1 >> [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph02:/dev/xvdb1: >> [ceph02][DEBUG ] connected to host: ceph02 >> [ceph02][DEBUG ] detect platform information from remote host >> [ceph02][DEBUG ] detect machine type >> [ceph_deploy.osd][INFO ] Distro info: Scientific Linux 6.2 Carbon >> [ceph_deploy.osd][DEBUG ] activating host ceph02 disk /dev/xvdb1 >> [ceph_deploy.osd][DEBUG ] will use init type: sysvinit >> [ceph02][INFO ] Running command: sudo ceph-disk-activate --mark-init >> sysvinit --mount /dev/xvdb1 >> [ceph02][WARNIN] got monmap epoch 2 >> [ceph02][WARNIN] 2014-06-27 16:35:49.948865 7f143254e7a0 -1 >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkjournal error creating journal on >> /var/lib/ceph/tmp/mnt.9VZHpR/journal: (2) No such file or directory >> [ceph02][WARNIN] 2014-06-27 16:35:49.948893 7f143254e7a0 -1 OSD::mkfs: >> ObjectStore::mkfs failed with error -2 >> [ceph02][WARNIN] 2014-06-27 16:35:49.948957 7f143254e7a0 -1 ** ERROR: >> error creating empty object store in /var/lib/ceph/tmp/mnt.9VZHpR: (2) No >> such file or directory >> >> [ceph02][WARNIN] ERROR:ceph-disk:Failed to activate >> [ceph02][WARNIN] Traceback (most recent call last): >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 2579, in <module> >> [ceph02][WARNIN] main() >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 2557, in main >> [ceph02][WARNIN] args.func(args) >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1910, in main_activate >> [ceph02][WARNIN] init=args.mark_init, >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1686, in >> mount_activate >> [ceph02][WARNIN] (osd_id, cluster) = activate(path, >> activate_key_template, init) >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1849, in activate >> [ceph02][WARNIN] keyring=keyring, >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 1484, in mkfs >> [ceph02][WARNIN] '--keyring', os.path.join(path, 'keyring'), >> [ceph02][WARNIN] File "/usr/sbin/ceph-disk", line 303, in >> command_check_call >> [ceph02][WARNIN] return subprocess.check_call(arguments) >> [ceph02][WARNIN] File "/usr/lib64/python2.6/subprocess.py", line 505, in >> check_call >> [ceph02][WARNIN] raise CalledProcessError(retcode, cmd) >> [ceph02][WARNIN] subprocess.CalledProcessError: Command >> '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '0', >> '--monmap', '/var/lib/ceph/tmp/mnt.9VZHpR/activate.monmap', '--osd-data', >> '/var/lib/ceph/tmp/mnt.9VZHpR', '--osd-journal', >> '/var/lib/ceph/tmp/mnt.9VZHpR/journal', '--osd-uuid', >> '5e93fa7c-b6f7-4684-981b-bf73254bd87a', '--keyring', >> '/var/lib/ceph/tmp/mnt.9VZHpR/keyring']' returned non-zero exit status 1 >> [ceph02][ERROR ] RuntimeError: command returned non-zero exit status: 1 >> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: >> ceph-disk-activate --mark-init sysvinit --mount /dev/xvdb1 >>
Is it possible you have tried to install Ceph a few times to get into this state? It is odd that /var/lib/ceph/tmp doesn't exist (as presented by the logs). Are you able to replicate this problem from scratch? I would try to redo everything from a fresh host. If that is not possible, you could tell ceph-deploy to wipe everything. Just as long as you do not care about the data (the following will wipe all data stored in ceph) you could run this: ceph-deploy purge ceph02 && ceph-deploy purgedata ceph02 && ceph-deploy install ceph02 >> >> At OSD log: >> >> [ceph@ceph02 ~]$ tail -100 /var/log/ceph/ceph-osd.0.log >> 2014-06-27 16:35:49.859984 7f143254e7a0 0 ceph version 0.80.1 >> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 6590 >> 2014-06-27 16:35:49.861265 7f143254e7a0 1 >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkfs in /var/lib/ceph/tmp/mnt.9VZHpR >> 2014-06-27 16:35:49.861319 7f143254e7a0 1 >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkfs fsid is already set to >> 5e93fa7c-b6f7-4684-981b-bf73254bd87a >> 2014-06-27 16:35:49.948589 7f143254e7a0 1 >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) leveldb db exists/created >> 2014-06-27 16:35:49.948865 7f143254e7a0 -1 >> filestore(/var/lib/ceph/tmp/mnt.9VZHpR) mkjournal error creating journal on >> /var/lib/ceph/tmp/mnt.9VZHpR/journal: (2) No such file or directory >> 2014-06-27 16:35:49.948893 7f143254e7a0 -1 OSD::mkfs: ObjectStore::mkfs >> failed with error -2 >> 2014-06-27 16:35:49.948957 7f143254e7a0 -1 ** ERROR: error creating empty >> object store in /var/lib/ceph/tmp/mnt.9VZHpR: (2) No such file or directory >> [ceph@ceph02 ~]$ >> >> [ceph@ceph02 ~]$ sudo ceph-disk list >> /dev/xvda1 other, ext4, mounted on / >> /dev/xvdb : >> /dev/xvdb1 ceph data, prepared, cluster ceph, osd.0, journal /dev/xvdb2 >> /dev/xvdb2 ceph journal, for /dev/xvdb1 >> >> >> Thanks in advance, I >> >> >> >> 2014-06-27 15:30 GMT+02:00 Alfredo Deza <[email protected]>: >> >>> Can you paste the full ceph-deploy logs? there are a few reasons why >>> this might be happening. >>> >>> >>> >>> On Fri, Jun 27, 2014 at 6:42 AM, Iban Cabrillo <[email protected]> >>> wrote: >>> > Hi, >>> > >>> > I am a little frustrated. After 6 times trying to deploy a test ceph >>> > always >>> > get the same error in the osd activation stage. >>> > The version is firefly (for el6 repo), 3 mons, 3 osds all of then Xen >>> > VMs. >>> > >>> > The mons wake up correctly and I do not know why two osd servers too >>> > after >>> > a lot of errors, always the same >>> > >>> > #ceph-deploy --verbose osd prepare ceph02:xvdb (works fine) >>> > >>> > [root@ceph02 ~]# parted /dev/xvdb >>> > GNU Parted 2.1 >>> > Using /dev/xvdb >>> > Welcome to GNU Parted! Type 'help' to view a list of commands. >>> > (parted) p >>> > Model: Xen Virtual Block Device (xvd) >>> > Disk /dev/xvdb: 107GB >>> > Sector size (logical/physical): 512B/512B >>> > Partition Table: gpt >>> > >>> > Number Start End Size File system Name Flags >>> > 2 1049kB 10,7GB 10,7GB xfs ceph journal >>> > 1 10,7GB 107GB 96,6GB xfs ceph data >>> > >>> > >>> > But the activate give us this error: >>> > >>> > ceph-deploy --verbose osd activate ceph02:xvdb1:/dev/xvdb2 >>> > >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750160 7f123b33d7a0 -1 >>> > filestore(/var/lib/ceph/tmp/mnt.HacFAP) mkjournal error creating >>> > journal on >>> > /var/lib/ceph/tmp/mnt.HacFAP/journal: (2) No such file or directory >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750281 7f123b33d7a0 -1 OSD::mkfs: >>> > ObjectStore::mkfs failed with error -2 >>> > [ceph02][WARNIN] 2014-06-27 12:27:34.750416 7f123b33d7a0 -1 ** ERROR: >>> > error >>> > creating empty object store in /var/lib/ceph/tmp/mnt.HacFAP: (2) No >>> > such >>> > file or directory >>> > [ceph02][WARNIN] ERROR:ceph-disk:Failed to activate >>> > >>> > Two of then following the same procedure after the same error several >>> > time, >>> > raise up. >>> > [ceph@ceph03 ~]$ df -h >>> > Filesystem Size Used Avail Use% Mounted on >>> > /dev/xvda1 5.0G 2.2G 2.6G 46% / >>> > tmpfs 935M 0 935M 0% /dev/shm >>> > /dev/xvdb1 90G 37M 90G 1% /var/lib/ceph/osd/ceph-6 >>> > >>> > >>> > Any idea please? >>> > >>> > >>> > Bertrand Russell: >>> > "El problema con el mundo es que los estúpidos están seguros de todo y >>> > los >>> > inteligentes están llenos de dudas" >>> > >>> > _______________________________________________ >>> > ceph-users mailing list >>> > [email protected] >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> > >> >> > _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
