Hi, The previous log is based on cuttlefish. This one is based on bobtail. I'm not using cephx, may be that's what causing the problem?
temp dir is /tmp/mkcephfs.xf5TsinRsL preparing monmap in /tmp/mkcephfs.xf5TsinRsL/monmap /usr/bin/monmaptool --create --clobber --add a 192.168.128.10:6789 --add b 192.168.128.11:6789 --add c 192.168.128.12:6789 --print /tmp/mkcephfs.xf5TsinRsL/monmap /usr/bin/monmaptool: monmap file /tmp/mkcephfs.xf5TsinRsL/monmap /usr/bin/monmaptool: generated fsid 1168e717-5db5-488c-a1f7-0b61e7f19138 epoch 0 fsid 1168e717-5db5-488c-a1f7-0b61e7f19138 last_changed 2013-05-24 09:51:15.012839 created 2013-05-24 09:51:15.012839 0: 192.168.128.10:6789/0 mon.a 1: 192.168.128.11:6789/0 mon.b 2: 192.168.128.12:6789/0 mon.c /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.xf5TsinRsL/monmap (3 monitors) === osd.0 === 2013-05-24 09:51:16.247443 7f19b2ce5780 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2013-05-24 09:51:16.604586 7f19b2ce5780 -1 created object store /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsid 1168e717-5db5-488c-a1f7-0b61e7f19138 2013-05-24 09:51:16.604667 7f19b2ce5780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2013-05-24 09:51:16.604850 7f19b2ce5780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring === osd.1 === pushing conf and monmap to ceph1:/tmp/mkfs.ceph.f0a8d758e9f1a3f32160f67a12149281 2013-05-24 09:50:46.405722 7fd7fa8c5780 -1 filestore(/var/lib/ceph/osd/ceph-1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2013-05-24 09:50:46.750885 7fd7fa8c5780 -1 created object store /var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1 fsid 1168e717-5db5-488c-a1f7-0b61e7f19138 2013-05-24 09:50:46.750945 7fd7fa8c5780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2013-05-24 09:50:46.751120 7fd7fa8c5780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring collecting osd.1 key === osd.2 === pushing conf and monmap to ceph2:/tmp/mkfs.ceph.e07be4351777982bb28d1cc7ab52e01b 2013-05-24 09:51:33.623922 7fa231b87780 -1 filestore(/var/lib/ceph/osd/ceph-2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2013-05-24 09:51:33.859703 7fa231b87780 -1 created object store /var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2 fsid 1168e717-5db5-488c-a1f7-0b61e7f19138 2013-05-24 09:51:33.859772 7fa231b87780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2013-05-24 09:51:33.859930 7fa231b87780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring collecting osd.2 key === mds.a === creating private key for mds.a keyring /var/lib/ceph/mds/ceph-a/keyring creating /var/lib/ceph/mds/ceph-a/keyring === mds.b === pushing conf and monmap to ceph1:/tmp/mkfs.ceph.5031963c92bcc7e98cd4422ee99d1220 creating private key for mds.b keyring /var/lib/ceph/mds/ceph-b/keyring creating /var/lib/ceph/mds/ceph-b/keyring collecting mds.b key === mds.c === pushing conf and monmap to ceph2:/tmp/mkfs.ceph.a7475d8f01e340fc9e410fd60a3d8a80 creating private key for mds.c keyring /var/lib/ceph/mds/ceph-c/keyring creating /var/lib/ceph/mds/ceph-c/keyring collecting mds.c key Building generic osdmap from /tmp/mkcephfs.xf5TsinRsL/conf /usr/bin/osdmaptool: osdmap file '/tmp/mkcephfs.xf5TsinRsL/osdmap' /usr/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.xf5TsinRsL/osdmap Generating admin key at /tmp/mkcephfs.xf5TsinRsL/keyring.admin creating /tmp/mkcephfs.xf5TsinRsL/keyring.admin Building initial monitor keyring added entity mds.a auth auth(auid = 18446744073709551615 key=AQC9455RSHUiFhAAcxXX2jmAnPk69KGQ7rUczA== with 0 caps) added entity mds.b auth auth(auid = 18446744073709551615 key=AQCe455RcAEhCRAARPQhkIuPvYIXDElerq+zJg== with 0 caps) added entity mds.c auth auth(auid = 18446744073709551615 key=AQDM455R0KRfEhAAPByEb/CBqaUK68tssLH/ug== with 0 caps) added entity osd.0 auth auth(auid = 18446744073709551615 key=AQC0455RWKsKJBAANgtn7Xq8g3u1CPXekMCy7g== with 0 caps) added entity osd.1 auth auth(auid = 18446744073709551615 key=AQCW455RQJ7CLBAAQzY2cAMjFwHLd36s1w8m6g== with 0 caps) added entity osd.2 auth auth(auid = 18446744073709551615 key=AQDF455RoDM/MxAAaC/sEyCE4xFpeNHCqkyZSA== with 0 caps) === mon.a === /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a === mon.b === pushing everything to ceph1 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-b for mon.b === mon.c === pushing everything to ceph2 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-c for mon.c placing client.admin keyring in /etc/ceph/keyring 2013-05-24 09:51:36.973708 7f774d118700 0 -- :/21213 >> 192.168.128.11:6789/0 pipe(0x2e16250 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:39.973650 7f7753848700 0 -- :/21213 >> 192.168.128.12:6789/0 pipe(0x7f7744000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:42.974064 7f774d118700 0 -- :/21213 >> 192.168.128.11:6789/0 pipe(0x7f7744003010 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:45.974173 7f7753848700 0 -- :/21213 >> 192.168.128.10:6789/0 pipe(0x7f77440037e0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:48.974462 7f774d118700 0 -- :/21213 >> 192.168.128.11:6789/0 pipe(0x7f7744002340 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:51.974747 7f7753848700 0 -- :/21213 >> 192.168.128.12:6789/0 pipe(0x7f7744002be0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:54.974958 7f774d118700 0 -- :/21213 >> 192.168.128.10:6789/0 pipe(0x7f7744003fb0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:51:57.975324 7f7753848700 0 -- :/21213 >> 192.168.128.12:6789/0 pipe(0x7f7744004850 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:52:00.975561 7f774d118700 0 -- :/21213 >> 192.168.128.10:6789/0 pipe(0x7f77440061b0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:52:03.976149 7f7753848700 0 -- :/21213 >> 192.168.128.11:6789/0 pipe(0x7f7744006b70 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-24 09:52:06.976329 7f774d118700 0 -- :/21213 >> 192.168.128.12:6789/0 pipe(0x7f7744004fd0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault ^C On Thu, May 23, 2013 at 11:41 PM, Dewan Shamsul Alam < [email protected]> wrote: > Hi, > > This is what I get while building the cluster: > > #/sbin/mkcephfs -a -c /etc/ceph/ceph.conf > temp dir is /tmp/mkcephfs.yzl9PFOJYo > preparing monmap in /tmp/mkcephfs.yzl9PFOJYo/monmap > /usr/bin/monmaptool --create --clobber --add a 192.168.128.10:6789 --add > b 192.168.128.11:6789 --add c 192.168.128.12:6789 --print > /tmp/mkcephfs.yzl9PFOJYo/monmap > /usr/bin/monmaptool: monmap file /tmp/mkcephfs.yzl9PFOJYo/monmap > /usr/bin/monmaptool: generated fsid 09136333-16dc-476f-8773-90262ad0b80d > epoch 0 > fsid 09136333-16dc-476f-8773-90262ad0b80d > last_changed 2013-05-23 23:36:41.325667 > created 2013-05-23 23:36:41.325667 > 0: 192.168.128.10:6789/0 mon.a > 1: 192.168.128.11:6789/0 mon.b > 2: 192.168.128.12:6789/0 mon.c > /usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.yzl9PFOJYo/monmap (3 > monitors) > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === osd.0 === > 2013-05-23 23:36:41.851599 7fe81f143780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:42.576549 7fe81f143780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:42.577795 7fe81f143780 -1 > filestore(/var/lib/ceph/osd/ceph-0) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory > 2013-05-23 23:36:42.918456 7fe81f143780 -1 created object store > /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 > fsid 09136333-16dc-476f-8773-90262ad0b80d > 2013-05-23 23:36:42.918520 7fe81f143780 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-0/keyring: can't open > /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory > 2013-05-23 23:36:42.918642 7fe81f143780 -1 created new key in keyring > /var/lib/ceph/osd/ceph-0/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === osd.1 === > pushing conf and monmap to > ceph1:/tmp/mkfs.ceph.HWyfcu95hsnB1jxdVqxGJNAOd2u3aj5I > 2013-05-23 23:36:12.380573 7ff116b63780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:13.026598 7ff116b63780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:13.037762 7ff116b63780 -1 > filestore(/var/lib/ceph/osd/ceph-1) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory > 2013-05-23 23:36:13.366445 7ff116b63780 -1 created object store > /var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1 > fsid 09136333-16dc-476f-8773-90262ad0b80d > 2013-05-23 23:36:13.366510 7ff116b63780 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-1/keyring: can't open > /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory > 2013-05-23 23:36:13.366621 7ff116b63780 -1 created new key in keyring > /var/lib/ceph/osd/ceph-1/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > collecting osd.1 key > === osd.2 === > pushing conf and monmap to > ceph2:/tmp/mkfs.ceph.tNt36unRvZ6lVKmz65OjiOhrpUfsw7xz > 2013-05-23 23:36:59.086209 7fe38a955780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:59.610999 7fe38a955780 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > 2013-05-23 23:36:59.623725 7fe38a955780 -1 > filestore(/var/lib/ceph/osd/ceph-2) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory > 2013-05-23 23:36:59.850510 7fe38a955780 -1 created object store > /var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2 > fsid 09136333-16dc-476f-8773-90262ad0b80d > 2013-05-23 23:36:59.850574 7fe38a955780 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-2/keyring: can't open > /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory > 2013-05-23 23:36:59.850688 7fe38a955780 -1 created new key in keyring > /var/lib/ceph/osd/ceph-2/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > collecting osd.2 key > === mds.a === > creating private key for mds.a keyring /var/lib/ceph/mds/ceph-a/keyring > creating /var/lib/ceph/mds/ceph-a/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === mds.b === > pushing conf and monmap to > ceph1:/tmp/mkfs.ceph.QDX0IZSEBd3469OT6yw6ISlckxfmO6nu > creating private key for mds.b keyring /var/lib/ceph/mds/ceph-b/keyring > creating /var/lib/ceph/mds/ceph-b/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > collecting mds.b key > === mds.c === > pushing conf and monmap to > ceph2:/tmp/mkfs.ceph.XimfAW4CrJR11rs8IAJhsHn0inBNJdhl > creating private key for mds.c keyring /var/lib/ceph/mds/ceph-c/keyring > creating /var/lib/ceph/mds/ceph-c/keyring > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > collecting mds.c key > Building generic osdmap from /tmp/mkcephfs.yzl9PFOJYo/conf > /usr/bin/osdmaptool: osdmap file '/tmp/mkcephfs.yzl9PFOJYo/osdmap' > /usr/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.yzl9PFOJYo/osdmap > Generating admin key at /tmp/mkcephfs.yzl9PFOJYo/keyring.admin > creating /tmp/mkcephfs.yzl9PFOJYo/keyring.admin > Building initial monitor keyring > added entity mds.a auth auth(auid = 18446744073709551615 > key=AQCzU55RqAU9KRAA5OMKKmz5dnXS0REaz5BSsw== with 0 caps) > added entity mds.b auth auth(auid = 18446744073709551615 > key=AQCUU55ReGCdLxAArT4yETLg2In9uCjjuVpkoQ== with 0 caps) > added entity mds.c auth auth(auid = 18446744073709551615 > key=AQDCU55RsKXMDxAAE59u5bGR+GDKznJYmulZmA== with 0 caps) > added entity osd.0 auth auth(auid = 18446744073709551615 > key=AQCqU55RMJi/NhAABfNaC5S63hrcmVL4OfNfRg== with 0 caps) > added entity osd.1 auth auth(auid = 18446744073709551615 > key=AQCNU55RIJfYFRAAtFlwerP2buAVUhm+rwuLIA== with 0 caps) > added entity osd.2 auth auth(auid = 18446744073709551615 > key=AQC7U55ROM6yMhAA9vOYRbYgh5qHtagbAhDwgw== with 0 caps) > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === mon.a === > /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === mon.b === > pushing everything to ceph1 > /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-b for mon.b > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > === mon.c === > pushing everything to ceph2 > /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-c for mon.c > > WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see: > http://github.com/ceph/ceph-deploy > placing client.admin keyring in /etc/ceph/keyring > HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons > down, quorum 1,2 ceph1,ceph2; clock skew detected on mon.ceph2 > > root@ceph0:~/cephscripts# ceph -s > health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no > osds; 1 mons down, quorum 1,2 ceph1,ceph2; clock skew detected on mon.ceph2 > monmap e1: 3 mons at {ceph0= > 192.168.128.10:6789/0,ceph1=192.168.128.11:6789/0,ceph2=192.168.128.12:6789/0}, > election epoch 10, quorum 1,2 ceph1,ceph2 > osdmap e1: 0 osds: 0 up, 0 in > pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB > avail > mdsmap e1: 0/0/1 up > > > > On Thu, May 23, 2013 at 11:31 PM, Sage Weil <[email protected]> wrote: > >> Can you be more specific? How does it fail? >> >> A copy of the actual output would be ideal, thanks! >> >> sage >> >> On Thu, 23 May 2013, Dewan Shamsul Alam wrote: >> >> > Hi, >> > >> > I had a running ceph cluster on bobtail. It was on 0.56.4. It is my test >> > cluster. I upgraded it to 0.56.6, now mkcephfs doesn't work with the >> same >> > working configuration and the following command: >> > >> > /sbin/mkcephfs -a -c /etc/ceph/ceph.conf >> > >> > ceph.conf >> > >> > [global] >> > auth supported = none >> > auth cluster required = none >> > auth service required = none >> > auth client required = none >> > [osd] >> > osd journal size = 1000 >> > filestore xattr use omap = true >> > osd mkfs type = btrfs >> > osd mkfs options btrfs = -m raid0 >> > osd mount options btrfs = rw, noatime >> > [mon.a] >> > host = ceph0 >> > mon addr = 192.168.128.10:6789 >> > [mon.b] >> > host = ceph1 >> > mon addr = 192.168.128.11:6789 >> > [mon.c] >> > host = ceph2 >> > mon addr = 192.168.128.12:6789 >> > [osd.0] >> > host = ceph0 >> > devs = /dev/sda3 >> > [osd.1] >> > host = ceph1 >> > devs = /dev/sda3 >> > [osd.2] >> > host = ceph2 >> > devs = /dev/sda3 >> > [mds.a] >> > host = ceph0 >> > [mds.b] >> > host = ceph1 >> > [mds.c] >> > host = ceph2 >> > >> > Best Regards, >> > Dewan Shamsul Alam >> > >> > >> > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
