Let me just clarify... the prepare process created all 10 partitions in sdg the thing is that only 2 (sdg1, sdg2) would be present in /dev. The partx bit is just a hack as I am not familiar with the entire sequence. Initially I was deploying this test cluster in 5 nodes, each with 10 spinners, 1 OS spinner, 1 ssd for journal. *All* nodes would only bring up the first 2 osds.
>From the start the partitions for journals are there: ~]# parted /dev/sdg GNU Parted 2.1 Using /dev/sdg Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: ATA Samsung SSD 840 (scsi) Disk /dev/sdg: 512GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 4295MB 4294MB ceph journal 2 4296MB 8590MB 4294MB ceph journal 3 8591MB 12.9GB 4294MB ceph journal 4 12.9GB 17.2GB 4294MB ceph journal 5 17.2GB 21.5GB 4294MB ceph journal 6 21.5GB 25.8GB 4294MB ceph journal 7 25.8GB 30.1GB 4294MB ceph journal 8 30.1GB 34.4GB 4294MB ceph journal 9 34.4GB 38.7GB 4294MB ceph journal 10 38.7GB 42.9GB 4294MB ceph journal After partx all the entries show up under /dev and I have been able to install the cluster successfully. The only weirdness happened with only one node. Not everything was entirely active+clean. That got resolved after I added the 2nd node. At the moment with 3 nodes: 2013-08-08 17:38:38.328991 mon.0 [INF] pgmap v412: 192 pgs: 192 active+clean; 9518 bytes data, 1153 MB used, 83793 GB / 83794 GB avail Thanks, On Thu, Aug 8, 2013 at 8:17 AM, Sage Weil <[email protected]> wrote: > On Wed, 7 Aug 2013, Tren Blackburn wrote: > > On Tue, Aug 6, 2013 at 11:14 AM, Joao Pedras <[email protected]> wrote: > > Greetings all. > > I am installing a test cluster using one ssd (/dev/sdg) to hold the > > journals. Ceph's version is 0.61.7 and I am using ceph-deploy obtained > > from ceph's git yesterday. This is on RHEL6.4, fresh install. > > > > When preparing the first 2 drives, sda and sdb, all goes well and the > > journals get created in sdg1 and sdg2: > > > > $> ceph-deploy osd prepare ceph00:sda:sdg ceph00:sdb:sdg > > [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks > > ceph00:/dev/sda:/dev/sdg ceph00:/dev/sdb:/dev/sdg > > [ceph_deploy.osd][DEBUG ] Deploying osd to ceph00 > > [ceph_deploy.osd][DEBUG ] Host ceph00 is now ready for osd use. > > [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sda journal > > /dev/sdg activate False > > [ceph_deploy.osd][DEBUG ] Preparing host ceph00 disk /dev/sdb journal > > /dev/sdg activate False > > > > When preparing sdc or any disk after the first 2 I get the following > > in that osd's log but no errors on ceph-deploy: > > > > # tail -f /var/log/ceph/ceph-osd.2.log > > 2013-08-06 10:51:36.655053 7f5ba701a780 0 ceph version 0.61.7 > > (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid > > 11596 > > 2013-08-06 10:51:36.658671 7f5ba701a780 1 > > filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs in > > /var/lib/ceph/tmp/mnt.i2NK47 > > 2013-08-06 10:51:36.658697 7f5ba701a780 1 > > filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkfs fsid is already set to > > 5d1beb09-1f80-421d-a88c-57789e2fc33e > > 2013-08-06 10:51:36.813783 7f5ba701a780 1 > > filestore(/var/lib/ceph/tmp/mnt.i2NK47) leveldb db exists/created > > 2013-08-06 10:51:36.813964 7f5ba701a780 -1 journal FileJournal::_open: > > disabling aio for non-block journal. Use journal_force_aio to force > > use of aio anyway > > 2013-08-06 10:51:36.813999 7f5ba701a780 1 journal _open > > /var/lib/ceph/tmp/mnt.i2NK47/journal fd 10: 0 bytes, block size 4096 > > bytes, directio = 1, aio = 0 > > 2013-08-06 10:51:36.814035 7f5ba701a780 -1 journal check: ondisk fsid > > 00000000-0000-0000-0000-000000000000 doesn't match expected > > 5d1beb09-1f80-421d-a88c-57789e2fc33e, invalid (someone else's?) > > journal > > 2013-08-06 10:51:36.814093 7f5ba701a780 -1 > > filestore(/var/lib/ceph/tmp/mnt.i2NK47) mkjournal error creating > > journal on /var/lib/ceph/tmp/mnt.i2NK47/journal: (22) Invalid argument > > 2013-08-06 10:51:36.814125 7f5ba701a780 -1 OSD::mkfs: FileStore::mkfs > > failed with error -22 > > 2013-08-06 10:51:36.814185 7f5ba701a780 -1 ** ERROR: error creating > > empty object store in /var/lib/ceph/tmp/mnt.i2NK47: (22) Invalid > > argument > > > > I have cleaned the disks with dd, zapped them and so forth but this > > always occurs. If doing sdc/sdd first, for example, then sda or > > whatever follows fails with similar errors. > > > > Does anyone have any insight on this issue? > > Very strange! > > What does the partition table look like at this point? Does the joural > nsymlink in the osd data directory point to the right partition/device on > the failing osd? > > sage -- Joao Pedras
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
