I was messing around with my storage cluster this evening and wanted to added a new node to it. I wasn't too sure how to actually do it and didn't find any info on it on the wiki so I tried to figure out how to do it. However, I ran in to a few issues so I thought I'd ask for the correct way to add a new node to a storage cluster.
Currently, I have 2 nodes in the storage cluster and I wanted to add a new node to the storage cluster. Information from the monitor was as follows: # cmonctl osd stat mon0 <- 'osd stat' mon0 -> 'e4: 2 osds: 2 up, 2 in' (0) # cmonctl osd getmaxosd mon0 <- 'osd getmaxosd' mon0 -> 'max_osd = 2 in epoch 4' (0) # So I increased the max_osd parameter and then initialized the object store on the new storage node and started up the osd daemon. # cmonctl osd setmaxosd 3 mon0 <- 'osd setmaxosd 3' mon0 -> 'set new max_osd = 3' (0) # cosd --mkfs_for_osd 2 /mnt/btrfs/osd2 080803.122127.676683 filestore(/mnt/btrfs/osd2) mkfs in /mnt/btrfs/osd2 080803.122127.725061 filestore(/mnt/btrfs/osd2) mkfs done in /mnt/btrfs/osd2 080803.122127.725313 filestore(/mnt/btrfs/osd2) mount detected btrfs created object store for osd2 fsid 14194a0f1f8a2419.5a35d8576aa6ea7c on /mnt/btrfs/osd2 # cosd /mnt/btrfs/osd2 -d --debug_osd 10 starting osd2 at 0.0.0.0:50110/3056/0 dev /mnt/btrfs/osd2 # I had expected the OSD to send the monitor an OSDBoot message which I see in the log file for osd2 080803.122314.210036 osd2 0 do_mon_report 080803.122314.210076 osd2 0 send_boot to mon0 080803.122314.210489 osd2 0 send_pg_stats 080803.122314.473473 osd2 0 heartbeat: stat(080803.122314.473366 rdlat=0 / 0 fshedin=0) However looking at the monitor's log file, I see the following: 080803.041119.037116 1108556096 -- 192.168.221.137:12345/0/0 mon0 --> mds0 192.168.221.137:33808/3125/0 -- mdsbeacon(up:active seq 255) -- 0x74bcb0 080803.041120.109373 1108556096 -- 192.168.221.137:12345/0/0 mon0 <== osd2 192.168.221.140:50110/3056/0 ==== osd_boot(osd2) ==== 0x74b460 080803.041120.109424 mon0(leader).osd e6 preprocess_query osd_boot(osd2) from osd2 192.168.221.140:50110/3056/0 080803.041120.109435 mon0(leader).osd e6 preprocess_boot from osd2 192.168.221.140:50110/3056/0 080803.041120.109443 mon0(leader).osd e6 prepare_update osd_boot(osd2) from osd2 192.168.221.140:50110/3056/0 080803.041120.109452 mon0(leader).osd e6 prepare_boot from osd2 192.168.221.140:50110/3056/0 080803.041120.109459 mon0(leader).osd e6 boot from non-existent osd2, increase max_osd? And cmonctl tells me I still only have 2 osd's: # cmonctl osd stat mon0 <- 'osd stat' mon0 -> 'e5: 2 osds: 2 up, 2 in' (0) # So my new node is not joining the storage cluster. I was under the impression that the storage node would join the cluster after contacting the monitor with the OSDBoot message (after the monitor sends incremental updates to the new node). Is there something else I need to do here? Do I need to explicitly mark osd2 as in the distribution? I'm under the impression from these log messages that the cluster map does not know about the new storage node but I don't know how to add it to the cluster map explicitly. -Padraig ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel
