A few more minor nits.
(1) at the "ceph-deploy admin ..." step:
...
[ceph-admin][DEBUG ] connected to host: ceph-admin
[ceph-admin][DEBUG ] detect platform information from remote host
[ceph-admin][DEBUG ] detect machine type
[ceph-admin][DEBUG ] get remote short hostname
[ceph-admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][ERROR ] RuntimeError: /etc/ceph/ does not exist -
could not write config
[ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts
Easily fixed by mkdir /etc/ceph on the admin node, and then re-running
the command.
(2) "ceph health" doesn't work on the admin node, as the binary doesn't
exist there.
# ceph health
The program 'ceph' is currently not installed. You can install it by
typing:
apt-get install ceph-common
I had prepared the admin node as per
http://ceph.com/docs/master/start/quick-start-preflight/#ceph-node-setup
which said to install ceph-deploy, but not ceph-common.
(3) "ceph health" says:
HEALTH_WARN 2 near full osd(s)
Indeed these 8GB VMs are 97% full. It seems there is a 5GB file
/var/local/osdX/journal:
$ ls -lh /var/local/osd0/journal
-rw-r--r-- 1 root staff 5.0G Apr 3 15:58 /var/local/osd0/journal
This is probably worth a warning in the installation notes. If you are
going to use VMs, make them at least 10-15GB each!
(4) Trying to create two more monitors bombs out:
# ceph-deploy mon create node2 node3
[ceph_deploy.cli][INFO ] Invoked (1.4.0): /usr/bin/ceph-deploy mon
create node2 node3
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node2 node3
[ceph_deploy.mon][DEBUG ] detecting platform for host node2 ...
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Ubuntu 12.04 precise
[node2][DEBUG ] determining if provided host has same hostname in remote
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] deploying mon to node2
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] remote hostname: node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][DEBUG ] create the mon path if it does not exist
[node2][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node2/done
[node2][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node2/done
[node2][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-node2.mon.keyring
[node2][DEBUG ] create the monitor keyring file
[node2][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i node2
--keyring /var/lib/ceph/tmp/ceph-node2.mon.keyring
[node2][DEBUG ] ceph-mon: set fsid to 989824c1-d565-46dd-8ae2-8c7d59fc7058
[node2][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node2
for mon.node2
[node2][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-node2.mon.keyring
[node2][DEBUG ] create a done file to avoid re-doing the mon deployment
[node2][DEBUG ] create the init path if it does not exist
[node2][DEBUG ] locating the `service` executable...
[node2][INFO ] Running command: initctl emit ceph-mon cluster=ceph id=node2
[node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions:
[Errno 2] No such file or directory
[node2][WARNIN] monitor: mon.node2, might not be running yet
[node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions:
[Errno 2] No such file or directory
[node2][WARNIN] node2 is not defined in `mon initial members`
[node2][WARNIN] monitor node2 does not exist in monmap
[node2][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[node2][WARNIN] monitors may not be able to form quorum
[ceph_deploy.mon][DEBUG ] detecting platform for host node3 ...
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: Ubuntu 12.04 precise
[node3][DEBUG ] determining if provided host has same hostname in remote
[node3][DEBUG ] get remote short hostname
[node3][DEBUG ] deploying mon to node3
[node3][DEBUG ] get remote short hostname
[node3][DEBUG ] remote hostname: node3
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node3][DEBUG ] create the mon path if it does not exist
[node3][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node3/done
[node3][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node3/done
[node3][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-node3.mon.keyring
[node3][DEBUG ] create the monitor keyring file
[node3][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i node3
--keyring /var/lib/ceph/tmp/ceph-node3.mon.keyring
[node3][DEBUG ] ceph-mon: set fsid to 989824c1-d565-46dd-8ae2-8c7d59fc7058
[node3][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node3
for mon.node3
[node3][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-node3.mon.keyring
[node3][DEBUG ] create a done file to avoid re-doing the mon deployment
[node3][DEBUG ] create the init path if it does not exist
[node3][DEBUG ] locating the `service` executable...
[node3][INFO ] Running command: initctl emit ceph-mon cluster=ceph id=node3
[node3][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.node3.asok mon_status
[node3][ERROR ] admin_socket: exception getting command descriptions:
[Errno 2] No such file or directory
[node3][WARNIN] monitor: mon.node3, might not be running yet
[node3][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.node3.asok mon_status
[node3][ERROR ] admin_socket: exception getting command descriptions:
[Errno 2] No such file or directory
[node3][WARNIN] node3 is not defined in `mon initial members`
[node3][WARNIN] monitor node3 does not exist in monmap
[node3][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[node3][WARNIN] monitors may not be able to form quorum
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
The admin node doesn't have a /var/run/ceph directory. The other three
nodes have these:
brian@node1:~$ ls /var/run/ceph
ceph-mds.node1.asok ceph-mon.node1.asok ceph-osd.2.asok
brian@node2:~$ ls /var/run/ceph
ceph-osd.0.asok
brian@node3:~$ ls /var/run/ceph
ceph-osd.1.asok
Only one monitor is seen:
brian@node1:~$ ceph quorum_status --format json-pretty
{ "election_epoch": 1,
"quorum": [
0],
"quorum_names": [
"node1"],
"quorum_leader_name": "node1",
"monmap": { "epoch": 1,
"fsid": "989824c1-d565-46dd-8ae2-8c7d59fc7058",
"modified": "0.000000",
"created": "0.000000",
"mons": [
{ "rank": 0,
"name": "node1",
"addr": "192.168.56.21:6789\/0"}]}}
On node2, /var/log/ceph/ceph-mon.node2.log contains repeatedly:
2014-04-03 16:10:45.042949 7f1e7f753780 0 ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 10867
2014-04-03 16:10:45.054943 7f1e7f753780 0 mon.node2 does not exist in
monmap, will attempt to join an existing cluster
2014-04-03 16:10:45.055196 7f1e7f753780 -1 no public_addr or
public_network specified, and mon.node2 not present in monmap or ceph.conf
2014-04-03 16:10:45.102269 7eff0b9c7780 0 ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 10876
2014-04-03 16:10:45.119786 7eff0b9c7780 0 mon.node2 does not exist in
monmap, will attempt to join an existing cluster
2014-04-03 16:10:45.120113 7eff0b9c7780 -1 no public_addr or
public_network specified, and mon.node2 not present in monmap or ceph.conf
But other than that, things are looking good so far with basic
put/remove objects.
Regards,
Brian.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
