Hi ,
I think I figuered it out .. I had to create folder
/var/lib/ceph/osd/ceph-$id and there might be a bug in ceph-disk-prepare I
had to run
--snip--
[root@cc01 ceph]# ceph-deploy osd --zap-disk create cc02:sdb
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd
--zap-disk create cc02:sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc02:/dev/sdb:
[cc02][DEBUG ] connected to host: cc02
[cc02][DEBUG ] detect platform information from remote host
[cc02][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc02
[cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc02][INFO ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None
activate True
[cc02][INFO ] Running command: ceph-disk-prepare --zap-disk --fs-type xfs
--cluster ceph -- /dev/sdb
[cc02][WARNIN] Caution: invalid backup GPT header, but valid main header;
regenerating
[cc02][WARNIN] backup header from main header.
[cc02][WARNIN]
[cc02][WARNIN] Warning! Main and backup partition tables differ! Use the
'c' and 'e' options
[cc02][WARNIN] on the recovery & transformation menu to examine the two
tables.
[cc02][WARNIN]
[cc02][WARNIN] Warning! One or more CRCs don't match. You should repair the
disk!
[cc02][WARNIN]
[cc02][WARNIN] Invalid partition data!
[cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk', '--zap-all',
'--clear', '--mbrtogpt', '--', '/dev/sdb']' returned non-zero exit status 3
[cc02][DEBUG ] GPT data structures destroyed! You may now partition the
disk using fdisk or
[cc02][DEBUG ] other utilities.
[cc02][DEBUG ] Information: Creating fresh partition table; will override
earlier problems!
[cc02][DEBUG ] Non-GPT disk; not saving changes. Use -g to override.
[cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--zap-disk --fs-type xfs --cluster ceph -- /dev/sdb
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
[root@cc01 ceph]# ceph-deploy osd --zap-disk create cc02:sdb
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd
--zap-disk create cc02:sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc02:/dev/sdb:
[cc02][DEBUG ] connected to host: cc02
[cc02][DEBUG ] detect platform information from remote host
[cc02][DEBUG ] detect machine type
[ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc02
[cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc02][INFO ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None
activate True
[cc02][INFO ] Running command: ceph-disk-prepare --zap-disk --fs-type xfs
--cluster ceph -- /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:re-reading known partitions will display
errors
[cc02][WARNIN] BLKPG: Device or resource busy
[cc02][WARNIN] error adding partition 2
[cc02][WARNIN] INFO:ceph-disk:calling partx on prepared device /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:re-reading known partitions will display
errors
[cc02][WARNIN] BLKPG: Device or resource busy
[cc02][WARNIN] error adding partition 1
[cc02][WARNIN] BLKPG: Device or resource busy
[cc02][WARNIN] error adding partition 2
[cc02][DEBUG ] Creating new GPT entries.
[cc02][DEBUG ] GPT data structures destroyed! You may now partition the
disk using fdisk or
[cc02][DEBUG ] other utilities.
[cc02][DEBUG ] The operation has completed successfully.
[cc02][DEBUG ] Setting name!
[cc02][DEBUG ] partNum is 1
[cc02][DEBUG ] REALLY setting name!
[cc02][DEBUG ] The operation has completed successfully.
[cc02][DEBUG ] Setting name!
[cc02][DEBUG ] partNum is 0
[cc02][DEBUG ] REALLY setting name!
[cc02][DEBUG ] The operation has completed successfully.
[cc02][DEBUG ] meta-data=/dev/sdb1 isize=2048 agcount=4,
agsize=30458561 blks
[cc02][DEBUG ] = sectsz=512 attr=2,
projid32bit=0
[cc02][DEBUG ] data = bsize=4096
blocks=121834241, imaxpct=25
[cc02][DEBUG ] = sunit=0 swidth=0 blks
[cc02][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[cc02][DEBUG ] log =internal log bsize=4096 blocks=59489,
version=2
[cc02][DEBUG ] = sectsz=512 sunit=0 blks,
lazy-count=1
[cc02][DEBUG ] realtime =none extsz=4096 blocks=0,
rtextents=0
[cc02][DEBUG ] The operation has completed successfully.
[cc02][INFO ] Running command: udevadm trigger --subsystem-match=block
--action=add
[cc02][INFO ] checking OSD status...
[cc02][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host cc02 is now ready for osd use.
--snip--
To get the ceph health status Ok . I had to readjust pool size
for i in $(rados lspools);do ceph osd pool set $i size 1;done
set pool 0 size to 1
set pool 1 size to 1
set pool 2 size to 1
set pool 3 size to 1
set pool 4 size to 1
set pool 5 size to 1
[root@cc01 ceph]# for i in $(rados lspools);do ceph osd pool set $i size
2;done
set pool 0 size to 2
set pool 1 size to 2
set pool 2 size to 2
set pool 3 size to 2
set pool 4 size to 2
set pool 5 size to 2
[root@cc01 ceph]# ceph -s
cluster 9f951603-0c31-4942-aefd-96f85b5ea908
health HEALTH_OK
monmap e1: 3 mons at {cc01=
172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0},
election epoch 26, quorum 0,1,2 cc01,cc02,cc03
osdmap e64: 2 osds: 2 up, 2 in
pgmap v121: 492 pgs, 6 pools, 0 bytes data, 0 objects
80636 kB used, 928 GB / 928 GB avail
492 active+clean
--snip--
Can I pass these values via ceph.conf ?
On Wed, May 21, 2014 at 4:05 PM, 10 minus <[email protected]> wrote:
> Hi,
>
> I have just started to dabble with ceph - went thru the docs
> http://ceph.com/howto/deploying-ceph-with-ceph-deploy/
>
>
> I have a 3 node setup with 2 nodes for OSD
>
> I use ceph-deploy mechanism.
>
> The ceph init scripts expects that cluster.conf to be ceph.conf . If I
> give any other name the init scripts dont work. So for test purpose Im
> using ceph.conf
>
>
> --ceph.conf--
> [global]
> auth_service_required = cephx
> filestore_xattr_use_omap = true
> auth_client_required = cephx
> auth_cluster_required = cephx
> mon_host = 172.18.1.31,172.18.1.32,172.18.1.33
> mon_initial_members = cc01, cc02, cc03
> fsid = b58e50f1-13a3-4b14-9cff-32b6edd851c9
> --snip--
>
> I managed to get mon deployed but ceph -s returns health error
>
> --snip--
> ceph -s
> cluster b58e50f1-13a3-4b14-9cff-32b6edd851c9
> health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no
> osds
> monmap e1: 3 mons at {cc01=
> 172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0},
> election epoch 4, quorum 0,1,2 cc01,cc02,cc03
> osdmap e1: 0 osds: 0 up, 0 in
> pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects
> 0 kB used, 0 kB / 0 kB avail
> 192 creating
> --snip--
>
> I tried creating two osds. Well they fail too probably has to do with
> health error message.
>
> --snip--
> ceph-deploy osd create cc01:/dev/sdb cc02:/dev/sdb
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /root/.cephdeploy.conf
> [ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy osd create
> cc01:/dev/sdb cc02:/dev/sdb
> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc01:/dev/sdb:
> cc02:/dev/sdb:
> [cc01][DEBUG ] connected to host: cc01
> [cc01][DEBUG ] detect platform information from remote host
> [cc01][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
> [ceph_deploy.osd][DEBUG ] Deploying osd to cc01
> [cc01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [cc01][INFO ] Running command: udevadm trigger --subsystem-match=block
> --action=add
> [ceph_deploy.osd][DEBUG ] Preparing host cc01 disk /dev/sdb journal None
> activate True
> [cc01][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster
> ceph -- /dev/sdb
> [cc01][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
> [cc01][WARNIN] Could not create partition 2 from 10485761 to 10485760
> [cc01][WARNIN] Error encountered; not saving changes.
> [cc01][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
> '--new=2:0:5120M', '--change-name=2:ceph journal',
> '--partition-guid=2:d882631c-0069-4238-86df-9762ad478daa',
> '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
> '/dev/sdb']' returned non-zero exit status 4
> [cc01][DEBUG ] Setting name!
> [cc01][DEBUG ] partNum is 1
> [cc01][DEBUG ] REALLY setting name!
> [cc01][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
> --fs-type xfs --cluster ceph -- /dev/sdb
> [cc02][DEBUG ] connected to host: cc02
> [cc02][DEBUG ] detect platform information from remote host
> [cc02][DEBUG ] detect machine type
> [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final
> [ceph_deploy.osd][DEBUG ] Deploying osd to cc02
> [cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [cc02][INFO ] Running command: udevadm trigger --subsystem-match=block
> --action=add
> [ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None
> activate True
> [cc02][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster
> ceph -- /dev/sdb
> [cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
> [cc02][WARNIN] Could not create partition 2 from 10485761 to 10485760
> [cc02][WARNIN] Error encountered; not saving changes.
> [cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
> '--new=2:0:5120M', '--change-name=2:ceph journal',
> '--partition-guid=2:486c9081-a73c-4906-b97a-c03458feba26',
> '--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
> '/dev/sdb']' returned non-zero exit status 4
> [cc02][DEBUG ] Found valid GPT with corrupt MBR; using GPT and will write
> new
> [cc02][DEBUG ] protective MBR on save.
> [cc02][DEBUG ] Setting name!
> [cc02][DEBUG ] partNum is 1
> [cc02][DEBUG ] REALLY setting name!
> [cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
> --fs-type xfs --cluster ceph -- /dev/sdb
> [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs
> --snip--
>
> Any pointers to fix the issue.
>
> Cheers
>
>
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
