Hi Markus
You should define the "osd device" and "host" then make ceph cluster work.
Take the types in your map (osd, host, chasis.....root) to design the
crushmap according to your needed.
Example:
host node1 {
id -1
alg straw
hash 0
item osd.0 weight 1.00
item osd.1 weight 1.00
}
host node2 {
id -2
alg straw
hash 0
item osd.2 weight 1.00
item osd.3 weight 1.00
}
root default {
id 0
alg straw
hash 0
item node1 weight 2.00 (sum of its item)
item node2 weight 2.00
}
Then you can use default ruleset. It is set to take the root "default".
2016-03-21 19:50 GMT+08:00 Markus Goldberg <goldb...@uni-hildesheim.de>:
> Hi desmond,
> this is my decompile_map:
> root@bd-a:/etc/ceph# cat decompile_map
> # begin crush map
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
> tunable straw_calc_version 1
>
> # devices
>
> # types
> type 0 osd
> type 1 host
> type 2 chassis
> type 3 rack
> type 4 row
> type 5 pdu
> type 6 pod
> type 7 room
> type 8 datacenter
> type 9 region
> type 10 root
>
> # buckets
> root default {
> id -1 # do not change unnecessarily
> # weight 0.000
> alg straw
> hash 0 # rjenkins1
> }
>
> # rules
> rule replicated_ruleset {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
>
> # end crush map
> root@bd-a:/etc/ceph#
>
> How should i change It?
> I never had to edit anything in this area in former versions of ceph. Has
> something changed?
> Is any new parameter nessessary in ceph.conf while installing?
>
> Thank you,
> Markus
>
> Am 21.03.2016 um 10:34 schrieb 施柏安:
>
> It seems that there no setting weight to all of your osd. So the pg stuck
> in creating.
> you can use some command to edit crushmap for setting weight:
>
> # ceph osd getcrushmap -o map
> # crushtool -d map -o decompile_map
> # vim decompile_map (then you can change the weight to all of your osd and
> its host weight)
> # crushtool -c decompile_map -o changed_map
> # ceph osd setcrushmap -i changed_map
>
> Then, it should work in your situation.
>
>
> 2016-03-21 17:20 GMT+08:00 Markus Goldberg <goldb...@uni-hildesheim.de>:
>
>> Hi,
>> root@bd-a:~# ceph osd tree
>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> -1 0 root default
>> 0 0 osd.0 down 0 1.00000
>> 1 0 osd.1 down 0 1.00000
>> 2 0 osd.2 down 0 1.00000
>> ... delete all the other OSDs as they are the same
>> ...
>> 88 0 osd.88 down 0 1.00000
>> 89 0 osd.89 down 0 1.00000
>> root@bd-a:~#
>>
>> bye,
>> Markus
>>
>> Am 21.03.2016 um 10:10 schrieb 施柏安:
>>
>> What's your crushmap show? Or command 'ceph osd tree' show.
>>
>> 2016-03-21 16:39 GMT+08:00 Markus Goldberg < <goldb...@uni-hildesheim.de>
>> goldb...@uni-hildesheim.de>:
>>
>>> Hi,
>>> i have upgraded my hardware and installed ceph totally new as described
>>> in <http://docs.ceph.com/docs/master/rados/deployment/>
>>> http://docs.ceph.com/docs/master/rados/deployment/
>>> The last job was creating the OSDs
>>> <http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/>
>>> http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/
>>> I have used the create command and after that, the OSDs should be in and
>>> up but they are all down and out.
>>> An additionally osd activate command does not help.
>>>
>>> Ubuntu 14.04.4 kernel 4.2.1
>>> ceph 10.0.2
>>>
>>> What should i do, where is my mistake?
>>>
>>> This is ceph.conf:
>>>
>>> [global]
>>> fsid = 122e929a-111b-4067-80e4-3fef39e66ecf
>>> mon_initial_members = bd-0, bd-1, bd-2
>>> mon_host = xxx.xxx.xxx.20,xxx.xxx.xxx.21,xxx.xxx.xxx.22
>>> auth_cluster_required = cephx
>>> auth_service_required = cephx
>>> auth_client_required = cephx
>>> public network = xxx.xxx.xxx.0/24
>>> cluster network = 192.168.1.0/24
>>> osd_journal_size = 10240
>>> osd pool default size = 2
>>> osd pool default min size = 1
>>> osd pool default pg num = 333
>>> osd pool default pgp num = 333
>>> osd crush chooseleaf type = 1
>>> osd_mkfs_type = btrfs
>>> osd_mkfs_options_btrfs = -f -n 32k -l 32k
>>> osd_mount_options_btrfs = rw,noatime,nodiratime,autodefrag
>>> mds_max_file_size = 50000000000000
>>>
>>>
>>> This is the log of the last osd:
>>> ##########
>>> bd-2:/dev/sdaf:/dev/sdaf2
>>> ceph-deploy disk zap bd-2:/dev/sdaf
>>> [ceph_deploy.conf][DEBUG ] found configuration file at:
>>> /root/.cephdeploy.conf
>>> [ceph_deploy.cli][INFO ] Invoked (1.5.31): /usr/bin/ceph-deploy osd
>>> create --fs-type btrfs bd-2:/dev/sdaf:/dev/sdaf2
>>> [ceph_deploy.cli][INFO ] ceph-deploy options:
>>> [ceph_deploy.cli][INFO ] username : None
>>> [ceph_deploy.cli][INFO ] disk : [('bd-2',
>>> '/dev/sdaf', '/dev/sdaf2')]
>>> [ceph_deploy.cli][INFO ] dmcrypt : False
>>> [ceph_deploy.cli][INFO ] verbose : False
>>> [ceph_deploy.cli][INFO ] overwrite_conf : False
>>> [ceph_deploy.cli][INFO ] subcommand : create
>>> [ceph_deploy.cli][INFO ] dmcrypt_key_dir :
>>> /etc/ceph/dmcrypt-keys
>>> [ceph_deploy.cli][INFO ] quiet : False
>>> [ceph_deploy.cli][INFO ] cd_conf :
>>> <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f944e197488>
>>> [ceph_deploy.cli][INFO ] cluster : ceph
>>> [ceph_deploy.cli][INFO ] fs_type : btrfs
>>> [ceph_deploy.cli][INFO ] func : <function osd
>>> at 0x7f944e16b500>
>>> [ceph_deploy.cli][INFO ] ceph_conf : None
>>> [ceph_deploy.cli][INFO ] default_release : False
>>> [ceph_deploy.cli][INFO ] zap_disk : False
>>> [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
>>> bd-2:/dev/sdaf:/dev/sdaf2
>>> [bd-2][DEBUG ] connected to host: bd-2
>>> [bd-2][DEBUG ] detect platform information from remote host
>>> [bd-2][DEBUG ] detect machine type
>>> [bd-2][DEBUG ] find the location of an executable
>>> [ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
>>> [ceph_deploy.osd][DEBUG ] Deploying osd to bd-2
>>> [bd-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>>> [ceph_deploy.osd][DEBUG ] Preparing host bd-2 disk /dev/sdaf journal
>>> /dev/sdaf2 activate True
>>> [bd-2][INFO ] Running command: ceph-disk -v prepare --cluster ceph
>>> --fs-type btrfs -- /dev/sdaf /dev/sdaf2
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>>> --check-allows-journal -i 0 --cluster ceph
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>>> --check-wants-journal -i 0 --cluster ceph
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>>> --check-needs-journal -i 0 --cluster ceph
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2 uuid path is
>>> /sys/dev/block/65:242/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2 uuid path is
>>> /sys/dev/block/65:242/dm/uuid
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>>> --cluster=ceph --show-config-value=fsid
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>>> --cluster=ceph --name=osd. --lookup osd_mkfs_options_btrfs
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>>> --cluster=ceph --name=osd. --lookup osd_mount_options_btrfs
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-osd
>>> --cluster=ceph --show-config-value=osd_journal_size
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>>> --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>>> --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /usr/bin/ceph-conf
>>> --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2 uuid path is
>>> /sys/dev/block/65:242/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Journal /dev/sdaf2 is a partition
>>> [bd-2][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if
>>> journal is not the same device as the osd data
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2 uuid path is
>>> /sys/dev/block/65:242/dm/uuid
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk -i 2
>>> /dev/sdaf
>>> [bd-2][WARNIN] WARNING:ceph-disk:Journal /dev/sdaf2 was not prepared
>>> with ceph-disk. Symlinking directly.
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Creating osd partition on /dev/sdaf
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk
>>> --largest-new=1 --change-name=1:ceph data
>>> --partition-guid=1:c9486257-e53d-40b8-b7f6-3d228d0cb1f7
>>> --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/sdaf
>>> [bd-2][DEBUG ] The operation has completed successfully.
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Calling partprobe on created device
>>> /dev/sdaf
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdaf
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Creating btrfs fs on /dev/sdaf1
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/mkfs -t btrfs -f -n
>>> 32k -l 32k -- /dev/sdaf1
>>> [bd-2][WARNIN] Turning ON incompat feature 'extref': increased hardlink
>>> limit per file to 65536
>>> [bd-2][DEBUG ]
>>> [bd-2][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL
>>> [bd-2][DEBUG ] WARNING! - see <http://btrfs.wiki.kernel.org>
>>> http://btrfs.wiki.kernel.org before using
>>> [bd-2][DEBUG ]
>>> [bd-2][DEBUG ] fs created label (null) on /dev/sdaf1
>>> [bd-2][DEBUG ] nodesize 32768 leafsize 32768 sectorsize 4096 size
>>> 3.63TiB
>>> [bd-2][DEBUG ] Btrfs v3.12
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdaf1 on
>>> /var/lib/ceph/tmp/mnt.lW5X6l with options rw,noatime,nodiratime,autodefrag
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/mount -t btrfs -o
>>> rw,noatime,nodiratime,autodefrag -- /dev/sdaf1 /var/lib/ceph/tmp/mnt.lW5X6l
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Preparing osd data dir
>>> /var/lib/ceph/tmp/mnt.lW5X6l
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Creating symlink
>>> /var/lib/ceph/tmp/mnt.lW5X6l/journal -> /dev/sdaf2
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/chown -R ceph:ceph
>>> /var/lib/ceph/tmp/mnt.lW5X6l/ceph_fsid.35649.tmp
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/chown -R ceph:ceph
>>> /var/lib/ceph/tmp/mnt.lW5X6l/fsid.35649.tmp
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/chown -R ceph:ceph
>>> /var/lib/ceph/tmp/mnt.lW5X6l/magic.35649.tmp
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/chown -R ceph:ceph
>>> /var/lib/ceph/tmp/mnt.lW5X6l
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.lW5X6l
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /bin/umount --
>>> /var/lib/ceph/tmp/mnt.lW5X6l
>>> [bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf uuid path is
>>> /sys/dev/block/65:240/dm/uuid
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/sgdisk
>>> --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdaf
>>> [bd-2][DEBUG ] The operation has completed successfully.
>>> [bd-2][WARNIN] DEBUG:ceph-disk:Calling partprobe on prepared device
>>> /dev/sdaf
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sdaf
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm settle
>>> [bd-2][WARNIN] INFO:ceph-disk:Running command: /sbin/udevadm trigger
>>> --action=add --sysname-match sdaf1
>>> [bd-2][INFO ] checking OSD status...
>>> [bd-2][INFO ] Running command: ceph --cluster=ceph osd stat
>>> --format=json
>>> [bd-2][WARNIN] there are 90 OSDs down
>>> [bd-2][WARNIN] there are 90 OSDs out
>>> [ceph_deploy.osd][DEBUG ] Host bd-2 is now ready for osd use.
>>> root@bd-a:/etc/ceph#
>>>
>>>
>>> root@bd-a:/etc/ceph# ceph -s
>>> cluster 122e929a-111b-4067-80e4-3fef39e66ecf
>>> health HEALTH_WARN
>>> 64 pgs stuck inactive
>>> 64 pgs stuck unclean
>>> monmap e1: 3 mons at
>>> {bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}
>>> election epoch 6, quorum 0,1,2 bd-0,bd-1,bd-2
>>> osdmap e91: 90 osds: 0 up, 0 in
>>> flags sortbitwise
>>> pgmap v92: 64 pgs, 1 pools, 0 bytes data, 0 objects
>>> 0 kB used, 0 kB / 0 kB avail
>>> 64 creating
>>> root@bd-a:/etc/ceph#
>>>
>>> --
>>> MfG,
>>> Markus Goldberg
>>>
>>>
>>> --------------------------------------------------------------------------
>>> Markus Goldberg Universität Hildesheim
>>> Rechenzentrum
>>> Tel +49 5121 88392822 <%2B49%205121%2088392822> Universitätsplatz 1,
>>> D-31141 Hildesheim, Germany
>>> Fax +49 5121 88392823 <%2B49%205121%2088392823> email
>>> <goldb...@uni-hildesheim.de>goldb...@uni-hildesheim.de
>>>
>>> --------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>>
>> --
>> MfG,
>> Markus Goldberg
>>
>> --------------------------------------------------------------------------
>> Markus Goldberg Universität Hildesheim
>> Rechenzentrum
>> Tel +49 5121 88392822 Universitätsplatz 1, D-31141 Hildesheim, Germany
>> Fax +49 5121 88392823 email goldb...@uni-hildesheim.de
>> --------------------------------------------------------------------------
>>
>>
>
>
> --
> MfG,
> Markus Goldberg
>
> --------------------------------------------------------------------------
> Markus Goldberg Universität Hildesheim
> Rechenzentrum
> Tel +49 5121 88392822 Universitätsplatz 1, D-31141 Hildesheim, Germany
> Fax +49 5121 88392823 email goldb...@uni-hildesheim.de
> --------------------------------------------------------------------------
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
