Hi desmond,
this seems to be much to do for 90 OSDs. And possible a few mistakes in
typing.
Every change of disk needs extra editing too.
This weighting was automatically done in former versions.
Do you know why and where this changed or was i faulty at some point?
Markus
Am 21.03.2016 um 13:28 schrieb 施柏安:
Hi Markus
You should define the "osd device" and "host" then make ceph cluster work.
Take the types in your map (osd, host, chasis.....root) to design the
crushmap according to your needed.
Example:
host node1 {
id -1
alg straw
hash 0
item osd.0 weight 1.00
item osd.1 weight 1.00
}
host node2 {
id -2
alg straw
hash 0
item osd.2 weight 1.00
item osd.3 weight 1.00
}
root default {
id 0
alg straw
hash 0
item node1 weight 2.00 (sum of its item)
item node2 weight 2.00
}
Then you can use default ruleset. It is set to take the root "default".
2016-03-21 19:50 GMT+08:00 Markus Goldberg <goldb...@uni-hildesheim.de
<mailto:goldb...@uni-hildesheim.de>>:
Hi desmond,
this is my decompile_map:
root@bd-a:/etc/ceph# cat decompile_map
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1
# devices
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# buckets
root default {
id -1 # do not change unnecessarily
# weight 0.000
alg straw
hash 0 # rjenkins1
}
# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush map
root@bd-a:/etc/ceph#
How should i change It?
I never had to edit anything in this area in former versions of
ceph. Has something changed?
Is any new parameter nessessary in ceph.conf while installing?
Thank you,
Markus
Am 21.03.2016 um 10:34 schrieb 施柏安:
It seems that there no setting weight to all of your osd. So the
pg stuck in creating.
you can use some command to edit crushmap for setting weight:
# ceph osd getcrushmap -o map
# crushtool -d map -o decompile_map
# vim decompile_map (then you can change the weight to all of
your osd and its host weight)
# crushtool -c decompile_map -o changed_map
# ceph osd setcrushmap -i changed_map
Then, it should work in your situation.
2016-03-21 17:20 GMT+08:00 Markus Goldberg
<goldb...@uni-hildesheim.de <mailto:goldb...@uni-hildesheim.de>>:
Hi,
root@bd-a:~# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0 root default
0 0 osd.0 down 0 1.00000
1 0 osd.1 down 0 1.00000
2 0 osd.2 down 0 1.00000
... delete all the other OSDs as they are the same
...
88 0 osd.88 down 0 1.00000
89 0 osd.89 down 0 1.00000
root@bd-a:~#
bye,
Markus
Am 21.03.2016 um 10:10 schrieb 施柏安:
What's your crushmap show? Or command 'ceph osd tree' show.
2016-03-21 16:39 GMT+08:00 Markus Goldberg
<goldb...@uni-hildesheim.de
<mailto:goldb...@uni-hildesheim.de>>:
Hi,
i have upgraded my hardware and installed ceph totally
new as described in
http://docs.ceph.com/docs/master/rados/deployment/
The last job was creating the OSDs
http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/
I have used the create command and after that, the OSDs
should be in and up but they are all down and out.
An additionally osd activate command does not help.
Ubuntu 14.04.4 kernel 4.2.1
ceph 10.0.2
What should i do, where is my mistake?
This is ceph.conf:
[global]
fsid = 122e929a-111b-4067-80e4-3fef39e66ecf
mon_initial_members = bd-0, bd-1, bd-2
mon_host = xxx.xxx.xxx.20,xxx.xxx.xxx.21,xxx.xxx.xxx.22
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public network = xxx.xxx.xxx.0/24
cluster network = 192.168.1.0/24 <http://192.168.1.0/24>
osd_journal_size = 10240
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1
osd_mkfs_type = btrfs
osd_mkfs_options_btrfs = -f -n 32k -l 32k
osd_mount_options_btrfs = rw,noatime,nodiratime,autodefrag
mds_max_file_size = 50000000000000
This is the log of the last osd:
##########
bd-2:/dev/sdaf:/dev/sdaf2
ceph-deploy disk zap bd-2:/dev/sdaf
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.31):
/usr/bin/ceph-deploy osd create --fs-type btrfs
bd-2:/dev/sdaf:/dev/sdaf2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] disk : [('bd-2', '/dev/sdaf',
'/dev/sdaf2')]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] dmcrypt_key_dir :
/etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at
0x7f944e197488>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : btrfs
[ceph_deploy.cli][INFO ] func : <function osd at
0x7f944e16b500>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks
bd-2:/dev/sdaf:/dev/sdaf2
[bd-2][DEBUG ] connected to host: bd-2
[bd-2][DEBUG ] detect platform information from remote host
[bd-2][DEBUG ] detect machine type
[bd-2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to bd-2
[bd-2][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[ceph_deploy.osd][DEBUG ] Preparing host bd-2 disk
/dev/sdaf journal /dev/sdaf2 activate True
[bd-2][INFO ] Running command: ceph-disk -v prepare
--cluster ceph --fs-type btrfs -- /dev/sdaf /dev/sdaf2
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --check-allows-journal -i 0 --cluster ceph
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2
uuid path is /sys/dev/block/65:242/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2
uuid path is /sys/dev/block/65:242/dm/uuid
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_mkfs_options_btrfs
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_mount_options_btrfs
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-osd --cluster=ceph
--show-config-value=osd_journal_size
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_cryptsetup_parameters
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_dmcrypt_key_size
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
osd_dmcrypt_type
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2
uuid path is /sys/dev/block/65:242/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:Journal /dev/sdaf2 is a
partition
[bd-2][WARNIN] WARNING:ceph-disk:OSD will not be
hot-swappable if journal is not the same device as the
osd data
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf2
uuid path is /sys/dev/block/65:242/dm/uuid
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/sgdisk -i 2 /dev/sdaf
[bd-2][WARNIN] WARNING:ceph-disk:Journal /dev/sdaf2 was
not prepared with ceph-disk. Symlinking directly.
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:Creating osd partition on
/dev/sdaf
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/sgdisk --largest-new=1 --change-name=1:ceph data
--partition-guid=1:c9486257-e53d-40b8-b7f6-3d228d0cb1f7
--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --
/dev/sdaf
[bd-2][DEBUG ] The operation has completed successfully.
[bd-2][WARNIN] DEBUG:ceph-disk:Calling partprobe on
created device /dev/sdaf
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/udevadm settle
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/partprobe /dev/sdaf
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/udevadm settle
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] DEBUG:ceph-disk:Creating btrfs fs on
/dev/sdaf1
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/mkfs -t btrfs -f -n 32k -l 32k -- /dev/sdaf1
[bd-2][WARNIN] Turning ON incompat feature 'extref':
increased hardlink limit per file to 65536
[bd-2][DEBUG ]
[bd-2][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL
[bd-2][DEBUG ] WARNING! - see
http://btrfs.wiki.kernel.org before using
[bd-2][DEBUG ]
[bd-2][DEBUG ] fs created label (null) on /dev/sdaf1
[bd-2][DEBUG ] nodesize 32768 leafsize 32768 sectorsize
4096 size 3.63TiB
[bd-2][DEBUG ] Btrfs v3.12
[bd-2][WARNIN] DEBUG:ceph-disk:Mounting /dev/sdaf1 on
/var/lib/ceph/tmp/mnt.lW5X6l with options
rw,noatime,nodiratime,autodefrag
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/mount -t btrfs -o rw,noatime,nodiratime,autodefrag
-- /dev/sdaf1 /var/lib/ceph/tmp/mnt.lW5X6l
[bd-2][WARNIN] DEBUG:ceph-disk:Preparing osd data dir
/var/lib/ceph/tmp/mnt.lW5X6l
[bd-2][WARNIN] DEBUG:ceph-disk:Creating symlink
/var/lib/ceph/tmp/mnt.lW5X6l/journal -> /dev/sdaf2
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.lW5X6l/ceph_fsid.35649.tmp
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.lW5X6l/fsid.35649.tmp
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/chown -R ceph:ceph
/var/lib/ceph/tmp/mnt.lW5X6l/magic.35649.tmp
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.lW5X6l
[bd-2][WARNIN] DEBUG:ceph-disk:Unmounting
/var/lib/ceph/tmp/mnt.lW5X6l
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/bin/umount -- /var/lib/ceph/tmp/mnt.lW5X6l
[bd-2][WARNIN] DEBUG:ceph-disk:get_dm_uuid /dev/sdaf
uuid path is /sys/dev/block/65:240/dm/uuid
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/sgdisk
--typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d --
/dev/sdaf
[bd-2][DEBUG ] The operation has completed successfully.
[bd-2][WARNIN] DEBUG:ceph-disk:Calling partprobe on
prepared device /dev/sdaf
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/udevadm settle
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/partprobe /dev/sdaf
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/udevadm settle
[bd-2][WARNIN] INFO:ceph-disk:Running command:
/sbin/udevadm trigger --action=add --sysname-match sdaf1
[bd-2][INFO ] checking OSD status...
[bd-2][INFO ] Running command: ceph --cluster=ceph osd
stat --format=json
[bd-2][WARNIN] there are 90 OSDs down
[bd-2][WARNIN] there are 90 OSDs out
[ceph_deploy.osd][DEBUG ] Host bd-2 is now ready for osd
use.
root@bd-a:/etc/ceph#
root@bd-a:/etc/ceph# ceph -s
cluster 122e929a-111b-4067-80e4-3fef39e66ecf
health HEALTH_WARN
64 pgs stuck inactive
64 pgs stuck unclean
monmap e1: 3 mons at
{bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}
election epoch 6, quorum 0,1,2 bd-0,bd-1,bd-2
osdmap e91: 90 osds: 0 up, 0 in
flags sortbitwise
pgmap v92: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating
root@bd-a:/etc/ceph#
--
MfG,
Markus Goldberg
--------------------------------------------------------------------------
Markus Goldberg Universität Hildesheim
Rechenzentrum
Tel +49 5121 88392822 <tel:%2B49%205121%2088392822>
Universitätsplatz 1, D-31141 Hildesheim, Germany
Fax +49 5121 88392823 <tel:%2B49%205121%2088392823>
email goldb...@uni-hildesheim.de
<mailto:goldb...@uni-hildesheim.de>
--------------------------------------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
MfG,
Markus Goldberg
--------------------------------------------------------------------------
Markus Goldberg Universität Hildesheim
Rechenzentrum
Tel+49 5121 88392822 <tel:%2B49%205121%2088392822> Universitätsplatz
1, D-31141 Hildesheim, Germany
Fax+49 5121 88392823 <tel:%2B49%205121%2088392823>
emailgoldb...@uni-hildesheim.de <mailto:goldb...@uni-hildesheim.de>
--------------------------------------------------------------------------
--
MfG,
Markus Goldberg
--------------------------------------------------------------------------
Markus Goldberg Universität Hildesheim
Rechenzentrum
Tel+49 5121 88392822 <tel:%2B49%205121%2088392822> Universitätsplatz 1,
D-31141 Hildesheim, Germany
Fax+49 5121 88392823 <tel:%2B49%205121%2088392823>
emailgoldb...@uni-hildesheim.de <mailto:goldb...@uni-hildesheim.de>
--------------------------------------------------------------------------
--
MfG,
Markus Goldberg
--------------------------------------------------------------------------
Markus Goldberg Universität Hildesheim
Rechenzentrum
Tel +49 5121 88392822 Universitätsplatz 1, D-31141 Hildesheim, Germany
Fax +49 5121 88392823 email goldb...@uni-hildesheim.de
--------------------------------------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
