Hi, did the information i gave sufficient to get a solution ? Thanks Best regards
Le 18/09/2017 à 21:12, Phil Schwarz a écrit :
Thanks for your help, Le 18/09/2017 à 12:37, Alwin Antreich a écrit :On Sun, Sep 17, 2017 at 11:18:51AM +0200, Phil Schwarz wrote:Hi, going on on the same problem (links [1] & [2] ) [1] : https://pve.proxmox.com/pipermail/pve-user/2017-July/168578.html[2] : https://pve.proxmox.com/pipermail/pve-user/2017-September/168775.html-Added a brand new node, updated to last ceph version (the proxmox team recompiled one)Can you please post a 'ceph versions' and a 'ceph osd tree' to get some overview on your setup?root@arya:~# ceph versionceph version 12.2.0 (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)root@arya:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 10.06328 root default -3 0 host daenerys -5 1.81360 host jaime 5 hdd 1.81360 osd.5 up 1.00000 1.00000 -2 6.59999 host jon 1 hdd 4.20000 osd.1 up 1.00000 1.00000 3 hdd 2.39999 osd.3 up 1.00000 1.00000 -4 1.64969 host tyrion 2 hdd 0.44969 osd.2 up 1.00000 1.00000 4 hdd 1.20000 osd.4 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000-plugged a new disk-used GUI (same result with pveceph createosd from cli) to create a new osd(with bluestore feature).PVE GUI and CLI use the same API for managing ceph1. The OSD doesn't appear on gui 2. The OSD is seen as down and out of any node 3. the /var/log/ceph/ceph-osd.admin.log logfile seems to figure a mismatch between filestore and bluestore:Do you see any errors in the mon logs or ceph.log itself?(Jaime is a mon& mgr) root@jaime:~# tail -f /var/log/ceph/ceph-mon.1.log2017-09-18 21:05:00.084847 7f8a1b4a8700 1 mon.1@0(leader).log v2152264 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 0 entries (version 2152264) 2017-09-18 21:05:09.963784 7f8a1868c700 0 mon.1@0(leader).data_health(2028) update_stats avail 90% total 58203 MB, used 2743 MB, avail 52474 MB 2017-09-18 21:05:29.878648 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"} v 0) v1 2017-09-18 21:05:29.878705 7f8a15e87700 0 log_channel(audit) log [INF] : from='client.6392525 -' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]: dispatch 2017-09-18 21:05:29.927377 7f8a1b4a8700 1 mon.1@0(leader).osd e1141 e1141: 6 total, 5 up, 5 in 2017-09-18 21:05:29.932253 7f8a1b4a8700 0 log_channel(audit) log [INF] : from='client.6392525 -' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]': finished 2017-09-18 21:05:29.932388 7f8a1b4a8700 0 log_channel(cluster) log [DBG] : osdmap e1141: 6 total, 5 up, 5 in 2017-09-18 21:05:29.932983 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1 2017-09-18 21:05:29.933040 7f8a15e87700 0 log_channel(audit) log [DBG] : from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2017-09-18 21:05:29.933337 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1 2017-09-18 21:05:29.933383 7f8a15e87700 0 log_channel(audit) log [DBG] : from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2017-09-18 21:05:29.933674 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1 2017-09-18 21:05:29.933692 7f8a15e87700 0 log_channel(audit) log [DBG] : from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2017-09-18 21:05:29.933880 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd metadata", "id": 4} v 0) v1 2017-09-18 21:05:29.933897 7f8a15e87700 0 log_channel(audit) log [DBG] : from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2017-09-18 21:05:29.934062 7f8a15e87700 0 mon.1@0(leader) e4 handle_command mon_command({"prefix": "osd metadata", "id": 5} v 0) v1 2017-09-18 21:05:29.934089 7f8a15e87700 0 log_channel(audit) log [DBG] : from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2017-09-18 21:05:30.113007 7f8a1b4a8700 1 mon.1@0(leader).log v2152265 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 3 entries (version 2152265) 2017-09-18 21:05:31.154227 7f8a1b4a8700 1 mon.1@0(leader).log v2152266 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 0 entries (version 2152266) 2017-09-18 21:05:32.289428 7f8a1b4a8700 1 mon.1@0(leader).log v2152267 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 0 entries (version 2152267) 2017-09-18 21:05:36.782573 7f8a1b4a8700 1 mon.1@0(leader).log v2152268 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 0 entries (version 2152268) 2017-09-18 21:06:09.964314 7f8a1868c700 0 mon.1@0(leader).data_health(2028) update_stats avail 90% total 58203 MB, used 2744 MB, avail 52473 MB 2017-09-18 21:06:20.040930 7f8a1b4a8700 1 mon.1@0(leader).log v2152269 check_sub sending message to client.5804116 10.250.0.23:0/4045099631 with 0 entries (version 2152269)And ceph.log root@jaime:~# tail -f /var/log/ceph/ceph.log2017-09-18 12:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38100 : cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs undersized; Degraded data redundancy (low space): 29 pgs backfill_toofull; application not enabled on 2 pool(s) 2017-09-18 13:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38101 : cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs undersized; Degraded data redundancy (low space): 29 pgs backfill_toofull; application not enabled on 2 pool(s) 2017-09-18 14:00:00.000133 mon.1 mon.0 10.250.0.21:6789/0 38102 : cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs undersized; Degraded data redundancy (low space): 29 pgs backfill_toofull; application not enabled on 2 pool(s)201 Yes, the cluster is not really healthy, indeed .... Thanks2017-09-16 19:12:00.468481 7f6469cdde00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5624 2017-09-16 19:12:00.470154 7f6469cdde00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:00.471408 7f6469cdde00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:00.471727 7f6469cdde00 1 journal close /dev/sdb22017-09-16 19:12:00.471994 7f6469cdde00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:05.042622 7f000b944e00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5702 2017-09-16 19:12:05.066343 7f000b944e00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:05.066549 7f000b944e00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:05.066717 7f000b944e00 1 journal close /dev/sdb22017-09-16 19:12:05.066843 7f000b944e00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:08.198548 7f5740748e00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5767 2017-09-16 19:12:08.223674 7f5740748e00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:08.223831 7f5740748e00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:08.224213 7f5740748e00 1 journal close /dev/sdb22017-09-16 19:12:08.224342 7f5740748e00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:09.149622 7f7b06058e00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5800 2017-09-16 19:12:09.173319 7f7b06058e00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:09.173402 7f7b06058e00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:09.173485 7f7b06058e00 1 journal close /dev/sdb22017-09-16 19:12:09.173511 7f7b06058e00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:10.197944 7f7561d50e00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5828 2017-09-16 19:12:10.222504 7f7561d50e00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:10.222723 7f7561d50e00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:10.222753 7f7561d50e00 1 journal close /dev/sdb22017-09-16 19:12:10.222785 7f7561d50e00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:14.370797 7f9fecb7fe00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 5964 2017-09-16 19:12:14.371221 7f9fecb7fe00 -1 bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset 102: buffer::malformed_input: voidbluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode pastend of struct encoding 2017-09-16 19:12:14.371350 7f9fecb7fe00 1 journal _open /dev/sdb2 fd 4: 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0 2017-09-16 19:12:14.371616 7f9fecb7fe00 1 journal close /dev/sdb22017-09-16 19:12:14.371745 7f9fecb7fe00 0 probe_block_device_fsid /dev/sdb2is filestore, 00000000-0000-0000-0000-000000000000 2017-09-16 19:12:21.171036 7f5d7579be00 0 ceph version 12.2.0(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),pid 61302017-09-16 19:12:21.209441 7f5d7579be00 0 probe_block_device_fsid /dev/sdb2is bluestore, 92a4a9eb-0a6a-405d-be83-11e4af42fa30 Any hint ? Thanks by advance Best regards-- Cheers, Alwin _______________________________________________ pve-user mailing list [email protected] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user_______________________________________________ pve-user mailing list [email protected] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________ pve-user mailing list [email protected] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
