Thanks for your help,
Le 18/09/2017 à 12:37, Alwin Antreich a écrit :
On Sun, Sep 17, 2017 at 11:18:51AM +0200, Phil Schwarz wrote:
Hi,
going on on the same problem (links [1] & [2] )
[1] : https://pve.proxmox.com/pipermail/pve-user/2017-July/168578.html
[2] : https://pve.proxmox.com/pipermail/pve-user/2017-September/168775.html
-Added a brand new node, updated to last ceph version (the proxmox team
recompiled one)
Can you please post a 'ceph versions' and a 'ceph osd tree' to get some
overview on your setup?
root@arya:~# ceph versionceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)
root@arya:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 10.06328 root default
-3 0 host daenerys
-5 1.81360 host jaime
5 hdd 1.81360 osd.5 up 1.00000 1.00000
-2 6.59999 host jon
1 hdd 4.20000 osd.1 up 1.00000 1.00000
3 hdd 2.39999 osd.3 up 1.00000 1.00000
-4 1.64969 host tyrion
2 hdd 0.44969 osd.2 up 1.00000 1.00000
4 hdd 1.20000 osd.4 up 1.00000 1.00000
0 0 osd.0 down 0 1.00000
-plugged a new disk
-used GUI (same result with pveceph createosd from cli) to create a new osd
(with bluestore feature).
PVE GUI and CLI use the same API for managing ceph
1. The OSD doesn't appear on gui
2. The OSD is seen as down and out of any node
3. the /var/log/ceph/ceph-osd.admin.log logfile seems to figure a
mismatch between filestore and bluestore:
Do you see any errors in the mon logs or ceph.log itself?
(Jaime is a mon& mgr)
root@jaime:~# tail -f /var/log/ceph/ceph-mon.1.log
2017-09-18 21:05:00.084847 7f8a1b4a8700 1 mon.1@0(leader).log v2152264
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152264)
2017-09-18 21:05:09.963784 7f8a1868c700 0
mon.1@0(leader).data_health(2028) update_stats avail 90% total 58203 MB,
used 2743 MB, avail 52474 MB
2017-09-18 21:05:29.878648 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd new", "uuid":
"89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"} v 0) v1
2017-09-18 21:05:29.878705 7f8a15e87700 0 log_channel(audit) log [INF]
: from='client.6392525 -' entity='client.bootstrap-osd' cmd=[{"prefix":
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]: dispatch
2017-09-18 21:05:29.927377 7f8a1b4a8700 1 mon.1@0(leader).osd e1141
e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932253 7f8a1b4a8700 0 log_channel(audit) log [INF]
: from='client.6392525 -' entity='client.bootstrap-osd' cmd='[{"prefix":
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]': finished
2017-09-18 21:05:29.932388 7f8a1b4a8700 0 log_channel(cluster) log
[DBG] : osdmap e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932983 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1
2017-09-18 21:05:29.933040 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch
2017-09-18 21:05:29.933337 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
2017-09-18 21:05:29.933383 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
2017-09-18 21:05:29.933674 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
2017-09-18 21:05:29.933692 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
2017-09-18 21:05:29.933880 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 4} v 0) v1
2017-09-18 21:05:29.933897 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch
2017-09-18 21:05:29.934062 7f8a15e87700 0 mon.1@0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 5} v 0) v1
2017-09-18 21:05:29.934089 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch
2017-09-18 21:05:30.113007 7f8a1b4a8700 1 mon.1@0(leader).log v2152265
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 3 entries (version 2152265)
2017-09-18 21:05:31.154227 7f8a1b4a8700 1 mon.1@0(leader).log v2152266
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152266)
2017-09-18 21:05:32.289428 7f8a1b4a8700 1 mon.1@0(leader).log v2152267
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152267)
2017-09-18 21:05:36.782573 7f8a1b4a8700 1 mon.1@0(leader).log v2152268
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152268)
2017-09-18 21:06:09.964314 7f8a1868c700 0
mon.1@0(leader).data_health(2028) update_stats avail 90% total 58203 MB,
used 2744 MB, avail 52473 MB
2017-09-18 21:06:20.040930 7f8a1b4a8700 1 mon.1@0(leader).log v2152269
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152269)
And ceph.log
root@jaime:~# tail -f /var/log/ceph/ceph.log
2017-09-18 12:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38100 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 13:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38101 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 14:00:00.000133 mon.1 mon.0 10.250.0.21:6789/0 38102 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
201
Yes, the cluster is not really healthy, indeed ....
Thanks
2017-09-16 19:12:00.468481 7f6469cdde00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5624
2017-09-16 19:12:00.470154 7f6469cdde00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:00.471408 7f6469cdde00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:00.471727 7f6469cdde00 1 journal close /dev/sdb2
2017-09-16 19:12:00.471994 7f6469cdde00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:05.042622 7f000b944e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5702
2017-09-16 19:12:05.066343 7f000b944e00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:05.066549 7f000b944e00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:05.066717 7f000b944e00 1 journal close /dev/sdb2
2017-09-16 19:12:05.066843 7f000b944e00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:08.198548 7f5740748e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5767
2017-09-16 19:12:08.223674 7f5740748e00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:08.223831 7f5740748e00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:08.224213 7f5740748e00 1 journal close /dev/sdb2
2017-09-16 19:12:08.224342 7f5740748e00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:09.149622 7f7b06058e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5800
2017-09-16 19:12:09.173319 7f7b06058e00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:09.173402 7f7b06058e00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:09.173485 7f7b06058e00 1 journal close /dev/sdb2
2017-09-16 19:12:09.173511 7f7b06058e00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:10.197944 7f7561d50e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5828
2017-09-16 19:12:10.222504 7f7561d50e00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:10.222723 7f7561d50e00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:10.222753 7f7561d50e00 1 journal close /dev/sdb2
2017-09-16 19:12:10.222785 7f7561d50e00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:14.370797 7f9fecb7fe00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 5964
2017-09-16 19:12:14.371221 7f9fecb7fe00 -1 bluestore(/dev/sdb2)
_read_bdev_label unable to decode label at offset 102:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding
2017-09-16 19:12:14.371350 7f9fecb7fe00 1 journal _open /dev/sdb2 fd 4:
750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-16 19:12:14.371616 7f9fecb7fe00 1 journal close /dev/sdb2
2017-09-16 19:12:14.371745 7f9fecb7fe00 0 probe_block_device_fsid /dev/sdb2
is filestore, 00000000-0000-0000-0000-000000000000
2017-09-16 19:12:21.171036 7f5d7579be00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
pid 6130
2017-09-16 19:12:21.209441 7f5d7579be00 0 probe_block_device_fsid /dev/sdb2
is bluestore, 92a4a9eb-0a6a-405d-be83-11e4af42fa30
Any hint ?
Thanks by advance
Best regards
--
Cheers,
Alwin
_______________________________________________
pve-user mailing list
[email protected]
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
[email protected]
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
