I must also add that I just found in the log the following. I don't know if this has something to do with the problem.
=> ceph-osd.admin.log <== 2016-05-10 18:21:46.060278 7fa8f30cc8c0 0 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 14135 2016-05-10 18:21:46.060460 7fa8f30cc8c0 -1 bluestore(/dev/sdc2) _read_bdev_label unable to decode label at offset 66: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding 2016-05-10 18:21:46.062949 7fa8f30cc8c0 1 journal _open /dev/sdc2 fd 4: 5367660544 bytes, block size 4096 bytes, directio = 0, aio = 0 2016-05-10 18:21:46.062991 7fa8f30cc8c0 1 journal close /dev/sdc2 2016-05-10 18:21:46.063026 7fa8f30cc8c0 0 probe_block_device_fsid /dev/sdc2 is filestore, 119a9f4e-73d8-4a1f-877c-d60b01840c96 2016-05-10 18:21:47.072082 7eff735598c0 0 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 14177 2016-05-10 18:21:47.072285 7eff735598c0 -1 bluestore(/dev/sdf2) _read_bdev_label unable to decode label at offset 66: buffer::malformed_input: void bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding 2016-05-10 18:21:47.074799 7eff735598c0 1 journal _open /dev/sdf2 fd 4: 5367660544 bytes, block size 4096 bytes, directio = 0, aio = 0 2016-05-10 18:21:47.074844 7eff735598c0 1 journal close /dev/sdf2 2016-05-10 18:21:47.074881 7eff735598c0 0 probe_block_device_fsid /dev/sdf2 is filestore, fd069e6a-9a62-4286-99cb-d8a523bd946a r On Tue, May 10, 2016 at 6:07 PM, Gonzalo Aguilar Delgado < [email protected]> wrote: > Hello, > > I just upgraded my cluster to the version 10.1.2 and it worked well for a > while until I saw that systemctl [email protected] was failed > and I reruned it. > > From there the OSD stopped working. > > This is ubuntu 16.04. > > I connected to the IRC looking for help where people pointed me to one or > another place but none of the investigations helped to resolve. > > My configuration is rather simple: > > oot@red-compute:~# ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -1 1.00000 root default > -4 1.00000 rack rack-1 > -2 1.00000 host blue-compute > 0 1.00000 osd.0 down 0 1.00000 > 2 1.00000 osd.2 down 0 1.00000 > -3 1.00000 host red-compute > 1 1.00000 osd.1 down 0 1.00000 > 3 0.50000 osd.3 up 1.00000 1.00000 > 4 1.00000 osd.4 down 0 1.00000 > > > > This is what I got sofar: > > > 1. Once upgraded I discovered that daemon runs under ceph. I just ran > chown on ceph directories. and it worked. > 2. Firewall is fully disabled. Checked connectivity with nc and nmap. > 3. Configuration seems to be right. I can post if you want. > 4. Enabling logging on OSD shows that for example osd.1 is > reconnecting all the time. > 1. 2016-05-10 14:35:48.199573 7f53e8f1a700 1 -- 0.0.0.0:6806/13962 > >> :/0 pipe(0x556f99413400 sd=84 :6806 s=0 pgs=0 cs=0 l=0 > c=0x556f993b3a80).accept sd=84 172.16.0.119:35388/0 > 2016-05-10 14:35:48.199966 7f53e8f1a700 2 -- 0.0.0.0:6806/13962 > >> :/0 pipe(0x556f99413400 sd=84 :6806 s=4 pgs=0 cs=0 l=0 > c=0x556f993b3a80).fault (0) Success > 2016-05-10 14:35:48.200018 7f53fb941700 1 osd.1 2468 > ms_handle_reset con 0x556f993b3a80 session 0 > 5. OSD.3 goes ok because never left out because ceph restriction. > 6. I rebooted all services at once for it to have available all OSD at > the same time and don't mark it down. Don't work. > 7. I forced up from commandline. ceph osd in 1-5. They appear as in > for a while then out. > 8. We tried ceph-disk activate-all to boot everything. Don't work. > > > The strange thing is that culster started worked just right after upgrade. > But the systemctrl command broke both servers. > > root@blue-compute:~# ceph -w > cluster 9028f4da-0d77-462b-be9b-dbdf7fa57771 > health HEALTH_ERR > 694 pgs are stuck inactive for more than 300 seconds > 694 pgs stale > 694 pgs stuck stale > too many PGs per OSD (1528 > max 300) > mds cluster is degraded > crush map has straw_calc_version=0 > monmap e10: 2 mons at {blue-compute= > 172.16.0.119:6789/0,red-compute=172.16.0.100:6789/0} > election epoch 3600, quorum 0,1 red-compute,blue-compute > fsmap e673: 1/1/1 up {0:0=blue-compute=up:replay} > osdmap e2495: 5 osds: 1 up, 1 in; 5 remapped pgs > pgmap v40765481: 764 pgs, 6 pools, 410 GB data, 103 kobjects > 87641 MB used, 212 GB / 297 GB avail > 694 stale+active+clean > 70 active+clean > > 2016-05-10 17:03:55.822440 mon.0 [INF] HEALTH_ERR; 694 pgs are stuck > inactive for more than 300 seconds; 694 pgs stale; 694 pgs stuck stale; too > many PGs per OSD (1528 > max 300); mds cluster is degraded; crush map has > straw_calc_version= > > cat /etc/ceph/ceph.conf > [global] > > fsid = 9028f4da-0d77-462b-be9b-dbdf7fa57771 > mon_initial_members = blue-compute, red-compute > mon_host = 172.16.0.119, 172.16.0.100 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > filestore_xattr_use_omap = true > public_network = 172.16.0.0/24 > osd_pool_default_pg_num = 100 > osd_pool_default_pgp_num = 100 > osd_pool_default_size = 2 # Write an object 3 times. > osd_pool_default_min_size = 1 # Allow writing one copy in a degraded state. > > ## Required upgrade > osd max object name len = 256 > osd max object namespace len = 64 > > [mon.] > > debug mon = 9 > caps mon = "allow *" > > > Any help on this? Any clue of what's going wrong? > > > Best regards, > > > -- No subestimes el poder de la gente estúpida en grupos grandes...
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
