Re: [ceph-users] fixable inconsistencies but more appears
Well. it seems memory I have 3 ODS per host with 8G RAM and block.db in SSD Setting bluestore_cache_size_ssd=1G seems to have fixed the problem. No new inconsistencies. On 21/08/18 16:09, Paul Emmerich wrote: Are you running tight on memory? Paul 2018-08-21 20:37 GMT+02:00 Alfredo Daniel Rezinovsky : My cluster suddenly shows many inconsistent PGs. with this kind of log 2018-08-21 15:29:39.065613 osd.2 osd.2 10.64.1.1:6801/1310438 146 : cluster [ERR] 2.61 shard 5: soid 2:864a5b37:::170510e.0004:head candidate had a read error 2018-08-21 15:31:38.542447 osd.2 osd.2 10.64.1.1:6801/1310438 147 : cluster [ERR] 2.61 shard 5: soid 2:86783f28:::1241f7f.:head candidate had a read error Al error fixes with "ceph pg repair" eventually but new inconsistencies appears. smart and kernel logs shows no hdd problems. I have bluestore OSDs in HDD with journal in an SDD partition. -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] fixable inconsistencies but more appears
I'm running kernel 4.15.0-32-generic. from ubuntu. Some kernel upgrade might have triggered the errors... On 21/08/18 18:24, Paul Emmerich wrote: You might be hitting http://tracker.ceph.com/issues/22464 in this configuration (it doesn't always show up as the CRC error described there) Which kernel are you running? Paul 2018-08-21 21:41 GMT+02:00 Alfredo Daniel Rezinovsky : Nope. I have plenty of RAM. 8Gb for 3 OSDs per node. Most of it used for buffering. On 21/08/18 16:09, Paul Emmerich wrote: Are you running tight on memory? Paul 2018-08-21 20:37 GMT+02:00 Alfredo Daniel Rezinovsky : My cluster suddenly shows many inconsistent PGs. with this kind of log 2018-08-21 15:29:39.065613 osd.2 osd.2 10.64.1.1:6801/1310438 146 : cluster [ERR] 2.61 shard 5: soid 2:864a5b37:::170510e.0004:head candidate had a read error 2018-08-21 15:31:38.542447 osd.2 osd.2 10.64.1.1:6801/1310438 147 : cluster [ERR] 2.61 shard 5: soid 2:86783f28:::1241f7f.:head candidate had a read error Al error fixes with "ceph pg repair" eventually but new inconsistencies appears. smart and kernel logs shows no hdd problems. I have bluestore OSDs in HDD with journal in an SDD partition. -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] fixable inconsistencies but more appears
Are you running tight on memory? Paul 2018-08-21 20:37 GMT+02:00 Alfredo Daniel Rezinovsky : > My cluster suddenly shows many inconsistent PGs. > > with this kind of log > > 2018-08-21 15:29:39.065613 osd.2 osd.2 10.64.1.1:6801/1310438 146 : cluster > [ERR] 2.61 shard 5: soid 2:864a5b37:::170510e.0004:head candidate > had a read error > 2018-08-21 15:31:38.542447 osd.2 osd.2 10.64.1.1:6801/1310438 147 : cluster > [ERR] 2.61 shard 5: soid 2:86783f28:::1241f7f.:head candidate > had a read error > > Al error fixes with "ceph pg repair" eventually but new inconsistencies > appears. > > smart and kernel logs shows no hdd problems. > > I have bluestore OSDs in HDD with journal in an SDD partition. > > -- > Alfredo Daniel Rezinovsky > Director de Tecnologías de Información y Comunicaciones > Facultad de Ingeniería - Universidad Nacional de Cuyo > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] fixable inconsistencies but more appears
My cluster suddenly shows many inconsistent PGs. with this kind of log 2018-08-21 15:29:39.065613 osd.2 osd.2 10.64.1.1:6801/1310438 146 : cluster [ERR] 2.61 shard 5: soid 2:864a5b37:::170510e.0004:head candidate had a read error 2018-08-21 15:31:38.542447 osd.2 osd.2 10.64.1.1:6801/1310438 147 : cluster [ERR] 2.61 shard 5: soid 2:86783f28:::1241f7f.:head candidate had a read error Al error fixes with "ceph pg repair" eventually but new inconsistencies appears. smart and kernel logs shows no hdd problems. I have bluestore OSDs in HDD with journal in an SDD partition. -- Alfredo Daniel Rezinovsky Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería - Universidad Nacional de Cuyo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
