You probably want to look at the central log (on your monitors) and see exactly what scrub errors it's reporting. There might also be useful info if you dump the pg info on the inconsistent PGs. But if you're getting this frequently, you're either hitting some unknown issues with the OSDs around some peering issue (unlikely, but possible), or there's an issue in ZFS or the way the OSD is using ZFS (much more possible). -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, Jun 24, 2014 at 9:38 AM, Peter Howell <[email protected]> wrote: > We are using two instances of version 8.0 of ceph both on ZFS. We are > frequently getting placement group inconsistent on both Ceph clusters > > We are suspecting that there is a problem with the network that is randomly > corrupting the update of placement groups. Does anyone have any suggestions > as to where and how to look for the problem. The network does not seem to > have any problems. ZFS is not reporting any problems with the disks and the > OSD's are fine. > > Thanks > > Peter. > > Log as follows > > health HEALTH_ERR 50 pgs inconsistent; 121 scrub errors > monmap e8: 6 mons at > {broll=10.5.8.9:6789/0,gelbin=10.5.8.10:6789/0,magni=10.5.8.12:6789/0,sicco=10.5.8.11:6789/0,tyrande=10.5.8.8:6789/0,varian=10.5.8.14:6789/0}, > election epoch 272, quorum 0,1,2,3,4,5 > tyrande,broll,gelbin,sicco,magni,varian > mdsmap e430: 1/1/1 up {0=broll=up:active}, 5 up:standby > osdmap e18928: 7 osds: 7 up, 7 in > pgmap v4910054: 512 pgs, 4 pools, 13043 MB data, 3681 objects > 40800 MB used, 856 GB / 895 GB avail > 462 active+clean > 50 active+clean+inconsistent > client io 12769 B/s rd, 5 op/s > > > --- > > Follow us on: > www.twitter.com/teamenergyeaa > www.youtube.com/user/teamenergyeaa > > Date for your diary > TEAM User Group Conference > 5th November 2014 > > Subscribe to our newsletter > http://www.teamenergy.com/newsletter-subscription/ > www.teamenergy.com +44 (0)1908 690018 [email protected] > Registered Office: TEAM (Energy Auditing Agency Ltd), 34 The Forum, > Rockingham Drive, Linford Wood, Milton Keynes, MK14 6LY > Registered in England No. 1916768 > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
