Hi Wido, On Sun, 16 Feb 2014, Wido den Hollander wrote: > On 02/16/2014 06:49 PM, Gregory Farnum wrote: > > Did you maybe upgrade that box to v0.67.6? This sounds like one of the > > bugs Sage mentioned in it. > > No, I checked it again. Version is: ceph version 0.67.5 > (a60ac9194718083a4b6a225fc17cad6096c69bd1) > > All machines in the cluster are on that version.
Are you sure none of the running ceph-osd processes aren't 0.67.6? Maybe check 'ceph daemon osd.NNN version'... sage > > Wido > > > -Greg > > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > > > > On Sun, Feb 16, 2014 at 4:23 AM, Wido den Hollander <[email protected]> wrote: > > > Hi, > > > > > > Yesterday I got a notification that a RGW setup was having issues with > > > objects suddenly giving errors (403 and 404) when trying to access them. > > > > > > I started digging and after cranking up the logs with 'debug rados' and > > > 'debug rgw' set to 20 I found what caused RGW to throw a error: > > > > > > librados: Objecter returned from getxattrs r=-2 > > > > > > Using "ceph osd map .rgw.buckets <object>" I found which OSDs were primary > > > for that object's PG and I saw that they all came from one machine which > > > got > > > a clean shutdown and start just 24 hours before that. > > > > > > After taking that machine out of production the other OSDs took over and > > > RGW > > > started serving the objects again, but I'm confused. > > > > > > The underlying filesystem is XFS and all 6 filesystems were clean and > > > healthy. Like I said, the machine only got a clean shutdown 24 hours > > > before > > > that due to a physical migration, but that's all. > > > > > > Did anybody see this before? Suddenly the xattrs for those objects were > > > gone. > > > > > > This was with Ceph 0.67.5 > > > > > > -- > > > Wido den Hollander > > > 42on B.V. > > > > > > Phone: +31 (0)20 700 9902 > > > Skype: contact42on > > > _______________________________________________ > > > ceph-users mailing list > > > [email protected] > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > Wido den Hollander > 42on B.V. > > Phone: +31 (0)20 700 9902 > Skype: contact42on > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
