RE: handling fs errors

2013-01-22 Thread Chen, Xiaoxi
@vger.kernel.org Subject: handling fs errors We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s), ceph-osd gave up waiting and committed suicide. XFS seemed to unwedge itself

Re: handling fs errors

2013-01-22 Thread Wido den Hollander
On 01/22/2013 07:12 AM, Yehuda Sadeh wrote: On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil s...@inktank.com wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s),

Re: handling fs errors

2013-01-22 Thread Gregory Farnum
On Tuesday, January 22, 2013 at 5:12 AM, Wido den Hollander wrote: On 01/22/2013 07:12 AM, Yehuda Sadeh wrote: On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil s...@inktank.com (mailto:s...@inktank.com) wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd

Re: handling fs errors

2013-01-22 Thread Dimitri Maziuk
On 01/22/2013 12:05 AM, Sage Weil wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. ... FWIW I see this often enough on cheap sata drives: they've a failure mode that makes sata driver

Re: handling fs errors

2013-01-22 Thread Sage Weil
On Wed, 23 Jan 2013, Andrey Korolyov wrote: On Tue, Jan 22, 2013 at 10:05 AM, Sage Weil s...@inktank.com wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s),

Re: handling fs errors

2013-01-22 Thread Sage Weil
On Tue, 22 Jan 2013, Dimitri Maziuk wrote: On 01/22/2013 12:05 AM, Sage Weil wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. ... FWIW I see this often enough on cheap sata

handling fs errors

2013-01-21 Thread Sage Weil
We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s), ceph-osd gave up waiting and committed suicide. XFS seemed to unwedge itself a bit after that, as the daemon was able

Re: handling fs errors

2013-01-21 Thread Yehuda Sadeh
On Mon, Jan 21, 2013 at 10:05 PM, Sage Weil s...@inktank.com wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s), ceph-osd gave up waiting and committed suicide. XFS

Re: handling fs errors

2013-01-21 Thread Andrey Korolyov
On Tue, Jan 22, 2013 at 10:05 AM, Sage Weil s...@inktank.com wrote: We observed an interesting situation over the weekend. The XFS volume ceph-osd locked up (hung in xfs_ilock) for somewhere between 2 and 4 minutes. After 3 minutes (180s), ceph-osd gave up waiting and committed suicide. XFS