Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-04-09 Thread Andreas Gruenbacher
On Tue, 9 Apr 2019 at 17:37, Ross Lagerwall wrote: > On 4/5/19 6:50 PM, Andreas Gruenbacher wrote: > > Hi Ross, > > > > On Tue, 2 Apr 2019 at 00:59, Andreas Gruenbacher > > wrote: > >> thanks for the great analysis. > >> > >> On Tue, 26 Mar 2019 at 20:14, Bob Peterson wrote: > >>> -

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-04-09 Thread Ross Lagerwall
On 4/5/19 6:50 PM, Andreas Gruenbacher wrote: Hi Ross, On Tue, 2 Apr 2019 at 00:59, Andreas Gruenbacher wrote: thanks for the great analysis. On Tue, 26 Mar 2019 at 20:14, Bob Peterson wrote: - Original Message - 6. gfs2_log_flush() continues and calls revoke_lo_after_commit() and

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-04-05 Thread Andreas Gruenbacher
Hi Ross, On Tue, 2 Apr 2019 at 00:59, Andreas Gruenbacher wrote: > thanks for the great analysis. > > On Tue, 26 Mar 2019 at 20:14, Bob Peterson wrote: > > - Original Message - > > > 6. gfs2_log_flush() continues and calls revoke_lo_after_commit() and > > > uses the freed glock (stack

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-04-01 Thread Andreas Gruenbacher
Hi Ross, thanks for the great analysis. On Tue, 26 Mar 2019 at 20:14, Bob Peterson wrote: > - Original Message - > > 6. gfs2_log_flush() continues and calls revoke_lo_after_commit() and > > uses the freed glock (stack trace above). > > > > Should evict_inode call gfs2_ail_flush() or

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-03-26 Thread Bob Peterson
Hi Ross, - Original Message - > 6. gfs2_log_flush() continues and calls revoke_lo_after_commit() and > uses the freed glock (stack trace above). > > Should evict_inode call gfs2_ail_flush() or something so that the revoke > is written before it drops its reference to the glock? > > Or

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-03-26 Thread Ross Lagerwall
On 1/31/19 5:18 PM, Andreas Gruenbacher wrote: Hi Ross, On Thu, 31 Jan 2019 at 11:56, Ross Lagerwall wrote: Each gfs2_bufdata stores a reference to a glock but the reference count isn't incremented. This causes an occasional use-after-free of the glock. Fix by taking a reference on the glock

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free (Another debug patch)

2019-02-01 Thread Bob Peterson
Hi Ross, - Original Message - > Do you have any suggestions for tracking down the root cause? Attached is the starting point for a generic "debug" kernel trace point, complete with examples of how it's used. It's always associated with a particular glock. You might find it helpful, but

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-02-01 Thread Bob Peterson
Hi Ross, - Original Message - > Do you have any suggestions for tracking down the root cause? One time, when I had a similar problem in rhel7, and couldn't use kernel tracing because there were millions of glocks involved. The trace was too huge and quickly swamped the biggest possible

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-02-01 Thread Bob Peterson
Hi Ross, - Original Message - (snip) > We haven't observed any problems that can be directly attributed to this > without KASAN, although it is hard to tell what a stray write may do. We > have hit sporadic asserts and filesystem corruption during testing. > > When I added tracing, the

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-02-01 Thread Ross Lagerwall
On 1/31/19 5:18 PM, Andreas Gruenbacher wrote: Hi Ross, On Thu, 31 Jan 2019 at 11:56, Ross Lagerwall wrote: Each gfs2_bufdata stores a reference to a glock but the reference count isn't incremented. This causes an occasional use-after-free of the glock. Fix by taking a reference on the glock

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-01-31 Thread Andreas Gruenbacher
Hi Ross, On Thu, 31 Jan 2019 at 11:56, Ross Lagerwall wrote: > Each gfs2_bufdata stores a reference to a glock but the reference count > isn't incremented. This causes an occasional use-after-free of the > glock. Fix by taking a reference on the glock during allocation and > dropping it when

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-01-31 Thread Bob Peterson
- Original Message - > Each gfs2_bufdata stores a reference to a glock but the reference count > isn't incremented. This causes an occasional use-after-free of the > glock. Fix by taking a reference on the glock during allocation and > dropping it when freeing. > > Found by KASAN: > >

Re: [Cluster-devel] [PATCH 1/2] gfs2: Fix occasional glock use-after-free

2019-01-31 Thread Steven Whitehouse
Hi, On 31/01/2019 10:55, Ross Lagerwall wrote: Each gfs2_bufdata stores a reference to a glock but the reference count isn't incremented. This causes an occasional use-after-free of the glock. Fix by taking a reference on the glock during allocation and dropping it when freeing. Another good