On 02/22, Christoph Hellwig wrote: > It looks like this patch landed in linux-next despite all the perfectly > reasonable objections. Jaegeuk, please drop it again.
I've been waiting for a generic solution as suggested here. Until then, I'd like to keep this in f2fs *only* in order to ship the fix in products. Once there's a right fix, let me drop or revise this patch again. > > On Wed, Jan 12, 2022 at 03:06:12PM +0100, Peter Zijlstra wrote: > > On Mon, Jan 10, 2022 at 11:41:23AM -0800, Tim Murray wrote: > > > > > 1. f2fs-ckpt thread is running f2fs_write_checkpoint(), holding the > > > cp_rwsem write lock while doing so via f2fs_lock_all() in > > > block_operations(). > > > 2. Random very-low-priority thread A makes some other f2fs call that > > > tries to get the cp_rwsem read lock by atomically adding on the rwsem, > > > fails and deschedules in uninterruptible sleep. cp_rwsem now has a > > > non-zero reader count but is write-locked. > > > 3. f2fs-ckpt thread releases the cp_rwsem write lock. cp_rwsem now has > > > a non-zero reader count and is not write-locked, so is reader-locked. > > > 4. Other threads call fsync(), which requests checkpoints from > > > f2fs-ckpt, and block on a completion event that f2fs-ckpt dispatches. > > > cp_rwsem still has a non-zero reader count because the low-prio thread > > > A from (2) has not been scheduled again yet. > > > 5. f2fs-ckpt wakes up to perform checkpoints, but it stalls on the > > > write lock via cmpxchg in block_operations() until the low-prio thread > > > A has run and released the cp_rwsem read lock. Because f2fs-ckpt can't > > > run, all fsync() callers are also effectively blocked by the > > > low-priority thread holding the read lock. > > > > > > I think this is the rough shape of the problem (vs readers holding the > > > lock for too long or something like that) because the low-priority > > > thread is never run between when it is initially made runnable by > > > f2fs-ckpt and when it runs tens/hundreds of milliseconds later then > > > immediately unblocks f2fs-ckpt. > > > > *urgh*... so you're making the worst case less likely but fundamentally > > you don't change anything. > > > > If one of those low prio threads manages to block while holding > > cp_rwsem your checkpoint thread will still block for a very long time. > > > > So while you improve the average case, the worst case doesn't improve > > much I think. > > > > Also, given that this is a system wide rwsem, would percpu-rwsem not be > > 'better' ? Arguably with the same hack cgroups uses for it (see > > cgroup_init()) to lower the cost of percpu_down_write(). > > > > Now, I'm not a filesystem developer and I'm not much familiar with the > > problem space, but this locking reads like a fairly big problem. I'm not > > sure optimizing the lock is the answer. > > > > > ---end quoted text--- _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
