Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Wed, Dec 13, 2017 at 02:09:49AM +0100, Rafael J. Wysocki wrote: > I'm assuming an update of this to be posted due to the comments from Jan > on patch [3/11] and patch [7/11] probably. > > Is there anything else that needs to be addressed? I was waiting on Jan Kara's feedback on how he'd like to proceed with the unthawing on error given his point on that the device mapper API seems to match the in kernel automatic freezing just that I didn't use that same interface. 0-day did come back with one RCU issue which I also have to address: [ 422.919958] kernel BUG at kernel/rcu/sync.c:228! [ 422.920115] invalid opcode: [#1] SMP [ 422.920212] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole sr_mod cdrom sd_mod sg snd_hda_codec_idt snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal +intel_powerclamp coretemp kvm irqbypass crct10dif_pclmul snd_hda_intel crc32_pclmul crc32c_intel snd_hda_codec snd_hda_core snd_hwdep i915 ghash_clmulni_intel cryptd snd_pcm pcspkr +drm_kms_helper snd_timer ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops snd libata shpchp soundcore drm video ip_tables [ 422.921168] CPU: 2 PID: 237 Comm: kworker/2:3 Not tainted 4.15.0-rc1-00030-gf95c16a #1 [ 422.921347] Hardware name: Hewlett-Packard p6-1451cx/2ADA, BIOS 8.15 02/05/2013 [ 422.921515] Workqueue: events destroy_super_work [ 422.921628] task: 8801bfcd task.stack: c90001128000 [ 422.921768] RIP: 0010:rcu_sync_dtor+0x65/0x70 [ 422.921874] RSP: :c9000112be60 EFLAGS: 00010286 [ 422.921985] RAX: 0008 RBX: 8801c00793d8 RCX: 0001fece [ 422.922133] RDX: fff6 RSI: 0282 RDI: 8801c00793d8 [ 422.922283] RBP: 880212f1b6c0 R08: R09: 009c [ 422.922432] R10: R11: R12: 880212f1f800 [ 422.922579] R13: R14: 8801c036db40 R15: 8801c00795b0 [ 422.922728] FS: () GS:880212f0() knlGS: [ 422.922931] CS: 0010 DS: ES: CR0: 80050033 [ 422.923054] CR2: 7ffe2f820ff8 CR3: 01e09002 CR4: 001606e0 [ 422.923203] Call Trace: [ 422.923266] percpu_free_rwsem+0x15/0x30 [ 422.923357] destroy_super_work+0x3d/0x50 [ 422.923449] process_one_work+0x18f/0x3e0 [ 422.923540]
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thursday, November 30, 2017 12:23:45 AM CET Luis R. Rodriguez wrote: > This is a followup from the original RFC which proposed to start > to kill kthread freezing all together [0]. Instead of going straight > out to the jugular for kthread freezing this series only addresses > killing freezer calls on filesystems which implement freeze_fs, after > we let the kernel freeze these filesystems for us on suspend. > > This approach puts on a slow but steady path towards the original goal > though. Each subsystem could look for similar solutions. Even with > filesystems we're not all done yet, after this we'll still have to > decide what to do about filesystems which do not implement freeze_fs(). > > Motivation and problem: > > kthreads have some semantics for freezing, which helps the kernel > freeze them when a system is going to suspend or hibernation. These > semantics are not well defined though, and it actually turns out > pretty hard to get it right. > > Without a proper solution suspend and hibernation are fragile on filesystems, > it can easily break suspend and fixing such issues are in no way trivial [1] > [2]. > > Proposed solution: > > Instead of fixing such semantics and trying to get all filesystems to do it > right, we can easily do away with all freezing calls if the filesystem > implements a proper freeze_fs() callback. The following 9 filesystems have > freeze_fs() implemented as such we can let the kernel issue the callback upon > suspend and thaw on resume automatically on our behalf. > > o xfs > o reiserfs > o nilfs2 > o jfs > o f2fs > o ext4 > o ext2 > o btrfs > > Of these, the following have freezer helpers, which can then be removed > after the kernel automaticaly calls freeze_fs for us on suspend: > > o xfs > o nilfs2 > o jfs > o f2fs > o ext4 > > I've tested this on a system with ext4 and XFS, and have let 0-day go at > without issues. I have branches availabe for linux-next [3] and Linus' > latest tree [4]. > > Further testing, thoughts, reviews, flames are all equally appreciated. > > [0] https://lkml.kernel.org/r/20171003185313.1017-1-mcg...@kernel.org > [1] https://bugzilla.suse.com/show_bug.cgi?id=1043449 > [2] https://lkml.kernel.org/r/20171113103139.ga18...@yu-chen.sh.intel.com > [3] > https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20171129-fs-freeze-cleanup > [4] > https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20171129-fs-freeze-cleanup > > Luis R. Rodriguez (11): > fs: provide unlocked helper for freeze_super() > fs: provide unlocked helper thaw_super() > fs: add frozen sb state helpers > fs: distinguish between user initiated freeze and kernel initiated > freeze > fs: add iterate_supers_excl() and iterate_supers_reverse_excl() > fs: freeze on suspend and thaw on resume > xfs: remove not needed freezing calls > ext4: remove not needed freezing calls > f2fs: remove not needed freezing calls > nilfs2: remove not needed freezing calls > jfs: remove not needed freezing calls > > fs/ext4/ext4_jbd2.c| 2 +- > fs/ext4/super.c| 2 - > fs/f2fs/gc.c | 5 +- > fs/f2fs/segment.c | 6 +- > fs/jfs/jfs_logmgr.c| 11 +- > fs/jfs/jfs_txnmgr.c| 31 ++--- > fs/nilfs2/segment.c| 48 > fs/super.c | 320 > - > fs/xfs/xfs_trans.c | 2 +- > fs/xfs/xfs_trans_ail.c | 7 +- > include/linux/fs.h | 63 +- > kernel/power/process.c | 15 ++- > 12 files changed, 378 insertions(+), 134 deletions(-) > I'm assuming an update of this to be posted due to the comments from Jan on patch [3/11] and patch [7/11] probably. Is there anything else that needs to be addressed? Thanks, Rafael
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Fri, Dec 01, 2017 at 02:05:44PM -0500, Jeff Layton wrote: > On Thu, 2017-11-30 at 17:41 +0100, Jiri Kosina wrote: > > On Fri, 1 Dec 2017, Yu Chen wrote: > > > > > BTW, is nfs able to be included in this set? I also encountered a > > > freeze() failure due to nfs access during that stage recently. > > > > The freezer usage in NFS is magnitudes more complicated, so it makes sense > > to first go after the lower hanging fruit to figure out the viability of > > the whole aproach in practice. > > > > Agreed that we should do this in stages. It doesn't help that freezer > handling in the client is a bit of a mess at this point... > > At a high level for NFS, I think we need to have freeze_fs make the RPC > engine "park" newly issued RPCs for that fs' client onto a > rpc_wait_queue. Any RPC that has already been sent however, we need to > wait for a reply. > > Once everything is quiesced we can return and call it frozen. > unfreeze_fs can then just have the engine stop parking RPCs and wake up > the waitq. That seems pretty reasonable. freezing is expected to take a bit of time to run - local filesystems can do a fair bit of IO draining queues, inflight operations and bringing the journal into a consistent state on disk before declaring the filesystem is frozen. > That should be enough to make suspend and resume work more reliably. If, > however, you're interested in making the cgroup freezer also work, then > we may need to do a bit more work to ensure that we don't end up with > frozen tasks squatting on VFS locks. None of the existing freezing code gives those guarantees. In fact, freezing a filesystem pretty much guarantees the opposite - that tasks *will freeze when holding VFS locks* - and so the cgroup freezer is broken by design if it requires tasks to be frozen without holding any VFS/filesystem lock context. So I wouldn't really worry about the cgroup freezer Cheers, Dave. -- Dave Chinner da...@fromorbit.com
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, 2017-11-30 at 17:41 +0100, Jiri Kosina wrote: > On Fri, 1 Dec 2017, Yu Chen wrote: > > > BTW, is nfs able to be included in this set? I also encountered a > > freeze() failure due to nfs access during that stage recently. > > The freezer usage in NFS is magnitudes more complicated, so it makes sense > to first go after the lower hanging fruit to figure out the viability of > the whole aproach in practice. > Agreed that we should do this in stages. It doesn't help that freezer handling in the client is a bit of a mess at this point... At a high level for NFS, I think we need to have freeze_fs make the RPC engine "park" newly issued RPCs for that fs' client onto a rpc_wait_queue. Any RPC that has already been sent however, we need to wait for a reply. Once everything is quiesced we can return and call it frozen. unfreeze_fs can then just have the engine stop parking RPCs and wake up the waitq. That should be enough to make suspend and resume work more reliably. If, however, you're interested in making the cgroup freezer also work, then we may need to do a bit more work to ensure that we don't end up with frozen tasks squatting on VFS locks. -- Jeff Layton
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, Nov 30, 2017 at 10:51:57PM +0100, Pavel Machek wrote: > Hi! > > > Proposed solution: > > > > Instead of fixing such semantics and trying to get all filesystems to do it > > right, we can easily do away with all freezing calls if the filesystem > > implements a proper freeze_fs() callback. The following 9 filesystems have > > freeze_fs() implemented as such we can let the kernel issue the callback > > upon > > suspend and thaw on resume automatically on our behalf. > > > > o xfs > > o reiserfs > > o nilfs2 > > o jfs > > o f2fs > > o ext4 > > o ext2 > > o btrfs > > > > Of these, the following have freezer helpers, which can then be removed > > after the kernel automaticaly calls freeze_fs for us on suspend: > > > > o xfs > > o nilfs2 > > o jfs > > o f2fs > > o ext4 > > > > I've tested this on a system with ext4 and XFS, and have let 0-day go at > > without issues. I have branches availabe for linux-next [3] and Linus' > > latest tree [4]. > > Was hibernation tested? uswsusp? I had not done a test before but I just did and it worked, in fact I was able to keep my ssh connection to my qemu guest after resume from hibernation with this. root@piggy:~# echo shutdown > /sys/power/disk; echo disk > /sys/power/state [ 87.930446] PM: hibernation entry [ 87.936294] firmware_class: device_cache_fw_images [ 87.936363] PM: Syncing filesystems ... [ 87.979960] PM: done. [ 87.980594] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 87.983839] Freezing filesystems ... [ 87.983844] xfs (sdb1): freezing [ 88.013313] ext4 (sda1): freezing [ 88.057635] done. [ 88.058242] OOM killer disabled. ... [ 88.145364] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 88.221583] Disabling non-boot CPUs ... [ 88.241231] Unregister pv shared memory for cpu 1 [ 88.244139] smpboot: CPU 1 is now offline [ 88.273290] Unregister pv shared memory for cpu 2 [ 88.276170] smpboot: CPU 2 is now offline [ 88.297296] Unregister pv shared memory for cpu 3 [ 88.21] smpboot: CPU 3 is now offline [ 88.302665] PM: Creating hibernation image: [ 88.305633] PM: Need to copy 107866 pages [ 88.305633] PM: Normal pages needed: 107866 + 1024, available pages: 940488 ... At this point my qemu session ends. I start it up again. 88.305633] Enabling non-boot CPUs ... [ 88.305633] x86: Booting SMP configuration: [ 88.305633] smpboot: Booting Node 0 Processor 1 APIC 0x1 [ 88.244097] kvm-clock: cpu 1, msr 1:3ffef041, secondary cpu clock [ 88.334899] KVM setup async PF for cpu 1 [ 88.335174] kvm-stealtime: cpu 1, msr 13fc8d9c0 ... [ 88.424852] sd 0:0:0:0: [sda] Starting disk [ 88.424893] sd 0:0:1:0: [sdb] Starting disk [ 88.424928] sd 1:0:0:0: [sdc] Starting disk [ 88.473888] PM: Basic memory bitmaps freed [ 88.473890] OOM killer enabled. [ 88.474876] ext4 (sda1): thawing [ 88.585233] ata2.01: NODEV after polling detection [ 88.587805] ata2.00: configured for MWDMA2 [ 88.589436] ata1.00: configured for MWDMA2 [ 88.592421] ata1.01: configured for MWDMA2 [ 88.601141] xfs (sdb1): thawing [ 88.602430] Restarting tasks ... done. [ 88.611055] PM: hibernation exit [ 90.492926] e1000: ens3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 98.780159] firmware_class: device_uncache_fw_images And ssh is working after this, without requiring to initiate another connection. Let me know if you'd like me to test something else. Luis
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
Hi! > Proposed solution: > > Instead of fixing such semantics and trying to get all filesystems to do it > right, we can easily do away with all freezing calls if the filesystem > implements a proper freeze_fs() callback. The following 9 filesystems have > freeze_fs() implemented as such we can let the kernel issue the callback upon > suspend and thaw on resume automatically on our behalf. > > o xfs > o reiserfs > o nilfs2 > o jfs > o f2fs > o ext4 > o ext2 > o btrfs > > Of these, the following have freezer helpers, which can then be removed > after the kernel automaticaly calls freeze_fs for us on suspend: > > o xfs > o nilfs2 > o jfs > o f2fs > o ext4 > > I've tested this on a system with ext4 and XFS, and have let 0-day go at > without issues. I have branches availabe for linux-next [3] and Linus' > latest tree [4]. Was hibernation tested? uswsusp? Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html signature.asc Description: Digital signature
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, Nov 30, 2017 at 08:53:52PM +, Bart Van Assche wrote: > On Thu, 2017-11-30 at 20:42 +0100, Luis R. Rodriguez wrote: > > On Thu, Nov 30, 2017 at 05:01:13PM +, Bart Van Assche wrote: > > > The md resync > > > thread must be stopped before a system is frozen. Today the md driver uses > > > the kthread freezing mechanism for that purpose. Do you have a plan for > > > handling the more complicated scenarios, e.g. a filesystem that exists on > > > top > > > of an md device where the md device uses one or more files as backing > > > store > > > and with the loop driver between the md device and the files? > > > > Nope not yet. It seems you have given this some thought though so you're > > help here is greatly appreciated. In fact the way we should see the long > > term 'kill kthread freezing' effort should be a collaborative one. I've > > never touched md, so folks more familiar with md should give this some > > thought. > > > > Can for instance md register_pm_notifier() and register_syscore_ops() > > and do handy work to pause some work to replace kthread freezing? > > Note that I believe a pm notifier is needed in case syscore_suspend() > > is not called, say on a suspend fail. > > Sorry but I don't think that a solution can be based on a notifier mechanism. > Freezing has to happen in the order in which drivers and filesystems have > been stacked (filesystem > md device > filesystem for the above example). Yup, we need top down stack freezing. We get that at the filesystem layer by the reverse order superblock iteration - that freezes newer mounts before older mounts, so things like filesystems on loopback freeze before the lower filesystem that hosts the loopback image. I think what we need is the opposite of the "freeze_bdev()" interface. That allows a block device to freeze the superblock on the block device. i.e. when the block device needs to quiesce (e.g. for a snapshot to be taken) it locks the filesystem and waits for it to quiesce, then does what it needs to and unlocks the filesystem. ISTM that you're asking for the opposite of this - a call that allows the superblock to quiesce the underlying block device into a known, unchanging state. This would allow MD/dm to suspend whatever on-going maintenance operations it has in progress until the filesystem says "we need you to start again" when it gets a thaw call Cheers, Dave. -- Dave Chinner da...@fromorbit.com
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, 2017-11-30 at 20:42 +0100, Luis R. Rodriguez wrote: > On Thu, Nov 30, 2017 at 05:01:13PM +, Bart Van Assche wrote: > > The md resync > > thread must be stopped before a system is frozen. Today the md driver uses > > the kthread freezing mechanism for that purpose. Do you have a plan for > > handling the more complicated scenarios, e.g. a filesystem that exists on > > top > > of an md device where the md device uses one or more files as backing store > > and with the loop driver between the md device and the files? > > Nope not yet. It seems you have given this some thought though so you're > help here is greatly appreciated. In fact the way we should see the long > term 'kill kthread freezing' effort should be a collaborative one. I've > never touched md, so folks more familiar with md should give this some > thought. > > Can for instance md register_pm_notifier() and register_syscore_ops() > and do handy work to pause some work to replace kthread freezing? > Note that I believe a pm notifier is needed in case syscore_suspend() > is not called, say on a suspend fail. Sorry but I don't think that a solution can be based on a notifier mechanism. Freezing has to happen in the order in which drivers and filesystems have been stacked (filesystem > md device > filesystem for the above example). Since the order in which notifiers are called is related to the order in which notifiers have been registered I don't think that a solution for the example I described can be based on notifiers. What I think we need is a mechanism for traversing the storage stack that includes block drivers and an equivalent of freeze_fs() for block drivers. Freezing should occur by calling the freeze_fs() methods for each storage layer starting at the top of the storage stack and proceeding towards the bottom. Bart.
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, Nov 30, 2017 at 05:01:13PM +, Bart Van Assche wrote:
> It's great to see that you are making progress with this work :-) However,
> what's not clear to me is what your (long-term) plan is for freezing
> filesystems that e.g. exist on top of a md RAID1 block device?
The original approach of doing a full swing kill on kthread freezing helpers
in one shot was too much. We must therefore approach this turtle style and
carefully. This patch series currently only addresses filesystems which
have a freeze_fs() callback implemented, that's it. The rest of the kernel
needs more work.
So this work is far as I've gotten so far.
Note, it does *not* remove the kthread freezing API calls on any other drivers,
as such that should still work.
> The md resync
> thread must be stopped before a system is frozen. Today the md driver uses
> the kthread freezing mechanism for that purpose. Do you have a plan for
> handling the more complicated scenarios, e.g. a filesystem that exists on top
> of an md device where the md device uses one or more files as backing store
> and with the loop driver between the md device and the files?
Nope not yet. It seems you have given this some thought though so you're
help here is greatly appreciated. In fact the way we should see the long
term 'kill kthread freezing' effort should be a collaborative one. I've
never touched md, so folks more familiar with md should give this some
thought.
Can for instance md register_pm_notifier() and register_syscore_ops()
and do handy work to pause some work to replace kthread freezing?
Note that I believe a pm notifier is needed in case syscore_suspend()
is not called, say on a suspend fail.
> How about filesystems implemented in user space using the FUSE driver?
Not sure, someone more familiar with FUSE drivers should chime in. Right
now all this does is leverage filesystems which *do* implement freeze_fs()
only, and removes the kthread freezing calls after that. That's all. It
should not affect any other filesystems
Likewise consider filesystems that implement freeze_super() instead (like gfs2)
which don't hold sb->s_umount -- see commit 48b6bca6b7b8309 ("fs: add
freeze_super/thaw_super fs hooks"). If its desirable to address those as well,
it would seem a pair of non-locking iters somehow are needed. More work to
be done.
This all needs to be considered for the future as well.
> Patch 6/11 of this series freezes user space processes before freezing
> filesystems. Will that work for FUSE filesystems?
It doesn't add new code which should negatively affect FUSE drivers.
Long term if we wanted to address FUSE, it may require other work.
In the future then fs_suspend_freeze() may need to work for different
types of filesystems. Right now only filesystems which implement freeze_fs()
are addressed.
Luis
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Wed, 2017-11-29 at 15:23 -0800, Luis R. Rodriguez wrote: > This is a followup from the original RFC which proposed to start > to kill kthread freezing all together [0]. Instead of going straight > out to the jugular for kthread freezing this series only addresses > killing freezer calls on filesystems which implement freeze_fs, after > we let the kernel freeze these filesystems for us on suspend. > > This approach puts on a slow but steady path towards the original goal > though. Each subsystem could look for similar solutions. Even with > filesystems we're not all done yet, after this we'll still have to > decide what to do about filesystems which do not implement freeze_fs(). > > Motivation and problem: > > kthreads have some semantics for freezing, which helps the kernel > freeze them when a system is going to suspend or hibernation. These > semantics are not well defined though, and it actually turns out > pretty hard to get it right. > > Without a proper solution suspend and hibernation are fragile on filesystems, > it can easily break suspend and fixing such issues are in no way trivial [1] > [2]. > > Proposed solution: > > Instead of fixing such semantics and trying to get all filesystems to do it > right, we can easily do away with all freezing calls if the filesystem > implements a proper freeze_fs() callback. The following 9 filesystems have > freeze_fs() implemented as such we can let the kernel issue the callback upon > suspend and thaw on resume automatically on our behalf. > > o xfs > o reiserfs > o nilfs2 > o jfs > o f2fs > o ext4 > o ext2 > o btrfs > > Of these, the following have freezer helpers, which can then be removed > after the kernel automaticaly calls freeze_fs for us on suspend: > > o xfs > o nilfs2 > o jfs > o f2fs > o ext4 > > I've tested this on a system with ext4 and XFS, and have let 0-day go at > without issues. I have branches availabe for linux-next [3] and Linus' > latest tree [4]. > > Further testing, thoughts, reviews, flames are all equally appreciated. Hello Luis, It's great to see that you are making progress with this work :-) However, what's not clear to me is what your (long-term) plan is for freezing filesystems that e.g. exist on top of a md RAID1 block device? The md resync thread must be stopped before a system is frozen. Today the md driver uses the kthread freezing mechanism for that purpose. Do you have a plan for handling the more complicated scenarios, e.g. a filesystem that exists on top of an md device where the md device uses one or more files as backing store and with the loop driver between the md device and the files? How about filesystems implemented in user space using the FUSE driver? Patch 6/11 of this series freezes user space processes before freezing filesystems. Will that work for FUSE filesystems? Thanks, Bart.
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Fri, Dec 1, 2017 at 12:41 AM, Jiri Kosina wrote: > On Fri, 1 Dec 2017, Yu Chen wrote: > >> BTW, is nfs able to be included in this set? I also encountered a >> freeze() failure due to nfs access during that stage recently. > > The freezer usage in NFS is magnitudes more complicated, so it makes sense > to first go after the lower hanging fruit to figure out the viability of > the whole aproach in practice. Ok, thanks!
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Fri, 1 Dec 2017, Yu Chen wrote: > BTW, is nfs able to be included in this set? I also encountered a > freeze() failure due to nfs access during that stage recently. The freezer usage in NFS is magnitudes more complicated, so it makes sense to first go after the lower hanging fruit to figure out the viability of the whole aproach in practice. -- Jiri Kosina SUSE Labs
Re: [PATCH 00/11] fs: use freeze_fs on suspend/hibernate
On Thu, Nov 30, 2017 at 7:23 AM, Luis R. Rodriguez wrote: > This is a followup from the original RFC which proposed to start > to kill kthread freezing all together [0]. Instead of going straight > out to the jugular for kthread freezing this series only addresses > killing freezer calls on filesystems which implement freeze_fs, after > we let the kernel freeze these filesystems for us on suspend. > > > o xfs > o reiserfs > o nilfs2 > o jfs > o f2fs > o ext4 > o ext2 > o btrfs > Thanks for your work on this! I'll test xfs part first. BTW, is nfs able to be included in this set? I also encountered a freeze() failure due to nfs access during that stage recently. -- Thanks, Yu
