[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Attachment added: "test-loop.sh" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+attachment/5524208/+files/test-loop.sh -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. And the stress-ng test-case as well. (Numbers/details in the LP bug.) [Other info] The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; only Focal and Bionic need it. There are little changes in the patches between Focal and Bionic (mostly minor backport adjustments, mainly due to no vm_fault_t) but unfortunately that needs separate versions for most patches. ... [Original Bug Description] [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Attachment added: "fstests-check-logs.sh" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+attachment/5524207/+files/fstests-check-logs.sh -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. And the stress-ng test-case as well. (Numbers/details in the LP bug.) [Other info] The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; only Focal and Bionic need it. There are little changes in the patches between Focal and Bionic (mostly minor backport adjustments, mainly due to no vm_fault_t) but unfortunately that needs separate versions for most patches. ... [Original Bug Description] [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHel
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Attaching the scripts used for testing (run/check xfstests, and run stress-ng) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. And the stress-ng test-case as well. (Numbers/details in the LP bug.) [Other info] The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; only Focal and Bionic need it. There are little changes in the patches between Focal and Bionic (mostly minor backport adjustments, mainly due to no vm_fault_t) but unfortunately that needs separate versions for most patches. ... [Original Bug Description] [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Attachment added: "fstests-run.sh" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+attachment/5524206/+files/fstests-run.sh -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. And the stress-ng test-case as well. (Numbers/details in the LP bug.) [Other info] The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; only Focal and Bionic need it. There are little changes in the patches between Focal and Bionic (mostly minor backport adjustments, mainly due to no vm_fault_t) but unfortunately that needs separate versions for most patches. ... [Original Bug Description] [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Test results with stress-ng. The original kernels hit the issue (very often on Bionic, less often on Focal.) The modified kernels don't (in 2x the number of runs on both Bionic and Focal.) Number of tests (stress-ng + crash + mount) that mount PASSED or FAILED (ie, problem was hit.) --- Bionic / Original: ubuntu@mfo-sf250123-b:~$ grep -e ^FAILED -e ^PASSED /test.log | cut -d' ' -f1 | sort | uniq -c 189 FAILED 573 PASSED Bionic / Modified: ubuntu@mfo-sf250123-b2:~$ grep -e ^FAILED -e ^PASSED /test.log | cut -d' ' -f1 | sort | uniq -c 1362 PASSED Focal / Original: ubuntu@mfo-sf250123-f:~$ grep -e ^FAILED -e ^PASSED /test.log | cut -d' ' -f1 | sort | uniq -c 1 FAILED 693 PASSED Focal / Modified: ubuntu@mfo-sf250123-f2:~$ grep -e ^FAILED -e ^PASSED /test.log | cut -d' ' -f1 | sort | uniq -c 1354 PASSED -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. And the stress-ng test-case as well. (Numbers/details in the LP bug.) [Other info] The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; only Focal and Bionic need it. There are little changes in the patches between Focal and Bionic (mostly minor backport adjustments, mainly due to no vm_fault_t) but unfortunately that needs separate versions for most patches. ... [Original Bug Description] [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools i
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Regression test results with (x)fstests. Focal, data=ordered: no regression. Original: --- $ ./fstests-check-logs.sh orig-ordered/xfstests.log* Kernel version: 5.4.0-84-generic #94-Ubuntu - Number of Runs: 10 - Always fail (failed every time) 10 ext4/024 10 ext4/034 10 ext4/045 10 generic/044 10 generic/045 10 generic/046 10 generic/082 10 generic/219 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/381 10 generic/382 10 generic/397 10 generic/398 10 generic/399 10 generic/419 10 generic/421 10 generic/429 10 generic/435 10 generic/440 10 generic/484 10 generic/548 10 generic/549 10 generic/550 10 generic/566 10 generic/587 10 generic/600 - Flaky (failed, but not every time) 4 generic/388 Modified: --- $ ./fstests-check-logs.sh mod-ordered/xfstests.log* Kernel version: 5.4.0-84-generic #94+test20210906b1 - Number of Runs: 10 - Always fail (failed every time) 10 ext4/024 10 ext4/034 10 ext4/045 10 generic/044 10 generic/045 10 generic/046 10 generic/082 10 generic/219 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/381 10 generic/382 10 generic/397 10 generic/398 10 generic/399 10 generic/419 10 generic/421 10 generic/429 10 generic/435 10 generic/440 10 generic/484 10 generic/548 10 generic/549 10 generic/550 10 generic/566 10 generic/587 10 generic/600 - Flaky (failed, but not every time) 6 generic/388 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both B
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Regression test results with (x)fstests. Bionic, data=journal: no regression. Original: --- $ ./fstests-check-logs.sh orig-journal/xfstests.log* Kernel version: 5.4.0-84-generic #94-Ubuntu - Number of Runs: 10 - Always fail (failed every time) 10 ext4/024 10 ext4/034 10 ext4/045 10 generic/082 10 generic/219 10 generic/223 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/381 10 generic/382 10 generic/397 10 generic/398 10 generic/399 10 generic/419 10 generic/421 10 generic/429 10 generic/435 10 generic/440 10 generic/441 10 generic/537 10 generic/548 10 generic/549 10 generic/550 10 generic/566 10 generic/600 - Flaky (failed, but not every time) 2 generic/388 2 generic/475 9 generic/347 Modified: --- $ ./fstests-check-logs.sh mod-journal/xfstests.log* Kernel version: 5.4.0-84-generic #94+test20210906b1 - Number of Runs: 10 - Always fail (failed every time) 10 ext4/024 10 ext4/034 10 ext4/045 10 generic/082 10 generic/219 10 generic/223 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/381 10 generic/382 10 generic/397 10 generic/398 10 generic/399 10 generic/419 10 generic/421 10 generic/429 10 generic/435 10 generic/440 10 generic/441 10 generic/537 10 generic/548 10 generic/549 10 generic/550 10 generic/566 10 generic/600 - Flaky (failed, but not every time) 1 generic/388 6 generic/484 * 9 generic/347 * This test has been confirmed to fail on the original kernel too, with a larger sample size (more runs w/ only the generic/484 test.) Even though it takes more than a few hundred runs to manifest a few times, it demonstrates that the problematic behavior is also there, and might be just a timing change that made it happen more often now (but it's still flaky, doesn't fail every single time.) $ ls -1 xfstests.log* | wc -l 518 $ grep -h ^Failures: xfstests.log* | sort | uniq -c 6 Failures: generic/484 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Regression test results with (x)fstests. Bionic, data=journal: no regression. Original: --- $ ./fstests-check-logs.sh orig-journal/xfstests.log* Kernel version: 4.15.0-156-generic #163-Ubuntu - Number of Runs: 10 - Always fail (failed every time) 10 ext4/045 10 generic/082 10 generic/219 10 generic/223 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/347 10 generic/371 10 generic/381 10 generic/382 10 generic/441 10 generic/491 10 generic/553 10 generic/554 10 generic/555 10 generic/565 10 generic/566 10 generic/569 10 generic/598 10 generic/600 10 generic/623 10 generic/629 10 shared/298 - Flaky (failed, but not every time) 3 generic/475 6 generic/388 Modified: --- $ ./fstests-check-logs.sh mod-journal/xfstests.log* Kernel version: 4.15.0-156-generic #163+test20210906b1 - Number of Runs: 10 - Always fail (failed every time) 10 ext4/045 10 generic/082 10 generic/219 10 generic/223 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/381 10 generic/382 10 generic/441 10 generic/491 10 generic/553 10 generic/554 10 generic/555 10 generic/565 10 generic/566 10 generic/569 10 generic/598 10 generic/600 10 generic/623 10 generic/629 10 shared/298 - Flaky (failed, but not every time) 6 generic/347 (improved from orig, now flaky, prev always) 6 generic/388 9 generic/371 (improved from orig, now flaky, prev always) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with kernel ve
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Regression test results with (x)fstests. Bionic, data=ordered: no regression. Original: --- $ ./fstests-check-logs.sh orig-ordered/xfstests.log* Kernel version: 4.15.0-156-generic #163-Ubuntu - Number of Runs: 10 - Always fail (failed every time) 10 ext4/045 10 generic/044 10 generic/045 10 generic/046 10 generic/082 10 generic/095 10 generic/219 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/299 10 generic/300 10 generic/381 10 generic/382 10 generic/491 10 generic/553 10 generic/554 10 generic/555 10 generic/565 10 generic/566 10 generic/569 10 generic/587 10 generic/598 10 generic/600 10 generic/623 10 generic/629 10 shared/298 - Flaky (failed, but not every time) 6 generic/388 Modified: --- $ ./fstests-check-logs.sh mod-ordered/xfstests.log* Kernel version: 4.15.0-156-generic #163+test20210906b1 - Number of Runs: 10 - Always fail (failed every time) 10 ext4/045 10 generic/044 10 generic/045 10 generic/046 10 generic/082 10 generic/095 10 generic/219 10 generic/230 10 generic/231 10 generic/232 10 generic/233 10 generic/235 10 generic/270 10 generic/299 10 generic/300 10 generic/381 10 generic/382 10 generic/491 10 generic/553 10 generic/554 10 generic/555 10 generic/565 10 generic/566 10 generic/569 10 generic/587 10 generic/598 10 generic/600 10 generic/623 10 generic/629 10 shared/298 - Flaky (failed, but not every time) 9 generic/388 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after that, but before such journal entry makes it to the filesystem, the journal replay on the next mount just fails, and the filesystem now requires fsck. (apparently curtin might set up /etc/fstab with passno=0, requiring manual intervention.) [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! [39751.114435] JBD2: Invalid checksum recovering block 87305 in log [39751.146133] JBD2: Invalid checksum recovering block 88039 in log [39751.195950] JBD2: Invalid checksum recovering block 49633 in log [39751.265158] JBD2: recovery failed [39751.265163] EXT4-fs (vdc): error loading journal [Fix] The fix is to write-protect the pages during journal transaction commit, so that writes to mapped pages hit a page fault, then ext4's page_mkwrite hook can block until the commit finishes and the buffers can be modified. In order to do that, add jbd2 journal callbacks that the filesystems can customize, called before/after the critical region in transaction commit, then have ext4 in data journaling mode to write-protect the pages whose buffers are being committed (and handle cases that need pages redirtied.) The changes are restricted to the data journaling mode and page_mkwrite hook, and other modes/paths use the same code/behavior in the callbacks. [Test Case] Set up an ext4 filesystem in data journaling mode, and run stress-ng's mmap file test on it, then crash the system after a bit; check whether the filesystem can mount again or not (i.e., with jbd2 checksum errors.) # mkfs.ext4 $DEV # mount -o data=journal $DEV $DIR # cd $DIR # stress-ng --mmap $((4*$(nproc))) --mmap-file & # sleep 60 # echo c >/proc/sysrq-trigger ... # mount -o data=journal $DEV $DIR # PASS/FAIL. # dmesg | tail [Regression Potential] Regressions would likely manifest in ext4 data journaling mode (which is not the default mode, 'ordered') with memory mapped access, as the other modes/paths are largely unaffected by the changes/same behavior. This has been tested with (x)fstests, that showed no regressions on data=ordered and data=journal on both Bionic and Focal (with k
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Description changed: + [Impact] + + With mmap()ed files on ext4's data journaling it's possible to change + a mapped page's buffers contents during their jbd2 transaction commit + (as currently nothing prevents/blocks the write access at that time.) + + This might happen between the buffers checksum calculation and actual + write to journal, so the (old) checksum is invalid for the (new) data. + + If the system crashes after that, but before such journal entry makes + it to the filesystem, the journal replay on the next mount just fails, + and the filesystem now requires fsck. (apparently curtin might set up + /etc/fstab with passno=0, requiring manual intervention.) + + [39751.096455] EXT4-fs: Warning: mounting with data=journal disables delayed allocation and O_DIRECT support! + [39751.114435] JBD2: Invalid checksum recovering block 87305 in log + [39751.146133] JBD2: Invalid checksum recovering block 88039 in log + [39751.195950] JBD2: Invalid checksum recovering block 49633 in log + [39751.265158] JBD2: recovery failed + [39751.265163] EXT4-fs (vdc): error loading journal + + [Fix] + + The fix is to write-protect the pages during journal transaction commit, + so that writes to mapped pages hit a page fault, then ext4's page_mkwrite + hook can block until the commit finishes and the buffers can be modified. + + In order to do that, add jbd2 journal callbacks that the filesystems can + customize, called before/after the critical region in transaction commit, + then have ext4 in data journaling mode to write-protect the pages whose + buffers are being committed (and handle cases that need pages redirtied.) + + The changes are restricted to the data journaling mode and page_mkwrite + hook, and other modes/paths use the same code/behavior in the callbacks. + + [Test Case] + + Set up an ext4 filesystem in data journaling mode, and run stress-ng's + mmap file test on it, then crash the system after a bit; check whether + the filesystem can mount again or not (i.e., with jbd2 checksum errors.) + + # mkfs.ext4 $DEV + # mount -o data=journal $DEV $DIR + # cd $DIR + # stress-ng --mmap $((4*$(nproc))) --mmap-file & + # sleep 60 + # echo c >/proc/sysrq-trigger + ... + # mount -o data=journal $DEV $DIR # PASS/FAIL. + # dmesg | tail + + [Regression Potential] + + Regressions would likely manifest in ext4 data journaling mode (which + is not the default mode, 'ordered') with memory mapped access, as the + other modes/paths are largely unaffected by the changes/same behavior. + + This has been tested with (x)fstests, that showed no regressions on + data=ordered and data=journal on both Bionic and Focal (with kernel + versions 4.15.0-156-generic and 5.4.0-84-generic) w/in 10 runs each. + And the stress-ng test-case as well. (Numbers/details in the LP bug.) + + [Other info] + + The patchset is applied on 5.10, so Hirsute (5.11) is already fixed; + only Focal and Bionic need it. + + There are little changes in the patches between Focal and Bionic + (mostly minor backport adjustments, mainly due to no vm_fault_t) + but unfortunately that needs separate versions for most patches. + + ... + + + [Original Bug Description] + [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] With mmap()ed files on ext4's data journaling it's possible to change a mapped page's buffers contents during their jbd2 transaction commit (as currently nothing prevents/blocks the write access at that time.) This might happen between the buffers checksum calculation and actual write to journal, so the (old) checksum is invalid for the (new) data. If the system crashes after
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Also affects: linux (Ubuntu Hirsute) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Mauricio Faria de Oliveira (mfo) ** Changed in: linux (Ubuntu Focal) Status: New => In Progress ** Changed in: linux (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Focal) Assignee: (unassigned) => Mauricio Faria de Oliveira (mfo) ** Changed in: linux (Ubuntu Hirsute) Status: New => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: In Progress Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: In Progress Status in linux source package in Hirsute: Fix Released Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Changed in: linux (Ubuntu Eoan) Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Confirmed Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Won't Fix Status in linux source package in Focal: New Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Changed in: linux (Ubuntu Disco) Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Confirmed Status in linux source package in Disco: Won't Fix Status in linux source package in Eoan: Incomplete Status in linux source package in Focal: New Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed ** Also affects: linux (Ubuntu Eoan) Importance: Undecided Status: Confirmed ** Also affects: linux (Ubuntu Ff-series) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Disco) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Status: New => Confirmed ** Changed in: linux (Ubuntu Eoan) Status: Confirmed => New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: New Status in linux source package in Bionic: Confirmed Status in linux source package in Disco: New Status in linux source package in Eoan: New Status in linux source package in FF-Series: New Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
And the precursor to that the upstream thread in comment #1: https://www.spinics.net/lists/linux-fsdevel/msg151486.html And here's a captured filesystem image (referenced in above thread): https://people.canonical.com/~dannf/md2.e2ic.qcow2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: New Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1847340] Re: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap
Upstream thread: https://lists.openwall.net/linux-ext4/2019/08/30/1 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1847340 Title: ext4 journal recovery fails w/ data=journal + journal_checksum + mmap Status in linux package in Ubuntu: New Bug description: [Impact] In the event of a loss of power, ext4 filesystems mounted w/ data=journal,journal_checksum are subject to a corruption issue that requires a fsck to recover. This is exacerbated by installations by curtin that set passno=0 in /etc/fstab, preventing fsck from running automatically and thus requiring a manual recovery. And *that* is further exacerbated because initramfs-tools is smart enough to not include fsck.ext4 when passno=0 is detected in /etc/fstab, requiring the user to boot from recovery media. [Test Case] Forcibly power cycle a system running 'stress-ng --dir 0'. I've created a package to automate the reproduction: https://git.launchpad.net/~dannf/+git/dgx2-ext4-csum-repro?h=master [Fix] [Regression Risk] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1847340/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
