Fixing up format to match SRU Justification
** Description changed:
- [Expected]
+ SRU Justification
- Page-cache reclaim runs without faulting, regardless of the upper MMIO
- region splitting System RAM.
+ [Impact]
- [What's happened]
+ Expected:
+ Page-cache reclaim runs without faulting, regardless of the upper MMIO region
splitting System RAM.
- Three of our Azure VMs (`Standard_L16as_v4` with the upper MMIO region
- below `0x1000000000` on `linux-azure-6.8 6.8.0-1062.69~22.04.1`) running
- as Elasticsearch data-nodes oopsed - two in kswapd0, one in the
- elasticsearch process itself. Crashed machines were rebuilt urgently,
- non-crashed ones a bit later, so we got apport and /proc/iomem from
- them:
+ What's happened:
+ Three of our Azure VMs (`Standard_L16as_v4` with the upper MMIO region below
`0x1000000000` on `linux-azure-6.8 6.8.0-1062.69~22.04.1`) running as
Elasticsearch data-nodes oopsed - two in kswapd0, one in the elasticsearch
process itself. Crashed machines were rebuilt urgently, non-crashed ones a bit
later, so we got apport and /proc/iomem from them:
- oops-node1-kswapd.log # fragment from crashed node
- oops-node2-kswapd.log # fragment from crashed node
- oops-node3-elasticsearch.log # fragment from crashed node
- proc-iomem-node4.log # output of same set machine, not crashed
- apport-node4.txt # apport of same set machine, not crashed
- [Suspect]
-
+ Suspect:
An oops in `shrink_folio_list()`.
- [Steps to reproduce]
+ [Fix]
- ./reproduce.sh azure-1062 # or: noble-31 noble-137 noble-139
- azure-1064 azure-1062-read-deleted azure-1062-mglru-off
+ Probably it's in `mm/vmscan.c:1277`
+
+ if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
+ flags |= TTU_SYNC;
+
+ The line appeared in `Ubuntu-6.8.0-48.48` with the backport for LP
+ #2076147.
+
+ 73bc32875ee9 mm: hold PTL from the first PTE while reclaiming a
+ large folio
+
+ Mainline dropped it - it seems that wasn't backported.
+
+ e5a119c4a683 mm/vmscan: drop checking if _deferred_list is empty
+ before using TTU_SYNC
+
+ The `azure-1062-read-deleted` target applies exactly this one-line
+ removal.
+
+ * Clean cherry-pick of e5a119c4a683: "mm/vmscan: drop checking if
+ _deferred_list is empty before using TTU_SYNC" upstream
+
+ [Test Plan]
+
+ Steps to reproduce:
+ ./reproduce.sh azure-1062 # or: noble-31 noble-137 noble-139
azure-1064 azure-1062-read-deleted azure-1062-mglru-off
The script fetches the kernel from Launchpad, builds a rootfs, and boots
QEMU: 4 GiB guest, 4 x 128 MB holes via `memmap=`, XFS on /dev/vda with
a 16 GiB file, read via mmap with jumps to random offsets.
Ext4 is untested.
- [Versions tested]
-
- target kernel result
log
- noble-31 linux 6.8.0-31.31 no oops,
5,000,000 jumps reproduce-noble-31.log
- noble-137 linux 6.8.0-137.137 oops, before
jump 200,000 reproduce-noble-137.log
- noble-139 linux 6.8.0-139.139 (proposed) oops, before
jump 400,000 reproduce-noble-139.log
- azure-1062 linux-azure-6.8 1062.69~22.04.1 oops, before
jump 200,000 reproduce-azure-1062.log
- azure-1064 linux-azure-6.8 1064.72~22.04.1 oops, before
jump 800,000 reproduce-azure-1064.log
- azure-1062-read-deleted rebuilt from 1062 source no oops,
5,000,000 jumps reproduce-azure-1062-read-deleted.log
- azure-1062-mglru-off rebuilt from 1062 source oops, before
jump 800,000 reproduce-azure-1062-mglru-off.log
+ Versions tested:
+ target kernel result
log
+ noble-31 linux 6.8.0-31.31 no oops,
5,000,000 jumps reproduce-noble-31.log
+ noble-137 linux 6.8.0-137.137 oops, before
jump 200,000 reproduce-noble-137.log
+ noble-139 linux 6.8.0-139.139 (proposed) oops, before
jump 400,000 reproduce-noble-139.log
+ azure-1062 linux-azure-6.8 1062.69~22.04.1 oops, before
jump 200,000 reproduce-azure-1062.log
+ azure-1064 linux-azure-6.8 1064.72~22.04.1 oops, before
jump 800,000 reproduce-azure-1064.log
+ azure-1062-read-deleted rebuilt from 1062 source no oops,
5,000,000 jumps reproduce-azure-1062-read-deleted.log
+ azure-1062-mglru-off rebuilt from 1062 source oops, before
jump 800,000 reproduce-azure-1062-mglru-off.log
In most of these runs the fault came twice: first in kswapd0, then in
the reading process itself. This matches both patterns we saw in
production.
All these logs come from runs with 4 holes and a limit of 5,000,000
jumps. Fewer holes make the fault less likely - with a single hole we
still got it, but the run took much longer. With 4 holes a bad kernel
faults in minutes, so we did not have to run each version for days. We
can run more tests if needed.
137.137 (noble Updates) and 1064.72 (jammy Updates) are the newest
published builds; 31.31 is noble Release before the backport named under
[Cause].
-
- [Cause]
-
- Probably it's in `mm/vmscan.c:1277`
-
- if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
- flags |= TTU_SYNC;
-
- The line appeared in `Ubuntu-6.8.0-48.48` with the backport for LP
- #2076147.
-
- 73bc32875ee9 mm: hold PTL from the first PTE while reclaiming a
- large folio
-
- Mainline dropped it - it seems that wasn't backported.
-
- e5a119c4a683 mm/vmscan: drop checking if _deferred_list is empty
- before using TTU_SYNC
-
- The `azure-1062-read-deleted` target applies exactly this one-line
- removal.
-
- If we filed this against the wrong package, or if you need anything else
- from us, please let us know.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163714
Title:
oops in shrink_folio_list(): out-of-bounds read of
folio->_deferred_list
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2163714/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
