On Mon, 2 Nov 2026, Stefan Monnier wrote:
I've had this happen maybe a dozen times over the years and I wonder if
anyone knows what I'm doing to cause it.
Seeing the hordes rushing to drown you in suggestions, I'll take
my chance. Based on what you say, I suspect I won't tell you anything
you don't already know, tho. 🙁
I reboot and grub fails to find anything. ls shows memdisk and the main disk
but nothing else. insmod lvm etc makes no difference at all.
But once you run pvresize on the PV, grub is happy again and finds
everything it needs to boot.
- I suspect you might get better info from Grub insiders.
Maybe it's worth opening a bug report, even if you don't have much
concrete data to provide, in the hope to get them to tell you how you
might collect useful data next time it happens.
Thanks!
I wasn't clear in my original email but this isn't limited to one
particular machine or one particular disk. If it weren't for the fact
that it happens apparently completely randomly, I'd assume there was a
bug from some old version of something that was leaving the lvm metadata
in an inconsistent state somewhere, nothing noticed until grub was
upgraded and gets upset until it's fixed.
But while this most recent incidence happened after an upgrade, it
happened on a reboot well after the upgrade was done. (in fact, in this
case I was doing some final cleanup on the next day and had removed old
config files that hadn't been purged, nothing else, and just did a
"final reboot to check I'd not broken something during the purge")
And I've had other instances where it's even more random. Probably
always following a kernel upgrade (because that's just about the only
time I reboot), but given that this is grub failing to even find the VG,
it surely cannot be kernel related.
I've had it happen on both SSD and magnetic.
- Maybe something corrupts some copy of the LVM info, which is the one
Grub uses, whereas `pvresize` is able to read some other copy and
silently refresh/fix the corrupted copy?
I've done some more digging:
vg-mail {
id = "u2IS1S-14ue-2bPk-NV20-jP7f-Aogl-g1Fe8U"
- seqno = 38233
+ seqno = 38354
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
@@ -27,7 +27,7 @@ vg-mail {
status = ["ALLOCATABLE"]
flags = []
- dev_size = 128040960 # 61.0547 Gigabytes
+ dev_size = 128038912 # 61.0537 Gigabytes
pe_start = 2048
pe_count = 15629 # 61.0508 Gigabytes
}
So that pvresize did change something - but I can see that that size
hasn't changed as far back as Feb 2025 (which is the oldest backup of
the vg metadata I have online). (sadly, it turns out that lvm/archive is
excluded from etckeeper by default)
I've found this:
https://lists.gnu.org/archive/html/grub-devel/2024-05/msg00040.html
And this was booting on an old xen installation where I see:
ls -al /usr/lib/grub-xen/grub-i386-xen_pvh.bin
-rw-r--r-- 1 root root 1572400 Nov 13 2022
/usr/lib/grub-xen/grub-i386-xen_pvh.bin
I'm almost certain that this is the problem. From my current metadata:
pvck --dump headers /dev/xvda
mda_header_1.raw_locn[0].offset 373760
mda_header_1.raw_locn[0].size 3207
And from seqno 38233 to 38354 is 121 updates - which is 388047 bytes.
That would put it wrapping somewere around 116 updates ago - or roughly
5 metadata updates after I upgraded.
I can't quite make the numbers line up, I backup 5 lvs so I'd expect
roughly 10 metadata updates per day but it looks closer to 20, but I'm
starting to think that this is the problem, and I've hit it because I
backup every day and so am constantly having a churn of metadata
versions.
And the pvresize could be a red herring - the significant effect is that
it causes a metadata rewrite. It seems to bump the seqno even if nothing
changes.
So I guess my next job is to write a script to detect when the metadata
is close to wrapping and bump it past the wrap. I keep some old systems
for testing purposes, so upgrade isn't always possible.
Thanks for your reply that prompted me to dig more.
Tim.
