Hi,
Sorry to resurrect an old issue, but I've just come across the same (or
very similar-looking) problem. I'm also on an Openstack Swift storage
node with lots of small writes to SSDs as in the OP, running on Debian
Stretch in our case with the following kernel:
Linux swift-storage-1 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3
(2018-03-02) x86_64 GNU/Linux
Kernel logs said:
[4769736.560752] XFS (sdc1): Metadata corruption detected at
xfs_attr3_leaf_write_verify+0xe8/0x100 [xfs], xfs_attr3_leaf block 0xe7dd89b0
[4769736.563285] XFS (sdc1): Unmount and run xfs_repair
[4769736.564554] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[4769736.565818] ffff960ab1d0d000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00
00 ................
[4769736.567064] ffff960ab1d0d010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00
00 ..... ..........
[4769736.568272] ffff960ab1d0d020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
[4769736.569446] ffff960ab1d0d030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
[4769736.570611] XFS (sdc1): xfs_do_force_shutdown(0x8) called from line 1339
of file /build/linux-YDazDa/linux-4.9.82/fs/xfs/xfs_buf.c. Return address =
0xffffffffc06c1ada
[4769736.573226] XFS (sdc1): Corruption of in-memory data detected. Shutting
down filesystem
[4769736.574419] XFS (sdc1): Please umount the filesystem and rectify the
problem(s)
As per the message, I unmounted the filesystem and ran xfs_repair on it. The
first run of xfs_repair told me to mount the filesystem to replay the log,
which I did. I then unmounted it and ran xfs_repair again:
~$ sudo xfs_repair /dev/sdc1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
The filesystem now seems to be back up and running OK. I don't know if there's
any more information I could provide to help track down this issue?
Thanks,
Chris
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1596550
Title:
Metadata corruption detected at xfs_attr3_leaf_write_verify+0xd7/0xf0
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1596550/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
