Re: btrfs becomes read only on removal of folders

2021-02-04 Thread Qu Wenruo




On 2021/2/4 下午7:02, mig...@rozsas.eng.br wrote:

Hi there,

I am using opensuse tumbleweed on a desktop.
More than 1 year ago, I setup a rotating disk to use btrfs and things
went nice until yesterday.
I was done with Civ VI on steam, and decided to remove the associated files.
I've started to remove files and I got the message the file system is read-only.

kimera:~ # findmnt --real -l | grep /mnt/btrfs
/mnt/btrfs/dev/sdc2   btrfs
rw,relatime,space_cache,subvolid=5,subvol=/

kimera:~ # cd /mnt/btrfs/miguel/Steam/
kimera:/mnt/btrfs/miguel/Steam # \ls
GameOverlayRenderer64.dllbin_steamdeps.py
fossilize_engine_filters.json  linux64   servers
steam_subscriber_agreement.txt  tenfoot
ThirdPartyLegalNotices.css   bootstrap.tar.xz  friends
logs  skinssteamapps
ubuntu12_32
ThirdPartyLegalNotices.doc   clientui  graphics
music ssfn9115004176610272489  steamchina
ubuntu12_64
ThirdPartyLegalNotices.html  config
installscriptevalutor_log.txt  package   steam
steamclient.dll update_hosts_cached.vdf
appcache controller_base   legacycompat
publicsteam.sh steamclient64.dll
userdata
bin  depotcachelinux32
resource  steam_msg.sh steamui
kimera:/mnt/btrfs/miguel/Steam #

kimera:/mnt/btrfs/miguel/Steam # rm -rf appcache bin clientui controller_base
rm: cannot remove
'appcache/httpcache/00/00d317630ee7d0825dc6a4cd5ce9f91f50e6993c_da39a3ee5e6b4b0d3255bfef95601890afd80709':
Read-only file system
rm: cannot remove
'appcache/httpcache/01/01edefade28afe779bad14a9d5ff954b0f4c4e60_da39a3ee5e6b4b0d3255bfef95601890afd80709':
Read-only file system
rm: cannot remove
'appcache/httpcache/01/0116c6a41aee246e028415352052bd79d323fc1f_da39a3ee5e6b4b0d3255bfef95601890afd80709':
Read-only file system
...
rm: cannot remove 'controller_base/bigpicture_mouseon.vdf': Read-only
file system
rm: cannot remove 'controller_base/gamepad_generic.vdf': Read-only file system
rm: cannot remove 'controller_base/basicui_gamepad.vdf': Read-only file system
rm: cannot remove 'controller_base/basicui_neptune.vdf': Read-only file system
rm: cannot remove 'controller_base/basicui.vdf': Read-only file system
kimera:/mnt/btrfs/miguel/Steam #

The associated kernel messages can be read here:  https://susepaste.org/80272017


Nothing useful here from that dmesg. It's just booting and no btrfs
specific thing.



The current usage is:
kimera:~ # btrfs filesystem usage -T /mnt/btrfs
Overall:
 Device size:   2.37TiB
 Device allocated:348.06GiB
 Device unallocated:2.03TiB
 Device missing:  0.00B
 Used:287.07GiB
 Free (estimated):  2.09TiB  (min: 1.08TiB)
 Data ratio:   1.00
 Metadata ratio:   2.00
 Global reserve:  349.28MiB  (used: 0.00B)
 Multiple profiles:  no

  Data  Metadata  System
Id Path  singleRAID1 RAID1Unallocated
-- - - -  ---
  1 /dev/sdc2 346.00GiB   1.00GiB 32.00MiB 1.16TiB
  2 /dev/sdc1 -   1.00GiB 32.00MiB   898.97GiB
-- - - -  ---
Total 346.00GiB   1.00GiB 32.00MiB 2.03TiB
Used  286.15GiB 470.45MiB 64.00KiB
kimera:~ #

Then I ran btrfs scrub (btrfs scrub start /mnt/btrfs) and got the
following kernel messages during the scrub:
https://susepaste.org/51166386


This is the real thing. You have a lot of metadata corrupted:

Feb 02 13:43:51 kimera.rozsas.eng.br kernel: BTRFS warning (device
sdc2): checksum/header error at logical 557849608192 on dev /dev/sdc1,
physical 514686976: metadata leaf (level 0) in tree 7

This means your csum tree is corrupted.


Feb 02 13:46:11 kimera.rozsas.eng.br kernel: BTRFS error (device sdc2):
parent transid verify failed on 557851131904 wanted 86067 found 76934

This means some of your metadata write didn't reach disk, and metadata
COW is broken, causing metadata corruption.

I'd recommend to just mount RO and copy as many data as you can.

Thanks,
Qu



And scrub ended with the status:
kimera:/mnt/btrfs/miguel/Steam # btrfs scrub status /mnt/btrfs
UUID: af3bd7fe-5796-4248-9917-3e71a5a56ec6
Scrub started:Tue Feb  2 13:43:34 2021
Status:   aborted
Duration: 0:02:37
Total to scrub:   287.07GiB
Rate: 106.22MiB/s
Error summary:verify=79 csum=5
  Corrected:  7
  Uncorrectable:  77
  Unverified: 0

So, what is going here ?
How can I fix this FS ?

Sorry to not know so much about how to fix a btrfs, it is my first
experience with btrfs since I migrated from Ubuntu to openSuSE more
than 1 year ago.

my system:
Operating System: openSUSE Tumbleweed 20210121
KDE Plasma Version: 5.20.5
KDE Frameworks Version: 5.78.0
Q

Re: btrfs becomes read only on removal of folders

2021-02-04 Thread Chris Murphy
On Thu, Feb 4, 2021 at 4:04 AM mig...@rozsas.eng.br
 wrote:


> https://susepaste.org/51166386

It's raid1 metadata on the same physical device, so depending on the
device, if the metadata writes are concurrent they may end up being
deduped by the drive firmware no matter that they're supposed to go to
separate partitions.

Feb 02 13:43:37 kimera.rozsas.eng.br kernel: BTRFS error (device
sdc2): unable to fixup (regular) error at logical 557651984384 on dev
/dev/sdc1
Feb 02 13:43:37 kimera.rozsas.eng.br kernel: BTRFS error (device
sdc2): unable to fixup (regular) error at logical 557651869696 on dev
/dev/sdc1

This suggests both copies are bad.

> So, what is going here ?
> How can I fix this FS ?

I would do a memory test, the longer the better. Memory defects can be evasive.

Take the opportunity to freshen backups while the file system still
mounts read-only. And then also provide the output from

btrfs check --readonly

It might be something that can be repaired, but until you've isolated
memory, any repair or new writes can end up with the same problem. But
if it's not just a bit flip, and both copies are bad, then it's
usually a case of backup, reformat, restore. Hence the backup needs to
be the top priority; and checking the memory the second priority.

-- 
Chris Murphy