On 7.02.19 г. 6:07 ч., Nazar Mokrynskyi wrote:
> NOTE: I don't need asistance with data recovery, everything was restored
> shortly from fully automated backups, I only hope this information is useful
> for developers in some way.
>
> So my primary BTRFS filesystem corrupted itself again.
>
<SNIP>
>> [ 82.910327] BTRFS error (device dm-0): unable to find ref byte nr
>> 65470324736 parent 62522359808 root 267 owner 819644 offset 1552384
Funnily enough, this looks like the problem that Josef sent patches for
several hours ago - 'Fix missing reference aborts when resuming snapshot
delete'
>> [ 82.910329] ------------[ cut here ]------------
>> [ 82.910329] BTRFS: Transaction aborted (error -2)
>> [ 82.910339] WARNING: CPU: 14 PID: 6082 at fs/btrfs/extent-tree.c:6950
>> __btrfs_free_extent.isra.72+0x7ad/0xac0 [btrfs]
>> [ 82.910339] Modules linked in: btrfs zstd_compress libcrc32c xor raid6_pq
>> dm_crypt algif_skcipher af_alg intel_rapl x86_pkg_temp_thermal
>> intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi snd_hda_intel kvm
>> snd_usb_audio snd_hda_codec snd_usbmidi_lib snd_hda_core snd_hwdep snd_pcm
>> irqbypass crct10dif_pclmul snd_seq_midi snd_seq_midi_event crc32_pclmul
>> snd_rawmidi ghash_clmulni_intel uvcvideo pcbc snd_seq videobuf2_vmalloc
>> videobuf2_memops videobuf2_v4l2 videobuf2_common snd_seq_device snd_timer
>> aesni_intel videodev snd aes_x86_64 crypto_simd cdc_acm soundcore media
>> input_leds joydev cryptd mei_me glue_helper intel_wmi_thunderbolt
>> intel_cstate mei intel_rapl_perf acpi_pad mac_hid sch_fq_codel parport_pc
>> ppdev lp parport ip_tables x_tables autofs4 overlay nls_iso8859_1 dm_mirror
>> dm_region_hash dm_log
>> [ 82.910354] hid_generic usbhid hid uas usb_storage amdkfd amd_iommu_v2
>> amdgpu nouveau chash gpu_sched ttm mxm_wmi drm_kms_helper syscopyarea
>> sysfillrect sysimgblt nvme fb_sys_fops igb e1000e atlantic drm ahci dca
>> i2c_algo_bit nvme_core libahci wmi video
>> [ 82.910361] CPU: 14 PID: 6082 Comm: btrfs-cleaner Tainted: G W
>> 4.18.0-10-generic #11-Ubuntu
>> [ 82.910361] Hardware name: To Be Filled By O.E.M. To Be Filled By
>> O.E.M./Z370 Professional Gaming i7, BIOS P3.40 11/08/2018
>> [ 82.910367] RIP: 0010:__btrfs_free_extent.isra.72+0x7ad/0xac0 [btrfs]
>> [ 82.910367] Code: 48 8b 40 50 f0 48 0f ba a8 10 ce 00 00 02 0f 92 c0 41
>> 59 84 c0 0f 85 1d c1 09 00 44 89 ee 48 c7 c7 70 e2 ee c0 e8 75 4a 45 f8 <0f>
>> 0b e9 07 c1 09 00 4c 89 e7 e8 14 07 ff ff 48 8b 7d 88 4d 89 f9
>> [ 82.910380] RSP: 0018:ffffa71bc6833bf8 EFLAGS: 00010282
>> [ 82.910381] RAX: 0000000000000000 RBX: 0000000f3e55e000 RCX:
>> 0000000000000006
>> [ 82.910381] RDX: 0000000000000007 RSI: 0000000000000092 RDI:
>> ffff89d09e5964b0
>> [ 82.910382] RBP: ffffa71bc6833ca0 R08: 0000000000000001 R09:
>> 0000000000000805
>> [ 82.910382] R10: 0000000000000004 R11: 0000000000000000 R12:
>> ffff89cfd8338af0
>> [ 82.910382] R13: 00000000fffffffe R14: 0000000e8e9f8000 R15:
>> 000000000000010b
>> [ 82.910383] FS: 0000000000000000(0000) GS:ffff89d09e580000(0000)
>> knlGS:0000000000000000
>> [ 82.910383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 82.910384] CR2: 0000564398e70760 CR3: 000000076ce0a002 CR4:
>> 00000000003606e0
>> [ 82.910384] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [ 82.910385] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>> 0000000000000400
>> [ 82.910385] Call Trace:
>> [ 82.910391] __btrfs_run_delayed_refs+0x20e/0x1010 [btrfs]
>> [ 82.910397] ? add_pinned_bytes+0x67/0x70 [btrfs]
>> [ 82.910402] ? btrfs_free_tree_block+0x167/0x2d0 [btrfs]
>> [ 82.910408] btrfs_run_delayed_refs+0x80/0x190 [btrfs]
>> [ 82.910415] btrfs_should_end_transaction+0x47/0x60 [btrfs]
>> [ 82.910421] btrfs_drop_snapshot+0x3d1/0x800 [btrfs]
>> [ 82.910428] btrfs_clean_one_deleted_snapshot+0xbb/0xf0 [btrfs]
>> [ 82.910434] cleaner_kthread+0x136/0x160 [btrfs]
>> [ 82.910435] kthread+0x120/0x140
>> [ 82.910442] ? btree_submit_bio_start+0x20/0x20 [btrfs]
>> [ 82.910443] ? kthread_bind+0x40/0x40
>> [ 82.910443] ret_from_fork+0x35/0x40
>> [ 82.910444] ---[ end trace 32964933c87d1d28 ]---
>> [ 82.910445] BTRFS: error (device dm-0) in __btrfs_free_extent:6950:
>> errno=-2 No such entry
>> [ 82.910446] BTRFS info (device dm-0): forced readonly
>> [ 82.910447] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:3057:
>> errno=-2 No such entry
>> [ 82.910763] BTRFS error (device dm-0): pending csums is 221184
> It became clear that filesystem is ready to be thown to /dev/null, so I've
> made an image for future investigation and recreated everything from ~10
> minutes old backup.
>
> If there is anything suspicios there and I can run useful tests on filesystem
> image - let me know, I'll spin up qcow2 CoW image and can do anything with it
> in VMs.
>
> Unfortunately, it was just over a month since filesystem creation. Other
> BTRFS filesystem containing backups is perfectly fine (was created late April
> 2018), but it is only used for sending snapshots to it and not much other
> activity.
>
