Thanks Qu, Just before I’ll go and accidentally mess up this FS more - I’ve mentioned originally that this problem started with FS not being able to create a snapshot ( it would get remounted RO automatically ) for about a month, and when I’ve realised that there is a problem like that I’ve attempted a full FS balance that caused this FS to be unmountable. Is there any other debug you would require before I proceed (I’ve got a lot i
On 16 Feb 2017, at 01:26, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: At 02/15/2017 10:11 PM, Tomasz Kusmierz wrote: So guys, any help here ? I’m kinda stuck now with system just idling and doing nothing while I wait for some feedback ... Sorry for the late reply. Busying debugging a kernel bug. On 14 Feb 2017, at 19:38, Tomasz Kusmierz <tom.kusmi...@gmail.com> wrote: [root@server ~]# btrfs-show-super -af /dev/sdc superblock: bytenr=65536, device=/dev/sdc --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x17d56ce0 [match] This superblock is good. bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 0576d577-8954-4a60-a02b-9492b3c29318 label main_pool generation 150682 root 5223857717248 sys_array_size 321 chunk_root_generation 150678 root_level 1 chunk_root 8669488005120 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 16003191472128 bytes_used 6411278503936 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 8 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 150682 uuid_tree_generation 150679 dev_item.uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 dev_item.fsid 0576d577-8954-4a60-a02b-9492b3c29318 [match] dev_item.type 0 dev_item.total_bytes 2000398934016 dev_item.bytes_used 1647136735232 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896) length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10 io_align 65536 io_width 65536 sector_size 4096 num_stripes 8 sub_stripes 2 stripe 0 devid 7 offset 1083674984448 dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6 stripe 1 devid 8 offset 1083674984448 dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c stripe 2 devid 1 offset 1365901312 dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 stripe 3 devid 3 offset 1345978368 dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755 stripe 4 devid 4 offset 1345978368 dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a stripe 5 devid 5 offset 1345978368 dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197 stripe 6 devid 6 offset 1345978368 dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1 stripe 7 devid 2 offset 1345978368 dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841 And I didn't see anything wrong in sys_chunk_array. Would you please try to mount the fs with latest kernel? Better later than v4.9, as in that version extra kernel messages are introduced to give more details about what's going wrong. Thanks, Qu backup_roots[4]: backup 0: backup_tree_root: 5223857717248 gen: 150680 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150680 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224791523328 gen: 150680 level: 1 backup_csum_root: 5224802140160 gen: 150680 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 1: backup_tree_root: 5224155807744 gen: 150681 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5224156233728 gen: 150681 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224633155584 gen: 150681 level: 1 backup_csum_root: 5224634941440 gen: 150681 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 2: backup_tree_root: 5223857717248 gen: 150682 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150682 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224622358528 gen: 150682 level: 1 backup_csum_root: 5224675344384 gen: 150682 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 3: backup_tree_root: 11179477942272 gen: 150679 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 11179488018432 gen: 150679 level: 2 backup_fs_root: 6217817456640 gen: 150497 level: 0 backup_dev_root: 5224337244160 gen: 150679 level: 1 backup_csum_root: 11179492540416 gen: 150679 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 superblock: bytenr=67108864, device=/dev/sdc --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xb7b4442e [match] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 0576d577-8954-4a60-a02b-9492b3c29318 label main_pool generation 150682 root 5223857717248 sys_array_size 321 chunk_root_generation 150678 root_level 1 chunk_root 8669488005120 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 16003191472128 bytes_used 6411278503936 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 8 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 150682 uuid_tree_generation 150679 dev_item.uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 dev_item.fsid 0576d577-8954-4a60-a02b-9492b3c29318 [match] dev_item.type 0 dev_item.total_bytes 2000398934016 dev_item.bytes_used 1647136735232 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896) length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10 io_align 65536 io_width 65536 sector_size 4096 num_stripes 8 sub_stripes 2 stripe 0 devid 7 offset 1083674984448 dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6 stripe 1 devid 8 offset 1083674984448 dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c stripe 2 devid 1 offset 1365901312 dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 stripe 3 devid 3 offset 1345978368 dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755 stripe 4 devid 4 offset 1345978368 dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a stripe 5 devid 5 offset 1345978368 dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197 stripe 6 devid 6 offset 1345978368 dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1 stripe 7 devid 2 offset 1345978368 dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841 backup_roots[4]: backup 0: backup_tree_root: 5223857717248 gen: 150680 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150680 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224791523328 gen: 150680 level: 1 backup_csum_root: 5224802140160 gen: 150680 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 1: backup_tree_root: 5224155807744 gen: 150681 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5224156233728 gen: 150681 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224633155584 gen: 150681 level: 1 backup_csum_root: 5224634941440 gen: 150681 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 2: backup_tree_root: 5223857717248 gen: 150682 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150682 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224622358528 gen: 150682 level: 1 backup_csum_root: 5224675344384 gen: 150682 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 3: backup_tree_root: 11179477942272 gen: 150679 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 11179488018432 gen: 150679 level: 2 backup_fs_root: 6217817456640 gen: 150497 level: 0 backup_dev_root: 5224337244160 gen: 150679 level: 1 backup_csum_root: 11179492540416 gen: 150679 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 superblock: bytenr=274877906944, device=/dev/sdc --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x4a33121f [match] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 0576d577-8954-4a60-a02b-9492b3c29318 label main_pool generation 150682 root 5223857717248 sys_array_size 321 chunk_root_generation 150678 root_level 1 chunk_root 8669488005120 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 16003191472128 bytes_used 6411278503936 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 8 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x161 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 150682 uuid_tree_generation 150679 dev_item.uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 dev_item.fsid 0576d577-8954-4a60-a02b-9492b3c29318 [match] dev_item.type 0 dev_item.total_bytes 2000398934016 dev_item.bytes_used 1647136735232 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896) length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10 io_align 65536 io_width 65536 sector_size 4096 num_stripes 8 sub_stripes 2 stripe 0 devid 7 offset 1083674984448 dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6 stripe 1 devid 8 offset 1083674984448 dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c stripe 2 devid 1 offset 1365901312 dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8 stripe 3 devid 3 offset 1345978368 dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755 stripe 4 devid 4 offset 1345978368 dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a stripe 5 devid 5 offset 1345978368 dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197 stripe 6 devid 6 offset 1345978368 dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1 stripe 7 devid 2 offset 1345978368 dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841 backup_roots[4]: backup 0: backup_tree_root: 5223857717248 gen: 150680 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150680 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224791523328 gen: 150680 level: 1 backup_csum_root: 5224802140160 gen: 150680 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 1: backup_tree_root: 5224155807744 gen: 150681 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5224156233728 gen: 150681 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224633155584 gen: 150681 level: 1 backup_csum_root: 5224634941440 gen: 150681 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 2: backup_tree_root: 5223857717248 gen: 150682 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 5223867383808 gen: 150682 level: 2 backup_fs_root: 0 gen: 0 level: 0 backup_dev_root: 5224622358528 gen: 150682 level: 1 backup_csum_root: 5224675344384 gen: 150682 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 backup 3: backup_tree_root: 11179477942272 gen: 150679 level: 1 backup_chunk_root: 8669488005120 gen: 150678 level: 1 backup_extent_root: 11179488018432 gen: 150679 level: 2 backup_fs_root: 6217817456640 gen: 150497 level: 0 backup_dev_root: 5224337244160 gen: 150679 level: 1 backup_csum_root: 11179492540416 gen: 150679 level: 3 backup_total_bytes: 16003191472128 backup_bytes_used: 6411278503936 backup_num_devices: 8 On 14 February 2017 at 00:25, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: At 02/14/2017 08:23 AM, Tomasz Kusmierz wrote: Forgot to mention: btrfs inspect-internal dump-super -af /dev/sdc Your btrfs-progs is somewhat old, which doesn't integrate dump super into inspect-internal. In that case, you can use btrfs-show-super -af instead. Thanks, Qu btrfs inspect-internal: unknown token 'dump-super' usage: btrfs inspect-internal <command> <args> btrfs inspect-internal inode-resolve [-v] <inode> <path> Get file system paths for the given inode btrfs inspect-internal logical-resolve [-Pv] [-s bufsize] <logical> <path> Get file system paths for the given logical address btrfs inspect-internal subvolid-resolve <subvolid> <path> Get file system paths for the given subvolume ID. btrfs inspect-internal rootid <path> Get tree ID of the containing subvolume of path. btrfs inspect-internal min-dev-size [options] <path> Get the minimum size the device can be shrunk to. The query various internal information On 13 February 2017 at 14:58, Tomasz Kusmierz <tom.kusmi...@gmail.com> wrote: Problem is to send a larger log into this mailing list :/ Anyway: uname -a Linux tevva-server 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10 20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux cut from messages (bear in mind that this is a single cut with a bit cut from inside of it to fit it in the email) Feb 10 00:17:14 server journal: ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== Feb 10 00:17:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:17:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:18:00 server kernel: BTRFS info (device sdc): found 22 extents Feb 10 00:18:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:17:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:18:05 server kernel: BTRFS info (device sdc): found 22 extents Feb 10 00:18:06 server kernel: BTRFS info (device sdc): relocating block group 12353563131904 flags 65 Feb 10 00:18:06 server journal: Feb 10 00:18:06 server journal: ==> /var/log/gitlab/sidekiq/current <== Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99341 2017-02-10T00:18:06.993Z 382 TID-otrr6ws48 PruneOldEventsWorker JID-99d3a4fb69be748c8674b5e1 INFO: start Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99571 2017-02-10T00:18:06.995Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:18:07 server journal: 2017-02-10_00:18:07.00454 2017-02-10T00:18:07.004Z 382 TID-otrr6ws48 PruneOldEventsWorker JID-99d3a4fb69be748c8674b5e1 INFO: done: 0.011 sec Feb 10 00:18:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:18:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:18:43 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:18:48 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:18:49 server kernel: BTRFS info (device sdc): relocating block group 12349268164608 flags 65 Feb 10 00:19:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:19:00 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51409 2017-02-10T00:19:02.513Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51449 2017-02-10T00:19:02.514Z 382 TID-otrspth10 PruneOldEventsWorker JID-4a162ace334771baf4befbb7 INFO: start Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.52994 2017-02-10T00:19:02.529Z 382 TID-otrspth10 PruneOldEventsWorker JID-4a162ace334771baf4befbb7 INFO: done: 0.015 sec Feb 10 00:19:26 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:19:31 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:19:31 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:19:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:19:32 server kernel: BTRFS info (device sdc): relocating block group 12344973197312 flags 65 Feb 10 00:19:51 server kernel: perf: interrupt took too long (2513 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Feb 10 00:20:00 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:19:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:20:10 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15695 2017-02-10T00:20:10.156Z 382 TID-otrsptg48 RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: start Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15968 2017-02-10T00:20:10.159Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: repository_check_worker Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17180 2017-02-10T00:20:10.171Z 382 TID-otrsptilo PruneOldEventsWorker JID-4fa75dc5a3d36957d1034f56 INFO: start Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17430 2017-02-10T00:20:10.174Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.18948 2017-02-10T00:20:10.189Z 382 TID-otrsptilo PruneOldEventsWorker JID-4fa75dc5a3d36957d1034f56 INFO: done: 0.018 sec Feb 10 00:20:11 server journal: 2017-02-10_00:20:11.00073 2017-02-10T00:20:11.000Z 382 TID-otrsptg48 RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: done: 0.844 sec Feb 10 00:20:14 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:20:15 server kernel: BTRFS info (device sdc): relocating block group 12340678230016 flags 65 Feb 10 00:20:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:20:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:20:41 server kernel: systemd-tmpfile: 127 output lines suppressed due to ratelimiting Feb 10 00:20:49 server kernel: BTRFS info (device sdc): found 34 extents Feb 10 00:20:54 server kernel: BTRFS info (device sdc): found 34 extents Feb 10 00:20:54 server kernel: BTRFS info (device sdc): relocating block group 12336383262720 flags 65 Feb 10 00:21:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:20:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.95882 2017-02-10T00:21:04.958Z 382 TID-otrsptep4 PruneOldEventsWorker JID-4bfca965ac38fafae6d9f88e INFO: start Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.96159 2017-02-10T00:21:04.961Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:21:04 server journal: 2017-02-10_00:21:04.97151 2017-02-10T00:21:04.971Z 382 TID-otrsptep4 PruneOldEventsWorker JID-4bfca965ac38fafae6d9f88e INFO: done: 0.013 sec Feb 10 00:21:07 server ntpd[3349]: 0.0.0.0 c612 02 freq_set kernel -21.061 PPM Feb 10 00:21:07 server ntpd[3349]: 0.0.0.0 c615 05 clock_sync Feb 10 00:21:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:21:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:21:31 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:21:35 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:21:36 server kernel: BTRFS info (device sdc): relocating block group 12332088295424 flags 65 Feb 10 00:22:00 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:21:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:22:04 server journal: Feb 10 00:22:04 server journal: ==> /var/log/gitlab/redis/current <== Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.03365 378:M 10 Feb 00:22:04.033 * 10 changes in 300 seconds. Saving... Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.03514 378:M 10 Feb 00:22:04.034 * Background saving started by pid 1480 Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.48684 1480:C 10 Feb 00:22:04.486 * DB saved on disk Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.48789 1480:C 10 Feb 00:22:04.487 * RDB: 8 MB of memory used by copy-on-write Feb 10 00:22:04 server journal: 2017-02-10_00:22:04.53761 378:M 10 Feb 00:22:04.537 * Background saving terminated with success Feb 10 00:22:11 server journal: Feb 10 00:22:11 server journal: ==> /var/log/gitlab/sidekiq/current <== Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.58099 2017-02-10T00:22:11.580Z 382 TID-otrsptep4 PruneOldEventsWorker JID-2b322e11d397491e806fdf9a INFO: start Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.58323 2017-02-10T00:22:11.583Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:22:11 server journal: 2017-02-10_00:22:11.59561 2017-02-10T00:22:11.595Z 382 TID-otrsptep4 PruneOldEventsWorker JID-2b322e11d397491e806fdf9a INFO: done: 0.015 sec Feb 10 00:22:14 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:22:19 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:22:20 server kernel: BTRFS info (device sdc): relocating block group 12327793328128 flags 65 Feb 10 00:22:31 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:22:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:22:54 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:22:59 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:22:59 server kernel: BTRFS info (device sdc): relocating block group 12323498360832 flags 65 Feb 10 00:23:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:22:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.86133 2017-02-10T00:23:05.861Z 382 TID-otrsptep4 PruneOldEventsWorker JID-95691f21edd3bd28b116508f INFO: start Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.86497 2017-02-10T00:23:05.864Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:23:05 server journal: 2017-02-10_00:23:05.87282 2017-02-10T00:23:05.872Z 382 TID-otrsptep4 PruneOldEventsWorker JID-95691f21edd3bd28b116508f INFO: done: 0.011 sec Feb 10 00:23:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:23:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:23:34 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:23:39 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:23:40 server kernel: BTRFS info (device sdc): relocating block group 12319203393536 flags 65 Feb 10 00:24:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:23:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.25079 2017-02-10T00:24:09.250Z 382 TID-otrr6wsw0 PruneOldEventsWorker JID-9f8cf009d88fd3ba6b5c41c8 INFO: start Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.25230 2017-02-10T00:24:09.252Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:24:09 server journal: 2017-02-10_00:24:09.26504 2017-02-10T00:24:09.264Z 382 TID-otrr6wsw0 PruneOldEventsWorker JID-9f8cf009d88fd3ba6b5c41c8 INFO: done: 0.014 sec Feb 10 00:24:15 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:24:19 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:24:20 server kernel: BTRFS info (device sdc): relocating block group 12314908426240 flags 65 Feb 10 00:24:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:24:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:24:54 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:24:59 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:24:59 server kernel: BTRFS info (device sdc): relocating block group 12310613458944 flags 65 Feb 10 00:25:00 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:24:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.43533 2017-02-10T00:25:13.435Z 382 TID-otrsptep4 PruneOldEventsWorker JID-0bafb305f0549fbfaedd0f21 INFO: start Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.43790 2017-02-10T00:25:13.437Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:25:13 server journal: 2017-02-10_00:25:13.44842 2017-02-10T00:25:13.448Z 382 TID-otrsptep4 PruneOldEventsWorker JID-0bafb305f0549fbfaedd0f21 INFO: done: 0.013 sec Feb 10 00:25:34 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:25:32 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:25:34 server kernel: BTRFS info (device sdc): found 37 extents Feb 10 00:25:39 server kernel: BTRFS info (device sdc): found 37 extents Feb 10 00:25:40 server kernel: BTRFS info (device sdc): relocating block group 12306318491648 flags 65 Feb 10 00:26:00 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:25:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:26:14 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.73291 2017-02-10T00:26:18.732Z 382 TID-otrr6wry4 PruneOldEventsWorker JID-444ab4e6f7b42090f23cfca2 INFO: start Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.73405 2017-02-10T00:26:18.734Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:26:18 server journal: 2017-02-10_00:26:18.74593 2017-02-10T00:26:18.745Z 382 TID-otrr6wry4 PruneOldEventsWorker JID-444ab4e6f7b42090f23cfca2 INFO: done: 0.013 sec Feb 10 00:26:18 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:26:19 server kernel: BTRFS info (device sdc): relocating block group 12302023524352 flags 65 Feb 10 00:26:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:26:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:26:54 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:26:58 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:26:59 server kernel: BTRFS info (device sdc): relocating block group 12297728557056 flags 65 Feb 10 00:27:05 server journal: Feb 10 00:27:05 server journal: ==> /var/log/gitlab/redis/current <== Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.05277 378:M 10 Feb 00:27:05.052 * 10 changes in 300 seconds. Saving... Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.05399 378:M 10 Feb 00:27:05.053 * Background saving started by pid 1796 Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.75682 1796:C 10 Feb 00:27:05.756 * DB saved on disk Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.75733 1796:C 10 Feb 00:27:05.757 * RDB: 8 MB of memory used by copy-on-write Feb 10 00:27:05 server journal: 2017-02-10_00:27:05.85460 378:M 10 Feb 00:27:05.854 * Background saving terminated with success Feb 10 00:27:08 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:27:06 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:27:11 server journal: Feb 10 00:27:11 server journal: ==> /var/log/gitlab/sidekiq/current <== Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.01598 2017-02-10T00:27:11.015Z 382 TID-otrsptskk PruneOldEventsWorker JID-9c0a9bd8da1b04e7fec41b09 INFO: start Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.01854 2017-02-10T00:27:11.018Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:27:11 server journal: 2017-02-10_00:27:11.02791 2017-02-10T00:27:11.027Z 382 TID-otrsptskk PruneOldEventsWorker JID-9c0a9bd8da1b04e7fec41b09 INFO: done: 0.012 sec Feb 10 00:27:30 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:27:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:27:35 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:27:40 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:27:40 server kernel: BTRFS info (device sdc): relocating block group 12293433589760 flags 65 Feb 10 00:28:01 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:27:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.23927 2017-02-10T00:28:02.239Z 382 TID-otrsptep4 PruneOldEventsWorker JID-59ab152a8efd6e7386e82f5a INFO: start Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.24045 2017-02-10T00:28:02.240Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:28:02 server journal: 2017-02-10_00:28:02.25251 2017-02-10T00:28:02.252Z 382 TID-otrsptep4 PruneOldEventsWorker JID-59ab152a8efd6e7386e82f5a INFO: done: 0.013 sec Feb 10 00:28:16 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:28:21 server kernel: BTRFS info (device sdc): found 32 extents Feb 10 00:28:21 server kernel: BTRFS info (device sdc): relocating block group 12289138622464 flags 65 Feb 10 00:28:32 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:28:29 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:28:58 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:29:00 server journal: 192.168.1.253 - wally_tm [10/Feb/2017:00:28:59 +0000] "PROPFIND /remote.php/webdav/Pictures HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1" Feb 10 00:29:02 server kernel: BTRFS info (device sdc): found 33 extents Feb 10 00:29:03 server kernel: BTRFS info (device sdc): relocating block group 12288064880640 flags 68 Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.54068 2017-02-10T00:29:10.540Z 382 TID-otrsptsws PruneOldEventsWorker JID-3da257e9bdbdb05b1966f9b6 INFO: start Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.54251 2017-02-10T00:29:10.542Z 382 TID-otrr6wqok INFO: Cron Jobs - add job with name: prune_old_events_worker Feb 10 00:29:10 server journal: 2017-02-10_00:29:10.55457 2017-02-10T00:29:10.554Z 382 TID-otrsptsws PruneOldEventsWorker JID-3da257e9bdbdb05b1966f9b6 INFO: done: 0.014 sec Feb 10 00:29:11 server kernel: ------------[ cut here ]------------ Feb 10 00:29:11 server kernel: WARNING: CPU: 7 PID: 27932 at fs/btrfs/extent-tree.c:6945 __btrfs_free_extent.isra.69+0xb62/0xca0 [btrfs] Feb 10 00:29:11 server kernel: Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat nf_conntrack ext4 jbd2 mbcache ipmi_devintf coretemp iTCO_wdt iTCO_vendor_support gpio_ich kvm_intel kvm irqbypass intel_cstate input_leds pcspkr hpwdt hpilo lpc_ich mfd_core i7core_edac edac_core ses ioatdma enclosure sg ipmi_si ipmi_msghandler acpi_power_meter shpchp pcc_cpufreq acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables btrfs xor raid6_pq sd_mod crc32c_intel amdkfd amd_iommu_v2 serio_raw radeon drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci fjes libata mpt3sas raid_class scsi_transport_sas igb ptp pps_core dca i2c_algo_bit Feb 10 00:29:11 server kernel: CPU: 7 PID: 27932 Comm: btrfs Tainted: G I 4.8.7-1.el7.elrepo.x86_64 #1 Feb 10 00:29:11 server kernel: Hardware name: HP ProLiant SE326M1 , BIOS R02 12/07/2010 Feb 10 00:29:11 server kernel: 0000000000000286 00000000e89bd9a3 ffff8805c9b67720 ffffffff8135406c Feb 10 00:29:11 server kernel: 0000000000000000 0000000000000000 ffff8805c9b67760 ffffffff810817b1 Feb 10 00:29:11 server kernel: 00001b21fffffffe ffff8804d6f47850 00000b2d1dfc4000 00000000fffffffe Feb 10 00:29:11 server kernel: Call Trace: Feb 10 00:29:11 server kernel: [<ffffffff8135406c>] dump_stack+0x63/0x87 Feb 10 00:29:11 server kernel: [<ffffffff810817b1>] __warn+0xd1/0xf0 Feb 10 00:29:11 server kernel: [<ffffffff810818ed>] warn_slowpath_null+0x1d/0x20 Feb 10 00:29:11 server kernel: [<ffffffffa0467722>] __btrfs_free_extent.isra.69+0xb62/0xca0 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa046bd61>] __btrfs_run_delayed_refs.constprop.78+0xa11/0x1250 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa046f46e>] btrfs_run_delayed_refs+0x8e/0x2c0 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04abddb>] ? free_extent_buffer+0x4b/0x90 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa0485079>] btrfs_should_end_transaction+0x49/0x60 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa046db87>] btrfs_drop_snapshot+0x3f7/0x7f0 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04e0a0a>] merge_reloc_roots+0x10a/0x260 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04e0f34>] relocate_block_group+0x3d4/0x670 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04e137b>] btrfs_relocate_block_group+0x1ab/0x290 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04b1df6>] btrfs_relocate_chunk.isra.40+0x46/0xd0 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04b369e>] __btrfs_balance+0x85e/0xc50 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffff810ac538>] ? try_to_wake_up+0x58/0x3c0 Feb 10 00:29:11 server kernel: [<ffffffffa04b3d4c>] btrfs_balance+0x2bc/0x630 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04bfa81>] btrfs_ioctl_balance+0x3f1/0x540 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffffa04c3c38>] btrfs_ioctl+0xab8/0x21e0 [btrfs] Feb 10 00:29:11 server kernel: [<ffffffff8120d8d5>] ? mem_cgroup_commit_charge+0x85/0x100 Feb 10 00:29:11 server kernel: [<ffffffff811d17d9>] ? page_add_new_anon_rmap+0x89/0xc0 Feb 10 00:29:11 server kernel: [<ffffffff8119f5d5>] ? lru_cache_add_active_or_unevictable+0x35/0xb0 Feb 10 00:29:11 server kernel: [<ffffffff811c6c60>] ? handle_mm_fault+0xed0/0x1240 Feb 10 00:29:11 server kernel: [<ffffffff811f4407>] ? kmem_cache_alloc+0xd7/0x1a0 Feb 10 00:29:11 server kernel: [<ffffffff8122ffc7>] do_vfs_ioctl+0xa7/0x5f0 Feb 10 00:29:11 server kernel: [<ffffffff8112d3ff>] ? __audit_syscall_entry+0xaf/0x100 Feb 10 00:29:11 server kernel: [<ffffffff8100365d>] ? syscall_trace_enter+0x1dd/0x2c0 Feb 10 00:29:11 server kernel: [<ffffffff81230589>] SyS_ioctl+0x79/0x90 Feb 10 00:29:11 server kernel: [<ffffffff81003a47>] do_syscall_64+0x67/0x160 Feb 10 00:29:11 server kernel: [<ffffffff8173bbe1>] entry_SYSCALL64_slow_path+0x25/0x25 Feb 10 00:29:11 server kernel: ---[ end trace 9b04f696ec89affd ]--- Feb 10 00:29:11 server kernel: BTRFS info (device sdc): leaf 5224642887680 total ptrs 280 free space 43 Feb 10 00:29:11 server kernel: #011item 0 key (12288253624320 169 0) itemoff 16250 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 1 key (12288253640704 169 0) itemoff 16217 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 2 key (12288253804544 169 0) itemoff 16184 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 3 key (12288253919232 169 0) itemoff 16151 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 4 key (12288253935616 169 0) itemoff 16118 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 5 key (12288253952000 169 0) itemoff 16085 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 6 key (12288254050304 169 0) itemoff 16052 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 7 key (12288254115840 169 0) itemoff 16019 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 8 key (12288254361600 169 0) itemoff 15986 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 9 key (12288254640128 169 0) itemoff 15953 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 10 key (12288255098880 169 0) itemoff 15920 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142939 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 11 key (12288258097152 169 0) itemoff 15887 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 12 key (12288258162688 169 0) itemoff 15854 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 13 key (12288258179072 169 0) itemoff 15821 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258 Feb 10 00:29:11 server kernel: #011#011shared block backref parent 5224641380352 Feb 10 00:29:11 server kernel: #011item 14 key (12288258375680 169 0) itemoff 15788 itemsize 33 Feb 10 00:29:11 server kernel: #011#011extent refs 1 gen 142940 flags 258 On 13 Feb 2017, at 00:49, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: At 02/12/2017 09:17 AM, Tomasz Kusmierz wrote: Hi all, So my main storage filesystem got some sort of veird corruption (that I can gather). Everything seems to work OK, but when I try to create a snapshot or run balance (no filters) it will get remounted read only. Kernel version please. Fun part is that balance seems to be running even on read only FS, and I continuously get kernel traces in /var/log/messages .... so it might as well in the back ground silently eat my data away :/ Kernel backtrace please. It would be better if you could paste the *first* kernel backtrace, as that could be the cause, and following kernel backtrace is just warning from btrfs_abort_transaction() without meaningful output. I just see some normal messages, but no kernel backtrace. UPDATE: Yeah, after rebooting the system it does not even mount the FS, mount.btrfs sits in some sort of spinlock and consumes 100% of singe core. UPDATE 2: System is completelly cooked :/ [root@server ~]# btrfs fi show Label: 'rockstor_server' uuid: 5581a647-40ef-4a7a-9d73-847bf35a142b Total devices 1 FS bytes used 5.72GiB devid 1 size 53.17GiB used 7.03GiB path /dev/sda2 Label: 'broken_pool' uuid: 26095277-a234-455b-8c97-8dac8ad934c8 Total devices 2 FS bytes used 193.52GiB devid 1 size 1.82TiB used 196.03GiB path /dev/sdb devid 2 size 1.82TiB used 196.03GiB path /dev/sdi Label: 'main_pool' uuid: 0576d577-8954-4a60-a02b-9492b3c29318 Total devices 8 FS bytes used 5.83TiB devid 1 size 1.82TiB used 1.50TiB path /dev/sdc devid 2 size 1.82TiB used 1.50TiB path /dev/sdd devid 3 size 1.82TiB used 1.50TiB path /dev/sde devid 4 size 1.82TiB used 1.50TiB path /dev/sdf devid 5 size 1.82TiB used 1.50TiB path /dev/sdg devid 6 size 1.82TiB used 1.50TiB path /dev/sdh devid 7 size 1.82TiB used 1.50TiB path /dev/sdj devid 8 size 1.82TiB used 1.50TiB path /dev/sdk [root@server ~]# mount /dev/sdc /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. [root@server ~]# mount /dev/sdd /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sdd, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. [root@server ~]# mount /dev/sde /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. dmesg tail retuns: [ 9507.835629] systemd-udevd[1873]: Validate module index [ 9507.835656] systemd-udevd[1873]: Check if link configuration needs reloading. [ 9507.835690] systemd-udevd[1873]: seq 3698 queued, 'add' 'bdi' [ 9507.835873] systemd-udevd[1873]: seq 3698 forked new worker [13858] [ 9507.836202] BTRFS info (device sdd): disk space caching is enabled [ 9507.836204] BTRFS info (device sdd): has skinny extents [ 9507.836322] systemd-udevd[13858]: seq 3698 running [ 9507.836443] systemd-udevd[13858]: no db file to read /run/udev/data/+bdi:btrfs-4: No such file or directory [ 9507.836474] systemd-udevd[13858]: RUN '/bin/mknod /dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1 [ 9507.837366] systemd-udevd[13861]: starting '/bin/mknod /dev/btrfs-control c 10 234' [ 9507.837833] BTRFS error (device sdd): failed to read the system array: -5 [ 9507.838231] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c 10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists' [ 9507.838262] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c 10 234' [13861] exit with return code 1 [ 9507.854757] BTRFS: open_ctree failed [ 9511.370878] BTRFS info (device sdd): disk space caching is enabled [ 9511.370881] BTRFS info (device sdd): has skinny extents [ 9511.375097] BTRFS error (device sdd): failed to read the system array: -5 Btrfs failed to read system chunk array from super block. Normally this means your primary super block is cooked up. There may still be chance to recover your fs using backup superblocks. Please paste the output of "btrfs inspect-internal dump-super -af <your fs>" Thanks, Qu [ 9511.392792] BTRFS: open_ctree failed [ 9514.233627] BTRFS: device label main_pool devid 3 transid 150680 /dev/sde [ 9514.234399] systemd-udevd[1873]: Validate module index [ 9514.234431] systemd-udevd[1873]: Check if link configuration needs reloading. [ 9514.234465] systemd-udevd[1873]: seq 3702 queued, 'add' 'bdi' [ 9514.234522] systemd-udevd[1873]: passed 142 bytes to netlink monitor 0x5628f65d40d0 [ 9514.234554] systemd-udevd[13882]: seq 3702 running [ 9514.234780] systemd-udevd[13882]: no db file to read /run/udev/data/+bdi:btrfs-6: No such file or directory [ 9514.234790] BTRFS info (device sde): disk space caching is enabled [ 9514.234792] BTRFS info (device sde): has skinny extents [ 9514.234798] systemd-udevd[13882]: RUN '/bin/mknod /dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1 [ 9514.235181] systemd-udevd[13906]: starting '/bin/mknod /dev/btrfs-control c 10 234' [ 9514.236448] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c 10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists' [ 9514.236514] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c 10 234' [13906] exit with return code 1 [ 9514.238726] BTRFS error (device sde): failed to read the system array: -5 [ 9514.255472] BTRFS: open_ctree failed -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html