Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

when looking through log for old messages I can see that there are
kernel problems before with extent tree while trying to create
snapshots:


Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: #011item 108 key (12288467451904 169 0)
itemoff 12677 itemsize 33
Jan 23 05:00:02 server kernel: #011#011extent refs 1 gen 144462 flags 2
Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: #011item 109 key (12288467468288 169 0)
itemoff 12644 itemsize 33
Jan 23 05:00:02 server kernel: #011#011extent refs 1 gen 144462 flags 2
Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: #011item 110 key (12288467484672 169 0)
itemoff 12611 itemsize 33
Jan 23 05:00:02 server kernel: #011#011extent refs 1 gen 144462 flags 2
Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: #011item 111 key (12288467501056 169 0)
itemoff 12578 itemsize 33
Jan 23 05:00:02 server kernel: #011#011extent refs 1 gen 144462 flags 2
Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: #011item 112 key (12288467533824 169 0)
itemoff 12545 itemsize 33
Jan 23 05:00:02 server kernel: #011#011extent refs 1 gen 144462 flags 2
Jan 23 05:00:02 server kernel: #011#011tree block backref root 7
Jan 23 05:00:02 server kernel: BTRFS error (device sdc): unable to
find ref byte nr 12288404504576 parent 0 root 258  owner 2 offset 0
Jan 23 05:00:02 server kernel: [ cut here ]
Jan 23 05:00:02 server kernel: WARNING: CPU: 8 PID: 28064 at
fs/btrfs/extent-tree.c:6951 __btrfs_free_extent.isra.69+0xbca/0xca0
[btrfs]
Jan 23 05:00:02 server kernel: Modules linked in: xt_nat veth
xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype
iptable_filter nf_nat nf_conntrack ipmi_devintf ext4 jbd2 mbcache
iTCO_wdt gpio_ich iTCO_vendor_support coretemp kvm_intel kvm irqbypass
intel_cstate input_leds pcspkr hpilo hpwdt lpc_ich mfd_core ioatdma
i7core_edac edac_core ses enclosure ipmi_si ipmi_msghandler sg
acpi_power_meter pcc_cpufreq shpchp acpi_cpufreq nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables btrfs xor raid6_pq sd_mod amdkfd
amd_iommu_v2 radeon crc32c_intel drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ahci libahci
libata fjes mpt3sas raid_class scsi_transport_sas igb ptp pps_core dca
i2c_algo_bit
Jan 23 05:00:02 server kernel: CPU: 8 PID: 28064 Comm: btrfs Tainted:
GW I 4.8.7-1.el7.elrepo.x86_64 #1
Jan 23 05:00:02 server kernel: Hardware name: HP ProLiant SE326M1   ,
BIOS R02 12/07/2010
Jan 23 05:00:02 server kernel: 0286 77bb5259
8802bbf1f778 8135406c
Jan 23 05:00:02 server kernel: 8802bbf1f7c8 
8802bbf1f7b8 810817b1
Jan 23 05:00:02 server kernel: 1b270002 8806f612
0b2d1dfc4000 fffe
Jan 23 05:00:02 server kernel: Call Trace:
Jan 23 05:00:02 server kernel: [] dump_stack+0x63/0x87
Jan 23 05:00:02 server kernel: [] __warn+0xd1/0xf0
Jan 23 05:00:02 server kernel: [] warn_slowpath_fmt+0x5f/0x80
Jan 23 05:00:02 server kernel: []
__btrfs_free_extent.isra.69+0xbca/0xca0 [btrfs]
Jan 23 05:00:02 server kernel: []
__btrfs_run_delayed_refs.constprop.78+0xa11/0x1250 [btrfs]
Jan 23 05:00:02 server kernel: []
btrfs_run_delayed_refs+0x8e/0x2c0 [btrfs]
Jan 23 05:00:02 server kernel: []
create_pending_snapshot.isra.26+0x5cd/0xdd0 [btrfs]
Jan 23 05:00:02 server kernel: []
create_pending_snapshots+0x78/0xa0 [btrfs]
Jan 23 05:00:02 server kernel: []
btrfs_commit_transaction+0x435/0xa70 [btrfs]
Jan 23 05:00:02 server kernel: []
btrfs_mksubvol.isra.39+0x513/0x520 [btrfs]
Jan 23 05:00:02 server kernel: [] ?
prepare_to_wait_event+0xf0/0xf0
Jan 23 05:00:02 server kernel: []
btrfs_ioctl_snap_create_transid+0x18f/0x1a0 [btrfs]
Jan 23 05:00:02 server kernel: []
btrfs_ioctl_snap_create_v2+0x125/0x180 [btrfs]
Jan 23 05:00:02 server kernel: []
btrfs_ioctl+0x6b3/0x21e0 [btrfs]
Jan 23 05:00:02 server kernel: [] ?
mem_cgroup_commit_charge+0x85/0x100
Jan 23 05:00:02 server kernel: [] ?
page_add_new_anon_rmap+0x89/0xc0
Jan 23 05:00:02 server kernel: [] ?
lru_cache_add_active_or_unevictable+0x35/0xb0
Jan 23 05:00:02 server kernel: [] ?
handle_mm_fault+0xed0/0x1240
Jan 23 05:00:02 server kernel: [] do_vfs_ioctl+0xa7/0x5f0
Jan 23 05:00:02 server kernel: [] ?
__audit_syscall_entry+0xaf/0x100
Jan 23 05:00:02 server kernel: [] ?
syscall_trace_enter+0x1dd/0x2c0
Jan 23 05:00:02 server kernel: [] SyS_ioctl+0x79/0x90
Jan 23 05:00:02 server kernel: [] do_syscall_64+0x67/0x160
Jan 23 05:00:02 server kernel: []
entry_SYSCALL64_slow_path+0x25/0x25
Jan 23 05:00:02 server kernel: ---[ end trace eb863872ca3491b1 ]---
Jan 23 05:00:02 server kernel: BTRFS: error (device sdc) in
__btrfs_free_extent:6951: errno=-2 No such entry
Jan 23 05:00:02 server kernel: BTRFS info (device sdc): 

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

Anyone ?

On 18 Feb 2017, at 16:44, Tomasz Kusmierz  wrote:

So Qu,

currently my situation is that:
I've tried to go btrfs scan --repair, and it did relair some stuff is
qgroup's ... then tried to mont it and, surprise surpeire system
locked out in 20 seconds.

Reboot, again scan --repair = a lot of missing back pointers were
repaired and system is supposedly "OK"  attempted to mount it and
within 20 seconds system locked out so hard it wold no even reboot
from acpi.

installed "ellrepo kernel-lm" and installed 4.9.10

another scan --repair = same problem with lot's of back pointer
missing, fixed  system again seems "OK" ... another attempt to
mount /dev/sdc /mnt2/main_pool and again after 20 seconds system locks
up hard.

There is nothing in messages, nothing in dmesg ... I think that system
lock up so hard that master btrfs filesystem does not get time those
logs pushed to disk.






On 16 February 2017 at 23:46, Tomasz Kusmierz  wrote:

Thanks Qu,

Just before I’ll go and accidentally mess up this FS more - I’ve
mentioned originally that this problem started with FS not being able
to create a snapshot ( it would get remounted RO automatically ) for
about a month, and when I’ve realised that there is a problem like
that I’ve attempted a full FS balance that caused this FS to be
unmountable. Is there any other debug you would require before I
proceed (I’ve got a lot i

On 16 Feb 2017, at 01:26, Qu Wenruo  wrote:



At 02/15/2017 10:11 PM, Tomasz Kusmierz wrote:

So guys, any help here ? I’m kinda stuck now with system just idling
and doing nothing while I wait for some feedback ...


Sorry for the late reply.

Busying debugging a kernel bug.

On 14 Feb 2017, at 19:38, Tomasz Kusmierz  wrote:

[root@server ~]#  btrfs-show-super -af /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
-
csum_type   0 (crc32c)
csum_size   4
csum0x17d56ce0 [match]


This superblock is good.

bytenr  65536
flags   0x1
 ( WRITTEN )
magic   _BHRfS_M [match]
fsid0576d577-8954-4a60-a02b-9492b3c29318
label   main_pool
generation  150682
root5223857717248
sys_array_size  321
chunk_root_generation   150678
root_level  1
chunk_root  8669488005120
chunk_root_level1
log_root0
log_root_transid0
log_root_level  0
total_bytes 16003191472128
bytes_used  6411278503936
sectorsize  4096
nodesize16384
leafsize16384
stripesize  4096
root_dir6
num_devices 8
compat_flags0x0
compat_ro_flags 0x0
incompat_flags  0x161
 ( MIXED_BACKREF |
   BIG_METADATA |
   EXTENDED_IREF |
   SKINNY_METADATA )
cache_generation150682
uuid_tree_generation150679
dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
dev_item.type   0
dev_item.total_bytes2000398934016
dev_item.bytes_used 1647136735232
dev_item.io_align   4096
dev_item.io_width   4096
dev_item.sector_size4096
dev_item.devid  1
dev_item.dev_group  0
dev_item.seek_speed 0
dev_item.bandwidth  0
dev_item.generation 0
sys_chunk_array[2048]:
 item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
 length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
 io_align 65536 io_width 65536 sector_size 4096
 num_stripes 8 sub_stripes 2
 stripe 0 devid 7 offset 1083674984448
 dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
 stripe 1 devid 8 offset 1083674984448
 dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
 stripe 2 devid 1 offset 1365901312
 dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
 stripe 3 devid 3 offset 1345978368
 dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
 stripe 4 devid 4 offset 1345978368
 dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
 stripe 5 devid 5 offset 1345978368
 dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
 stripe 6 devid 6 offset 1345978368
 dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
 stripe 7 devid 2 offset 1345978368
 dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841


And I didn't see anything wrong in sys_chunk_array.


Would you please try to mount the fs with latest kernel?

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

So Qu,

currently my situation is that:
I've tried to go btrfs scan --repair, and it did relair some stuff is
qgroup's ... then tried to mont it and, surprise surpeire system
locked out in 20 seconds.

Reboot, again scan --repair = a lot of missing back pointers were
repaired and system is supposedly "OK"  attempted to mount it and
within 20 seconds system locked out so hard it wold no even reboot
from acpi.

installed "ellrepo kernel-lm" and installed 4.9.10

another scan --repair = same problem with lot's of back pointer
missing, fixed  system again seems "OK" ... another attempt to
mount /dev/sdc /mnt2/main_pool and again after 20 seconds system locks
up hard.

There is nothing in messages, nothing in dmesg ... I think that system
lock up so hard that master btrfs filesystem does not get time those
logs pushed to disk.






On 16 February 2017 at 23:46, Tomasz Kusmierz  wrote:
> Thanks Qu,
>
> Just before I’ll go and accidentally mess up this FS more - I’ve
> mentioned originally that this problem started with FS not being able
> to create a snapshot ( it would get remounted RO automatically ) for
> about a month, and when I’ve realised that there is a problem like
> that I’ve attempted a full FS balance that caused this FS to be
> unmountable. Is there any other debug you would require before I
> proceed (I’ve got a lot i
>
> On 16 Feb 2017, at 01:26, Qu Wenruo  wrote:
>
>
>
> At 02/15/2017 10:11 PM, Tomasz Kusmierz wrote:
>
> So guys, any help here ? I’m kinda stuck now with system just idling
> and doing nothing while I wait for some feedback ...
>
>
> Sorry for the late reply.
>
> Busying debugging a kernel bug.
>
> On 14 Feb 2017, at 19:38, Tomasz Kusmierz  wrote:
>
> [root@server ~]#  btrfs-show-super -af /dev/sdc
> superblock: bytenr=65536, device=/dev/sdc
> -
> csum_type   0 (crc32c)
> csum_size   4
> csum0x17d56ce0 [match]
>
>
> This superblock is good.
>
> bytenr  65536
> flags   0x1
>   ( WRITTEN )
> magic   _BHRfS_M [match]
> fsid0576d577-8954-4a60-a02b-9492b3c29318
> label   main_pool
> generation  150682
> root5223857717248
> sys_array_size  321
> chunk_root_generation   150678
> root_level  1
> chunk_root  8669488005120
> chunk_root_level1
> log_root0
> log_root_transid0
> log_root_level  0
> total_bytes 16003191472128
> bytes_used  6411278503936
> sectorsize  4096
> nodesize16384
> leafsize16384
> stripesize  4096
> root_dir6
> num_devices 8
> compat_flags0x0
> compat_ro_flags 0x0
> incompat_flags  0x161
>   ( MIXED_BACKREF |
> BIG_METADATA |
> EXTENDED_IREF |
> SKINNY_METADATA )
> cache_generation150682
> uuid_tree_generation150679
> dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
> dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
> dev_item.type   0
> dev_item.total_bytes2000398934016
> dev_item.bytes_used 1647136735232
> dev_item.io_align   4096
> dev_item.io_width   4096
> dev_item.sector_size4096
> dev_item.devid  1
> dev_item.dev_group  0
> dev_item.seek_speed 0
> dev_item.bandwidth  0
> dev_item.generation 0
> sys_chunk_array[2048]:
>   item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
>   length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
>   io_align 65536 io_width 65536 sector_size 4096
>   num_stripes 8 sub_stripes 2
>   stripe 0 devid 7 offset 1083674984448
>   dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
>   stripe 1 devid 8 offset 1083674984448
>   dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
>   stripe 2 devid 1 offset 1365901312
>   dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
>   stripe 3 devid 3 offset 1345978368
>   dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
>   stripe 4 devid 4 offset 1345978368
>   dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
>   stripe 5 devid 5 offset 1345978368
>   dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
>   stripe 6 devid 6 offset 1345978368
>   dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
>   stripe 7 devid 2 offset 1345978368
>   dev_uuid 

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

Thanks Qu,

Just before I’ll go and accidentally mess up this FS more - I’ve
mentioned originally that this problem started with FS not being able
to create a snapshot ( it would get remounted RO automatically ) for
about a month, and when I’ve realised that there is a problem like
that I’ve attempted a full FS balance that caused this FS to be
unmountable. Is there any other debug you would require before I
proceed (I’ve got a lot i

On 16 Feb 2017, at 01:26, Qu Wenruo  wrote:



At 02/15/2017 10:11 PM, Tomasz Kusmierz wrote:

So guys, any help here ? I’m kinda stuck now with system just idling
and doing nothing while I wait for some feedback ...


Sorry for the late reply.

Busying debugging a kernel bug.

On 14 Feb 2017, at 19:38, Tomasz Kusmierz  wrote:

[root@server ~]#  btrfs-show-super -af /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
-
csum_type   0 (crc32c)
csum_size   4
csum0x17d56ce0 [match]


This superblock is good.

bytenr  65536
flags   0x1
  ( WRITTEN )
magic   _BHRfS_M [match]
fsid0576d577-8954-4a60-a02b-9492b3c29318
label   main_pool
generation  150682
root5223857717248
sys_array_size  321
chunk_root_generation   150678
root_level  1
chunk_root  8669488005120
chunk_root_level1
log_root0
log_root_transid0
log_root_level  0
total_bytes 16003191472128
bytes_used  6411278503936
sectorsize  4096
nodesize16384
leafsize16384
stripesize  4096
root_dir6
num_devices 8
compat_flags0x0
compat_ro_flags 0x0
incompat_flags  0x161
  ( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation150682
uuid_tree_generation150679
dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
dev_item.type   0
dev_item.total_bytes2000398934016
dev_item.bytes_used 1647136735232
dev_item.io_align   4096
dev_item.io_width   4096
dev_item.sector_size4096
dev_item.devid  1
dev_item.dev_group  0
dev_item.seek_speed 0
dev_item.bandwidth  0
dev_item.generation 0
sys_chunk_array[2048]:
  item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
  length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
  io_align 65536 io_width 65536 sector_size 4096
  num_stripes 8 sub_stripes 2
  stripe 0 devid 7 offset 1083674984448
  dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
  stripe 1 devid 8 offset 1083674984448
  dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
  stripe 2 devid 1 offset 1365901312
  dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
  stripe 3 devid 3 offset 1345978368
  dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
  stripe 4 devid 4 offset 1345978368
  dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
  stripe 5 devid 5 offset 1345978368
  dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
  stripe 6 devid 6 offset 1345978368
  dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
  stripe 7 devid 2 offset 1345978368
  dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841


And I didn't see anything wrong in sys_chunk_array.


Would you please try to mount the fs with latest kernel?
Better later than v4.9, as in that version extra kernel messages are
introduced to give more details about what's going wrong.

Thanks,
Qu

backup_roots[4]:
  backup 0:
  backup_tree_root:   5223857717248   gen: 150680 level: 1
  backup_chunk_root:  8669488005120   gen: 150678 level: 1
  backup_extent_root: 5223867383808   gen: 150680 level: 2
  backup_fs_root: 0   gen: 0  level: 0
  backup_dev_root:5224791523328   gen: 150680 level: 1
  backup_csum_root:   5224802140160   gen: 150680 level: 3
  backup_total_bytes: 16003191472128
  backup_bytes_used:  6411278503936
  backup_num_devices: 8

  backup 1:
  backup_tree_root:   5224155807744   gen: 150681 level: 1
  backup_chunk_root:  8669488005120   gen: 150678 level: 1
  backup_extent_root: 

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Qu Wenruo]

At 02/15/2017 10:11 PM, Tomasz Kusmierz wrote:

So guys, any help here ? I’m kinda stuck now with system just idling and doing 
nothing while I wait for some feedback ...


Sorry for the late reply.

Busying debugging a kernel bug.


On 14 Feb 2017, at 19:38, Tomasz Kusmierz  wrote:

[root@server ~]#  btrfs-show-super -af /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
-
csum_type   0 (crc32c)
csum_size   4
csum0x17d56ce0 [match]


This superblock is good.


bytenr  65536
flags   0x1
   ( WRITTEN )
magic   _BHRfS_M [match]
fsid0576d577-8954-4a60-a02b-9492b3c29318
label   main_pool
generation  150682
root5223857717248
sys_array_size  321
chunk_root_generation   150678
root_level  1
chunk_root  8669488005120
chunk_root_level1
log_root0
log_root_transid0
log_root_level  0
total_bytes 16003191472128
bytes_used  6411278503936
sectorsize  4096
nodesize16384
leafsize16384
stripesize  4096
root_dir6
num_devices 8
compat_flags0x0
compat_ro_flags 0x0
incompat_flags  0x161
   ( MIXED_BACKREF |
 BIG_METADATA |
 EXTENDED_IREF |
 SKINNY_METADATA )
cache_generation150682
uuid_tree_generation150679
dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
dev_item.type   0
dev_item.total_bytes2000398934016
dev_item.bytes_used 1647136735232
dev_item.io_align   4096
dev_item.io_width   4096
dev_item.sector_size4096
dev_item.devid  1
dev_item.dev_group  0
dev_item.seek_speed 0
dev_item.bandwidth  0
dev_item.generation 0
sys_chunk_array[2048]:
   item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
   length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
   io_align 65536 io_width 65536 sector_size 4096
   num_stripes 8 sub_stripes 2
   stripe 0 devid 7 offset 1083674984448
   dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
   stripe 1 devid 8 offset 1083674984448
   dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
   stripe 2 devid 1 offset 1365901312
   dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
   stripe 3 devid 3 offset 1345978368
   dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
   stripe 4 devid 4 offset 1345978368
   dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
   stripe 5 devid 5 offset 1345978368
   dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
   stripe 6 devid 6 offset 1345978368
   dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
   stripe 7 devid 2 offset 1345978368
   dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841


And I didn't see anything wrong in sys_chunk_array.


Would you please try to mount the fs with latest kernel?
Better later than v4.9, as in that version extra kernel messages are 
introduced to give more details about what's going wrong.


Thanks,
Qu


backup_roots[4]:
   backup 0:
   backup_tree_root:   5223857717248   gen: 150680 level: 1
   backup_chunk_root:  8669488005120   gen: 150678 level: 1
   backup_extent_root: 5223867383808   gen: 150680 level: 2
   backup_fs_root: 0   gen: 0  level: 0
   backup_dev_root:5224791523328   gen: 150680 level: 1
   backup_csum_root:   5224802140160   gen: 150680 level: 3
   backup_total_bytes: 16003191472128
   backup_bytes_used:  6411278503936
   backup_num_devices: 8

   backup 1:
   backup_tree_root:   5224155807744   gen: 150681 level: 1
   backup_chunk_root:  8669488005120   gen: 150678 level: 1
   backup_extent_root: 5224156233728   gen: 150681 level: 2
   backup_fs_root: 0   gen: 0  level: 0
   backup_dev_root:5224633155584   gen: 150681 level: 1
   backup_csum_root:   5224634941440   gen: 150681 level: 3
   backup_total_bytes: 16003191472128
   backup_bytes_used:  6411278503936
   backup_num_devices: 8

   backup 2:
   backup_tree_root:   

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

So guys, any help here ? I’m kinda stuck now with system just idling and doing 
nothing while I wait for some feedback ...
> On 14 Feb 2017, at 19:38, Tomasz Kusmierz  wrote:
> 
> [root@server ~]#  btrfs-show-super -af /dev/sdc
> superblock: bytenr=65536, device=/dev/sdc
> -
> csum_type   0 (crc32c)
> csum_size   4
> csum0x17d56ce0 [match]
> bytenr  65536
> flags   0x1
>( WRITTEN )
> magic   _BHRfS_M [match]
> fsid0576d577-8954-4a60-a02b-9492b3c29318
> label   main_pool
> generation  150682
> root5223857717248
> sys_array_size  321
> chunk_root_generation   150678
> root_level  1
> chunk_root  8669488005120
> chunk_root_level1
> log_root0
> log_root_transid0
> log_root_level  0
> total_bytes 16003191472128
> bytes_used  6411278503936
> sectorsize  4096
> nodesize16384
> leafsize16384
> stripesize  4096
> root_dir6
> num_devices 8
> compat_flags0x0
> compat_ro_flags 0x0
> incompat_flags  0x161
>( MIXED_BACKREF |
>  BIG_METADATA |
>  EXTENDED_IREF |
>  SKINNY_METADATA )
> cache_generation150682
> uuid_tree_generation150679
> dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
> dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
> dev_item.type   0
> dev_item.total_bytes2000398934016
> dev_item.bytes_used 1647136735232
> dev_item.io_align   4096
> dev_item.io_width   4096
> dev_item.sector_size4096
> dev_item.devid  1
> dev_item.dev_group  0
> dev_item.seek_speed 0
> dev_item.bandwidth  0
> dev_item.generation 0
> sys_chunk_array[2048]:
>item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
>length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
>io_align 65536 io_width 65536 sector_size 4096
>num_stripes 8 sub_stripes 2
>stripe 0 devid 7 offset 1083674984448
>dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
>stripe 1 devid 8 offset 1083674984448
>dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
>stripe 2 devid 1 offset 1365901312
>dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
>stripe 3 devid 3 offset 1345978368
>dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
>stripe 4 devid 4 offset 1345978368
>dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
>stripe 5 devid 5 offset 1345978368
>dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
>stripe 6 devid 6 offset 1345978368
>dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
>stripe 7 devid 2 offset 1345978368
>dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841
> backup_roots[4]:
>backup 0:
>backup_tree_root:   5223857717248   gen: 150680 level: 
> 1
>backup_chunk_root:  8669488005120   gen: 150678 level: 
> 1
>backup_extent_root: 5223867383808   gen: 150680 level: 
> 2
>backup_fs_root: 0   gen: 0  level: 0
>backup_dev_root:5224791523328   gen: 150680 level: 
> 1
>backup_csum_root:   5224802140160   gen: 150680 level: 
> 3
>backup_total_bytes: 16003191472128
>backup_bytes_used:  6411278503936
>backup_num_devices: 8
> 
>backup 1:
>backup_tree_root:   5224155807744   gen: 150681 level: 
> 1
>backup_chunk_root:  8669488005120   gen: 150678 level: 
> 1
>backup_extent_root: 5224156233728   gen: 150681 level: 
> 2
>backup_fs_root: 0   gen: 0  level: 0
>backup_dev_root:5224633155584   gen: 150681 level: 
> 1
>backup_csum_root:   5224634941440   gen: 150681 level: 
> 3
>backup_total_bytes: 16003191472128
>backup_bytes_used:  6411278503936
>backup_num_devices: 8
> 
>backup 2:
>backup_tree_root:   5223857717248   gen: 150682 level: 
> 1
>backup_chunk_root:  8669488005120   gen: 150678 level: 
> 1
>

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

[root@server ~]#  btrfs-show-super -af /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
-
csum_type   0 (crc32c)
csum_size   4
csum0x17d56ce0 [match]
bytenr  65536
flags   0x1
( WRITTEN )
magic   _BHRfS_M [match]
fsid0576d577-8954-4a60-a02b-9492b3c29318
label   main_pool
generation  150682
root5223857717248
sys_array_size  321
chunk_root_generation   150678
root_level  1
chunk_root  8669488005120
chunk_root_level1
log_root0
log_root_transid0
log_root_level  0
total_bytes 16003191472128
bytes_used  6411278503936
sectorsize  4096
nodesize16384
leafsize16384
stripesize  4096
root_dir6
num_devices 8
compat_flags0x0
compat_ro_flags 0x0
incompat_flags  0x161
( MIXED_BACKREF |
  BIG_METADATA |
  EXTENDED_IREF |
  SKINNY_METADATA )
cache_generation150682
uuid_tree_generation150679
dev_item.uuid   46abffa8-7afe-451f-93c6-abb8e589c4e8
dev_item.fsid   0576d577-8954-4a60-a02b-9492b3c29318 [match]
dev_item.type   0
dev_item.total_bytes2000398934016
dev_item.bytes_used 1647136735232
dev_item.io_align   4096
dev_item.io_width   4096
dev_item.sector_size4096
dev_item.devid  1
dev_item.dev_group  0
dev_item.seek_speed 0
dev_item.bandwidth  0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 8669487824896)
length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID10
io_align 65536 io_width 65536 sector_size 4096
num_stripes 8 sub_stripes 2
stripe 0 devid 7 offset 1083674984448
dev_uuid 566fb8a3-d6de-4230-8b70-a5fda0a120f6
stripe 1 devid 8 offset 1083674984448
dev_uuid 845aefb2-e0a6-479a-957b-a82fb7207d6c
stripe 2 devid 1 offset 1365901312
dev_uuid 46abffa8-7afe-451f-93c6-abb8e589c4e8
stripe 3 devid 3 offset 1345978368
dev_uuid 95921633-2fc1-479f-a3ba-e6e5a1989755
stripe 4 devid 4 offset 1345978368
dev_uuid 20828f0e-4661-4987-ac11-72814c1e423a
stripe 5 devid 5 offset 1345978368
dev_uuid 2c3cd71f-5178-48e7-8032-6b6eec023197
stripe 6 devid 6 offset 1345978368
dev_uuid 806a47e5-cac4-41c9-abb9-5c49506459e1
stripe 7 devid 2 offset 1345978368
dev_uuid e1358e0e-edaf-4505-9c71-ed0862c45841
backup_roots[4]:
backup 0:
backup_tree_root:   5223857717248   gen: 150680 level: 1
backup_chunk_root:  8669488005120   gen: 150678 level: 1
backup_extent_root: 5223867383808   gen: 150680 level: 2
backup_fs_root: 0   gen: 0  level: 0
backup_dev_root:5224791523328   gen: 150680 level: 1
backup_csum_root:   5224802140160   gen: 150680 level: 3
backup_total_bytes: 16003191472128
backup_bytes_used:  6411278503936
backup_num_devices: 8

backup 1:
backup_tree_root:   5224155807744   gen: 150681 level: 1
backup_chunk_root:  8669488005120   gen: 150678 level: 1
backup_extent_root: 5224156233728   gen: 150681 level: 2
backup_fs_root: 0   gen: 0  level: 0
backup_dev_root:5224633155584   gen: 150681 level: 1
backup_csum_root:   5224634941440   gen: 150681 level: 3
backup_total_bytes: 16003191472128
backup_bytes_used:  6411278503936
backup_num_devices: 8

backup 2:
backup_tree_root:   5223857717248   gen: 150682 level: 1
backup_chunk_root:  8669488005120   gen: 150678 level: 1
backup_extent_root: 5223867383808   gen: 150682 level: 2
backup_fs_root: 0   gen: 0  level: 0
backup_dev_root:5224622358528   gen: 150682 level: 1
backup_csum_root:   5224675344384   gen: 150682 level: 3
backup_total_bytes: 16003191472128
backup_bytes_used:  6411278503936
   

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Qu Wenruo]

At 02/14/2017 08:23 AM, Tomasz Kusmierz wrote:

Forgot to mention:

btrfs inspect-internal dump-super -af /dev/sdc


Your btrfs-progs is somewhat old, which doesn't integrate dump super 
into inspect-internal.


In that case, you can use btrfs-show-super -af instead.

Thanks,
Qu


btrfs inspect-internal: unknown token 'dump-super'
usage: btrfs inspect-internal  

btrfs inspect-internal inode-resolve [-v]  
Get file system paths for the given inode
btrfs inspect-internal logical-resolve [-Pv] [-s bufsize]  
Get file system paths for the given logical address
btrfs inspect-internal subvolid-resolve  
Get file system paths for the given subvolume ID.
btrfs inspect-internal rootid 
Get tree ID of the containing subvolume of path.
btrfs inspect-internal min-dev-size [options] 
Get the minimum size the device can be shrunk to. The

query various internal information

On 13 February 2017 at 14:58, Tomasz Kusmierz  wrote:

Problem is to send a larger log into this mailing list :/

Anyway: uname -a
Linux tevva-server 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10
20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux


cut from messages (bear in mind that this is a single cut with a bit
cut from inside of it to fit it in the email)

Feb 10 00:17:14 server journal: ==>
/var/log/gitlab/gitlab-shell/gitlab-shell.log <==
Feb 10 00:17:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:00 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:59 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:05 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:06 server kernel: BTRFS info (device sdc): relocating
block group 12353563131904 flags 65
Feb 10 00:18:06 server journal:
Feb 10 00:18:06 server journal: ==> /var/log/gitlab/sidekiq/current <==
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99341
2017-02-10T00:18:06.993Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: start
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99571
2017-02-10T00:18:06.995Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:18:07 server journal: 2017-02-10_00:18:07.00454
2017-02-10T00:18:07.004Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: done: 0.011 sec
Feb 10 00:18:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:18:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:43 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:48 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:49 server kernel: BTRFS info (device sdc): relocating
block group 12349268164608 flags 65
Feb 10 00:19:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:00 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51409
2017-02-10T00:19:02.513Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51449
2017-02-10T00:19:02.514Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: start
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.52994
2017-02-10T00:19:02.529Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: done: 0.015 sec
Feb 10 00:19:26 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:32 server kernel: BTRFS info (device sdc): relocating
block group 12344973197312 flags 65
Feb 10 00:19:51 server kernel: perf: interrupt took too long (2513 >
2500), lowering kernel.perf_event_max_sample_rate to 79000
Feb 10 00:20:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:59 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:20:10 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15695
2017-02-10T00:20:10.156Z 382 TID-otrsptg48
RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: start
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15968
2017-02-10T00:20:10.159Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: repository_check_worker
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17180
2017-02-10T00:20:10.171Z 382 TID-otrsptilo 

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

Forgot to mention:

btrfs inspect-internal dump-super -af /dev/sdc

btrfs inspect-internal: unknown token 'dump-super'
usage: btrfs inspect-internal  

btrfs inspect-internal inode-resolve [-v]  
Get file system paths for the given inode
btrfs inspect-internal logical-resolve [-Pv] [-s bufsize]  
Get file system paths for the given logical address
btrfs inspect-internal subvolid-resolve  
Get file system paths for the given subvolume ID.
btrfs inspect-internal rootid 
Get tree ID of the containing subvolume of path.
btrfs inspect-internal min-dev-size [options] 
Get the minimum size the device can be shrunk to. The

query various internal information

On 13 February 2017 at 14:58, Tomasz Kusmierz  wrote:
> Problem is to send a larger log into this mailing list :/
>
> Anyway: uname -a
> Linux tevva-server 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10
> 20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
>
>
> cut from messages (bear in mind that this is a single cut with a bit
> cut from inside of it to fit it in the email)
>
> Feb 10 00:17:14 server journal: ==>
> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
> Feb 10 00:17:30 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:17:29 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:18:00 server kernel: BTRFS info (device sdc): found 22 extents
> Feb 10 00:18:01 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:17:59 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:18:05 server kernel: BTRFS info (device sdc): found 22 extents
> Feb 10 00:18:06 server kernel: BTRFS info (device sdc): relocating
> block group 12353563131904 flags 65
> Feb 10 00:18:06 server journal:
> Feb 10 00:18:06 server journal: ==> /var/log/gitlab/sidekiq/current <==
> Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99341
> 2017-02-10T00:18:06.993Z 382 TID-otrr6ws48 PruneOldEventsWorker
> JID-99d3a4fb69be748c8674b5e1 INFO: start
> Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99571
> 2017-02-10T00:18:06.995Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
> with name: prune_old_events_worker
> Feb 10 00:18:07 server journal: 2017-02-10_00:18:07.00454
> 2017-02-10T00:18:07.004Z 382 TID-otrr6ws48 PruneOldEventsWorker
> JID-99d3a4fb69be748c8674b5e1 INFO: done: 0.011 sec
> Feb 10 00:18:30 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:18:29 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:18:43 server kernel: BTRFS info (device sdc): found 32 extents
> Feb 10 00:18:48 server kernel: BTRFS info (device sdc): found 32 extents
> Feb 10 00:18:49 server kernel: BTRFS info (device sdc): relocating
> block group 12349268164608 flags 65
> Feb 10 00:19:01 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:19:00 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51409
> 2017-02-10T00:19:02.513Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
> with name: prune_old_events_worker
> Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51449
> 2017-02-10T00:19:02.514Z 382 TID-otrspth10 PruneOldEventsWorker
> JID-4a162ace334771baf4befbb7 INFO: start
> Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.52994
> 2017-02-10T00:19:02.529Z 382 TID-otrspth10 PruneOldEventsWorker
> JID-4a162ace334771baf4befbb7 INFO: done: 0.015 sec
> Feb 10 00:19:26 server kernel: BTRFS info (device sdc): found 33 extents
> Feb 10 00:19:31 server kernel: BTRFS info (device sdc): found 33 extents
> Feb 10 00:19:31 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:19:29 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:19:32 server kernel: BTRFS info (device sdc): relocating
> block group 12344973197312 flags 65
> Feb 10 00:19:51 server kernel: perf: interrupt took too long (2513 >
> 2500), lowering kernel.perf_event_max_sample_rate to 79000
> Feb 10 00:20:00 server journal: 192.168.1.253 - wally_tm
> [10/Feb/2017:00:19:59 +] "PROPFIND /remote.php/webdav/Pictures
> HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
> Feb 10 00:20:10 server kernel: BTRFS info (device sdc): found 32 extents
> Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15695
> 2017-02-10T00:20:10.156Z 382 TID-otrsptg48
> RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: start
> Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15968
> 2017-02-10T00:20:10.159Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
> with name: repository_check_worker
> Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17180
> 2017-02-10T00:20:10.171Z 382 TID-otrsptilo PruneOldEventsWorker
> JID-4fa75dc5a3d36957d1034f56 INFO: start
> Feb 10 00:20:10 server 

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Tomasz Kusmierz]

Problem is to send a larger log into this mailing list :/

Anyway: uname -a
Linux tevva-server 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10
20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux


cut from messages (bear in mind that this is a single cut with a bit
cut from inside of it to fit it in the email)

Feb 10 00:17:14 server journal: ==>
/var/log/gitlab/gitlab-shell/gitlab-shell.log <==
Feb 10 00:17:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:00 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:17:59 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:05 server kernel: BTRFS info (device sdc): found 22 extents
Feb 10 00:18:06 server kernel: BTRFS info (device sdc): relocating
block group 12353563131904 flags 65
Feb 10 00:18:06 server journal:
Feb 10 00:18:06 server journal: ==> /var/log/gitlab/sidekiq/current <==
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99341
2017-02-10T00:18:06.993Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: start
Feb 10 00:18:06 server journal: 2017-02-10_00:18:06.99571
2017-02-10T00:18:06.995Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:18:07 server journal: 2017-02-10_00:18:07.00454
2017-02-10T00:18:07.004Z 382 TID-otrr6ws48 PruneOldEventsWorker
JID-99d3a4fb69be748c8674b5e1 INFO: done: 0.011 sec
Feb 10 00:18:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:18:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:18:43 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:48 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:18:49 server kernel: BTRFS info (device sdc): relocating
block group 12349268164608 flags 65
Feb 10 00:19:01 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:00 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51409
2017-02-10T00:19:02.513Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.51449
2017-02-10T00:19:02.514Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: start
Feb 10 00:19:02 server journal: 2017-02-10_00:19:02.52994
2017-02-10T00:19:02.529Z 382 TID-otrspth10 PruneOldEventsWorker
JID-4a162ace334771baf4befbb7 INFO: done: 0.015 sec
Feb 10 00:19:26 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server kernel: BTRFS info (device sdc): found 33 extents
Feb 10 00:19:31 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:19:32 server kernel: BTRFS info (device sdc): relocating
block group 12344973197312 flags 65
Feb 10 00:19:51 server kernel: perf: interrupt took too long (2513 >
2500), lowering kernel.perf_event_max_sample_rate to 79000
Feb 10 00:20:00 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:19:59 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:20:10 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15695
2017-02-10T00:20:10.156Z 382 TID-otrsptg48
RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: start
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.15968
2017-02-10T00:20:10.159Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: repository_check_worker
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17180
2017-02-10T00:20:10.171Z 382 TID-otrsptilo PruneOldEventsWorker
JID-4fa75dc5a3d36957d1034f56 INFO: start
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.17430
2017-02-10T00:20:10.174Z 382 TID-otrr6wqok INFO: Cron Jobs - add job
with name: prune_old_events_worker
Feb 10 00:20:10 server journal: 2017-02-10_00:20:10.18948
2017-02-10T00:20:10.189Z 382 TID-otrsptilo PruneOldEventsWorker
JID-4fa75dc5a3d36957d1034f56 INFO: done: 0.018 sec
Feb 10 00:20:11 server journal: 2017-02-10_00:20:11.00073
2017-02-10T00:20:11.000Z 382 TID-otrsptg48
RepositoryCheck::BatchWorker JID-a315de601bca406340583585 INFO: done:
0.844 sec
Feb 10 00:20:14 server kernel: BTRFS info (device sdc): found 32 extents
Feb 10 00:20:15 server kernel: BTRFS info (device sdc): relocating
block group 12340678230016 flags 65
Feb 10 00:20:30 server journal: 192.168.1.253 - wally_tm
[10/Feb/2017:00:20:29 +] "PROPFIND /remote.php/webdav/Pictures
HTTP/1.1" 207 1024 "-" "Mozilla/5.0 (Linux) mirall/2.1.1"
Feb 10 00:20:41 server kernel: systemd-tmpfile: 127 output lines
suppressed due to ratelimiting

Re: FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.

[Qu Wenruo]

At 02/12/2017 09:17 AM, Tomasz Kusmierz wrote:

Hi all,

So my main storage filesystem got some sort of veird corruption (that
I can gather). Everything seems to work OK, but when I try to create a
snapshot or run balance (no filters) it will get remounted read only.


Kernel version please.



Fun part is that balance seems to be running even on read only FS, and
I continuously get kernel traces in /var/log/messages  so it might
as well in the back ground silently eat my data away :/


Kernel backtrace please.

It would be better if you could paste the *first* kernel backtrace, as 
that could be the cause, and following kernel backtrace is just warning 
from btrfs_abort_transaction() without meaningful output.


I just see some normal messages, but no kernel backtrace.




UPDATE:

Yeah, after rebooting the system it does not even mount the FS,
mount.btrfs sits in some sort of spinlock and consumes 100% of singe
core.



UPDATE 2:

System is completelly cooked :/

[root@server ~]# btrfs fi show
Label: 'rockstor_server'  uuid: 5581a647-40ef-4a7a-9d73-847bf35a142b
Total devices 1 FS bytes used 5.72GiB
devid1 size 53.17GiB used 7.03GiB path /dev/sda2

Label: 'broken_pool'  uuid: 26095277-a234-455b-8c97-8dac8ad934c8
Total devices 2 FS bytes used 193.52GiB
devid1 size 1.82TiB used 196.03GiB path /dev/sdb
devid2 size 1.82TiB used 196.03GiB path /dev/sdi

Label: 'main_pool'  uuid: 0576d577-8954-4a60-a02b-9492b3c29318
Total devices 8 FS bytes used 5.83TiB
devid1 size 1.82TiB used 1.50TiB path /dev/sdc
devid2 size 1.82TiB used 1.50TiB path /dev/sdd
devid3 size 1.82TiB used 1.50TiB path /dev/sde
devid4 size 1.82TiB used 1.50TiB path /dev/sdf
devid5 size 1.82TiB used 1.50TiB path /dev/sdg
devid6 size 1.82TiB used 1.50TiB path /dev/sdh
devid7 size 1.82TiB used 1.50TiB path /dev/sdj
devid8 size 1.82TiB used 1.50TiB path /dev/sdk

[root@server ~]# mount /dev/sdc /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.
[root@server ~]# mount /dev/sdd /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.
[root@server ~]# mount /dev/sde /mnt2/main_pool/
mount: wrong fs type, bad option, bad superblock on /dev/sde,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.


dmesg tail retuns:
[ 9507.835629] systemd-udevd[1873]: Validate module index
[ 9507.835656] systemd-udevd[1873]: Check if link configuration needs reloading.
[ 9507.835690] systemd-udevd[1873]: seq 3698 queued, 'add' 'bdi'
[ 9507.835873] systemd-udevd[1873]: seq 3698 forked new worker [13858]
[ 9507.836202] BTRFS info (device sdd): disk space caching is enabled
[ 9507.836204] BTRFS info (device sdd): has skinny extents
[ 9507.836322] systemd-udevd[13858]: seq 3698 running
[ 9507.836443] systemd-udevd[13858]: no db file to read
/run/udev/data/+bdi:btrfs-4: No such file or directory
[ 9507.836474] systemd-udevd[13858]: RUN '/bin/mknod
/dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1
[ 9507.837366] systemd-udevd[13861]: starting '/bin/mknod
/dev/btrfs-control c 10 234'
[ 9507.837833] BTRFS error (device sdd): failed to read the system array: -5
[ 9507.838231] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c
10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists'
[ 9507.838262] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c
10 234' [13861] exit with return code 1
[ 9507.854757] BTRFS: open_ctree failed
[ 9511.370878] BTRFS info (device sdd): disk space caching is enabled
[ 9511.370881] BTRFS info (device sdd): has skinny extents
[ 9511.375097] BTRFS error (device sdd): failed to read the system array: -5


Btrfs failed to read system chunk array from super block.
Normally this means your primary super block is cooked up.
There may still be chance to recover your fs using backup superblocks.

Please paste the output of "btrfs inspect-internal dump-super -af "


Thanks,
Qu


[ 9511.392792] BTRFS: open_ctree failed
[ 9514.233627] BTRFS: device label main_pool devid 3 transid 150680 /dev/sde
[ 9514.234399] systemd-udevd[1873]: Validate module index
[ 9514.234431] systemd-udevd[1873]: Check if link configuration needs reloading.
[ 9514.234465] systemd-udevd[1873]: seq 3702 queued, 'add' 'bdi'
[ 9514.234522] systemd-udevd[1873]: passed 142 bytes to netlink
monitor 0x5628f65d40d0
[ 9514.234554] systemd-udevd[13882]: seq 3702 running
[ 9514.234780] systemd-udevd[13882]: no db file to read
/run/udev/data/+bdi:btrfs-6: No such file or directory
[ 9514.234790] BTRFS info (device sde): disk space caching is enabled
[