Re: how to understand "btrfs fi show" output? "No space left" issues

[Johannes Hirte]

On 2016 Sep 20, Peter Becker wrote:
> Data, RAID1: total=417.12GiB, used=131.33GiB
> 
> You have 417(total)-131(used) blocks wo are only partial filled.
> You should balance your file-system.
> 
> At first you need some free space. You could remove some files / old
> snapshots etc. or you add a empty USB-Stick with min. 4 GB to your
> BTRFS-Pool (after balancing complete you can remove the stick from the
> pool).

He has plenty of space. What you're describing is the case that either
data pool or metadata pool is full, the other has enough space and
nothing is left that could be allocated to the full pool. In this case
rebalancing would help. But in Tomasz' case there is enough space in
every pool, so the allocator should use it. This really sounds like a
bug.

> But at first you should try to free emty data and meta data blocks:
> 
> btrfs balance start -musage=0 /mnt
> btrfs balance start -dusage=0 /mnt

Since kernel 3.18 this is done automatically.


regards,
  Johannes
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to understand "btrfs fi show" output? "No space left" issues

[Tomasz Chmielewski]

On 2016-09-21 11:51, Chris Murphy wrote:


I'm coming into this late and realize most questions have been
answered. But I take the position this is a bug, clearly there's
enough space when df reports only 33% used, and therefore it's
important to gather information about the file system in its current
state so the devs can make decisions. Manually running balance is the
correct work around, but it's bad Ux and should not be necessary (even
though it's known to sometimes be necessary).

Anyway, in this case there is room in all chunks and GlobalReserve
used is 0.00B. Metadata has a bit over a gigabyte of unused space in
its allocated block groups. So at the moment I'm thinking it's a bug.
The two things that'd be useful if you can reproduce this problem at
some point, by NOT trying to prevent it again, are:

grep . -IR /sys/fs/btrfs//allocation/

 pick the UUID for the affected fs volume.

btrfs-debugfs found in btrfs-progs upstream as a python program but
typically not packaged by distros
https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs

Takes the form:

sudo ./btrfs-debugfs -b 

It'll show you the percent each block group is actually being used so
you can have a good idea what -dusage value to use (in your case) to
free up space. That should help, but ultimately it's a work around,
not a real fix. There shouldn't be enospc anyway.

So if it happens again, first capture the above two bits of
information, and then if  you feel like testing kernel 4.8rc7 do that.
It has a massive pile of enoscp related rework and I bet Josef would
like to know if the problem reproduces with that kernel. As in, just
change kernels, don't try to fix it with balance first.


OK, so it again came to a point where it's full when it's not.

Running 4.8.1 now for around 30 days.

Some btrfs utils output first:

# btrfs fi show /var/lib/lxd
Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
Total devices 2 FS bytes used 182.93GiB
devid1 size 423.13GiB used 423.13GiB path /dev/sda3
devid2 size 423.13GiB used 423.13GiB path /dev/sdb3


# btrfs fi df /var/lib/lxd
Data, RAID1: total=415.09GiB, used=177.41GiB
System, RAID1: total=32.00MiB, used=80.00KiB
Metadata, RAID1: total=8.00GiB, used=5.52GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi usage /var/lib/lxd
Overall:
Device size: 846.25GiB
Device allocated:846.25GiB
Device unallocated:2.05MiB
Device missing:  0.00B
Used:365.86GiB
Free (estimated):237.69GiB  (min: 237.69GiB)
Data ratio:   2.00
Metadata ratio:   2.00
Global reserve:  512.00MiB  (used: 0.00B)

Data,RAID1: Size:415.09GiB, Used:177.41GiB
   /dev/sda3 415.09GiB
   /dev/sdb3 415.09GiB

Metadata,RAID1: Size:8.00GiB, Used:5.52GiB
   /dev/sda3   8.00GiB
   /dev/sdb3   8.00GiB

System,RAID1: Size:32.00MiB, Used:80.00KiB
   /dev/sda3  32.00MiB
   /dev/sdb3  32.00MiB

Unallocated:
   /dev/sda3   1.02MiB
   /dev/sdb3   1.02MiB


# df -h
/dev/sda3   424G  184G  238G  44% /var/lib/lxd



Finally, the output from both /sys/fs/btrfs/ and btrfs-debugfs:


# grep . -IR 
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/

/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/flags:2
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/raid1/used_bytes:81920
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/raid1/total_bytes:33554432
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/bytes_pinned:0
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/disk_total:67108864
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/bytes_may_use:0
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/bytes_readonly:0
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/bytes_used:81920
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/bytes_reserved:0
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/disk_used:163840
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/total_bytes_pinned:0
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/system/total_bytes:33554432
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/flags:4
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/raid1/used_bytes:5927518208
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/raid1/total_bytes:8589934592
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/bytes_pinned:2080768
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/disk_total:17179869184
/sys/fs/btrfs/f5f30428-ec5b-4497-82de-6e20065e6f61/allocation/metadata/bytes_may_use:634126336

Re: how to understand "btrfs fi show" output? "No space left" issues

[Tomasz Chmielewski]

On 2016-09-21 11:51, Chris Murphy wrote:



So if it happens again, first capture the above two bits of
information, and then if  you feel like testing kernel 4.8rc7 do that.
It has a massive pile of enoscp related rework and I bet Josef would
like to know if the problem reproduces with that kernel. As in, just
change kernels, don't try to fix it with balance first.


Looks like 4.8 helped (running 4.8rc8 now).

With 4.7, after balance, the "used" value continued to grow, to around 
300 GB, although used space shown by "df" was more or less constant at 
130-140 GB:


# btrfs fi show /var/lib/lxd
Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
Total devices 2 FS bytes used 135.40GiB <- was growing
devid1 size 423.13GiB used 277.03GiB path /dev/sda3
devid2 size 423.13GiB used 277.03GiB path /dev/sdb3


After upgrading to 4.8rc8, "used" value dropped, so hopefully it's fixed 
now.



Tomasz Chmielewski
https://lxadm.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to understand "btrfs fi show" output? "No space left" issues

[Peter Becker]

for the future. disable COW for all database containers

2016-09-20 9:28 GMT+02:00 Peter Becker :
> * If this NOT solve the "No space left" issues you must remove old snapshots.
>
> 2016-09-20 9:27 GMT+02:00 Peter Becker :
>> Data, RAID1: total=417.12GiB, used=131.33GiB
>>
>> You have 417(total)-131(used) blocks wo are only partial filled.
>> You should balance your file-system.
>>
>> At first you need some free space. You could remove some files / old
>> snapshots etc. or you add a empty USB-Stick with min. 4 GB to your
>> BTRFS-Pool (after balancing complete you can remove the stick from the
>> pool).
>>
>> But at first you should try to free emty data and meta data blocks:
>>
>> btrfs balance start -musage=0 /mnt
>> btrfs balance start -dusage=0 /mnt
>>
>> Then you an run a full balance or a partial balance:
>>
>> #a partial balance with reorganize data blocks less then 50% filled
>> btrfs balance start -dusage=50 /mnt
>>
>> #or a full balance
>> btrfs balance start /mnt
>>
>> Because of a possible bug you should disable all snapshot scripts
>> (like cron-jobs) during the balance.
>>
>> If this solve the "No space left" issues you must remove old snapshots.
>>
>> 2016-09-20 8:58 GMT+02:00 Hugo Mills :
>>> On Tue, Sep 20, 2016 at 03:47:14PM +0900, Tomasz Chmielewski wrote:
 How to understand the following "btrfs fi show" output?
>>>
>>> This gives a write-up (and worked example) of an answer to your question:
>>>
>>> https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools
>>>
>>>If you've got any follow-up questions after reading it, please do
>>> come back and we can try to improve the FAQ entry. :)
>>>
>>>Hugo.
>>>
 # btrfs fi show /var/lib/lxd
 Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
 Total devices 2 FS bytes used 136.18GiB
 devid1 size 423.13GiB used 423.13GiB path /dev/sda3
 devid2 size 423.13GiB used 423.13GiB path /dev/sdb3

 Why is it "size 423.13GiB used 423.13GiB"? Is it full?

 I had "No space left" on this filesystem just yesterday (running
 kernel 4.7.4). This is btrfs RAID-1 on SSD disks. This filesystem is
 used for 20-30 LXD containers with different roles (mongo, mysql,
 postgres databases, webservers etc.), around 150 read-only
 snapshots, btrfs compression is disabled.


 Both "btrfs fi df" and "df -h" show plenty of space:

 # btrfs fi df /var/lib/lxd
 Data, RAID1: total=417.12GiB, used=131.33GiB
 System, RAID1: total=8.00MiB, used=80.00KiB
 Metadata, RAID1: total=6.00GiB, used=4.86GiB
 GlobalReserve, single: total=512.00MiB, used=0.00B


 # df -h
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/sda3   424G  137G  286G  33% /var/lib/lxd



 Tomasz Chmielewski
 https://lxadm.com
 --
 To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> Hugo Mills | I can resist everything except temptation.
>>> hugo@... carfax.org.uk |
>>> http://carfax.org.uk/  |
>>> PGP: E2AB1DE4  |
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to understand "btrfs fi show" output? "No space left" issues

[Hugo Mills]

On Tue, Sep 20, 2016 at 03:47:14PM +0900, Tomasz Chmielewski wrote:
> How to understand the following "btrfs fi show" output?

This gives a write-up (and worked example) of an answer to your question:

https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_using_the_original_tools

   If you've got any follow-up questions after reading it, please do
come back and we can try to improve the FAQ entry. :)

   Hugo.

> # btrfs fi show /var/lib/lxd
> Label: 'btrfs'  uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
> Total devices 2 FS bytes used 136.18GiB
> devid1 size 423.13GiB used 423.13GiB path /dev/sda3
> devid2 size 423.13GiB used 423.13GiB path /dev/sdb3
> 
> Why is it "size 423.13GiB used 423.13GiB"? Is it full?
> 
> I had "No space left" on this filesystem just yesterday (running
> kernel 4.7.4). This is btrfs RAID-1 on SSD disks. This filesystem is
> used for 20-30 LXD containers with different roles (mongo, mysql,
> postgres databases, webservers etc.), around 150 read-only
> snapshots, btrfs compression is disabled.
> 
> 
> Both "btrfs fi df" and "df -h" show plenty of space:
> 
> # btrfs fi df /var/lib/lxd
> Data, RAID1: total=417.12GiB, used=131.33GiB
> System, RAID1: total=8.00MiB, used=80.00KiB
> Metadata, RAID1: total=6.00GiB, used=4.86GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 
> # df -h
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda3   424G  137G  286G  33% /var/lib/lxd
> 
> 
> 
> Tomasz Chmielewski
> https://lxadm.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Hugo Mills | I can resist everything except temptation.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4  |


signature.asc
Description: Digital signature