Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Chris Murphy]

On Tue, Aug 28, 2018 at 1:14 PM, Menion  wrote:
> You are correct, indeed in order to cleanup you need
>
> 1) someone realize that snapshots have been created
> 2) apt-brtfs-snapshot is manually installed on the system
>
> Assuming also that the snapshots created during do-release-upgrade are
> managed for auto cleanup

Ha! I should have read all the emails.

Anyway, good sleuthing. I think it's a good idea to file a bug report
on it, so at the least other people can fix it manually.


-- 
Chris Murphy

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Chris Murphy]

On Tue, Aug 28, 2018 at 8:56 AM, Menion  wrote:
> [sudo] password for menion:
> ID  gen top level   path
> --  --- -   
> 257 600627  5   /@
> 258 600626  5   /@home
> 296 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
> 297 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
> 298 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
>
> So, there are snapshots, right?

Yep. So you can use 'sudo btrfs fi du -s ' to get a
report on  how much exclusive space is being used by each of those
snapshots and I'll bet it all adds up to about 10G or whatever you're
missing.

>The time stamp is when I have launched
> do-release-upgrade, but it didn't ask anything about snapshot, neither
> I asked for it.

Yep, not sure what's creating them or what the cleanup policy is (if
there is one). So it's worth asking in an Ubuntu forum what these
snapshots are where they came from and what cleans them up so you
don't run out of space, or otherwise how to configure it if you want
more space just because.

I mean, it's a neat idea. But also it needs to clean up after itself
if for no other reason than to avoid user confusion :-)


> If it is confirmed, how can I remove the unwanted snapshot, keeping
> the current "visible" filesystem contents
> Sorry, I am still learning BTRFS and I would like to avoid mistakes
> Bye

You can definitely use Btrfs specific tools to get rid of the
snapshots and not piss off Btrfs at all. However, if you delete them
behind the back of the thing that created them in the first place, it
might get pissed off if they just suddenly go missing. Sometimes those
tools want to do the cleanups because it's tracking the snapshots and
what their purpose is. So if they just go away, it's like having the
rug pulled out from under them.

Anyway:

'sudo btrfs sub del ' will delete it.


Also, I can't tell you for sure what sort of write amplification Btrfs
contributes in your use case on eMMC compared to F2FS. Btrfs has a
"wandering trees" problem that F2FS doesn't have as big a problem.
It's not a big deal (probably) on other kinds of SSDs like SATA/SAS
and NVMe. But on eMMC? If it were SD Card I'd say you can keep using
Btrfs, and maybe mitigate the wandering trees with compression to
reduce overall writes. But if your eMMC is soldered onto a board, I
might consider F2FS instead. And Btrfs for other things.


-- 
Chris Murphy

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Austin S. Hemmelgarn]

On 2018-08-28 15:14, Menion wrote:

You are correct, indeed in order to cleanup you need

1) someone realize that snapshots have been created
2) apt-brtfs-snapshot is manually installed on the system
Your second requirement is only needed if you want the nice automated 
cleanup.  There's absolutely nothing preventing you from manually 
removing the snapshots.


Assuming also that the snapshots created during do-release-upgrade are 
managed for auto cleanup


Il martedì 28 agosto 2018, Noah Massey > ha scritto:


On Tue, Aug 28, 2018 at 1:25 PM Menion mailto:men...@gmail.com>> wrote:
 >
 > Ok, I have removed the snapshot and the free expected space is
here, thank you!
 > As a side note: apt-btrfs-snapshot was not installed, but it is
 > present in Ubuntu repository and I have used it (and I like the idea
 > of automatic snapshot during upgrade)
 > This means that the do-release-upgrade does it's own job on BTRFS,
 > silently which I believe is not good from the usability perspective,

You are correct. DistUpgradeController.py from python3-distupgrade
imports 'apt_btrfs_snapshot', which I read as coming from
/usr/lib/python3/dist-packages/apt_btrfs_snapshot.py, supplied by
apt-btrfs-snapshot, but I missed the fact that python3-distupgrade
ships its own
/usr/lib/python3/dist-packages/DistUpgrade/apt_btrfs_snapshot.py

So now it looks like that cannot be easily disabled, and without the
apt-btrfs-snapshot package scheduling cleanups it's not ever
automatically removed?

 > just google it, there is no mention of this behaviour
 > Il giorno mar 28 ago 2018 alle ore 19:07 Austin S. Hemmelgarn
 > mailto:ahferro...@gmail.com>> ha scritto:
 > >
 > > On 2018-08-28 12:05, Noah Massey wrote:
 > > > On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn
 > > > mailto:ahferro...@gmail.com>> wrote:
 > > >>
 > > >> On 2018-08-28 11:27, Noah Massey wrote:
 > > >>> On Tue, Aug 28, 2018 at 10:59 AM Menion mailto:men...@gmail.com>> wrote:
 > > 
 > >  [sudo] password for menion:
 > >  ID      gen     top level       path
 > >  --      ---     -       
 > >  257     600627  5               /@
 > >  258     600626  5               /@home
 > >  296     599489  5
 > > 
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
 > >  297     599489  5
 > > 
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
 > >  298     599489  5
 > > 
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
 > > 
 > >  So, there are snapshots, right? The time stamp is when I
have launched
 > >  do-release-upgrade, but it didn't ask anything about
snapshot, neither
 > >  I asked for it.
 > > >>>
 > > >>> This is an Ubuntu thing
 > > >>> `apt show apt-btrfs-snapshot`
 > > >>> which "will create a btrfs snapshot of the root filesystem
each time
 > > >>> that apt installs/removes/upgrades a software package."
 > > >> Not Ubuntu, Debian.  It's just that Ubuntu installs and
configures the
 > > >> package by default, while Debian does not.
 > > >
 > > > Ubuntu also maintains the package, and I did not find it in
Debian repositories.
 > > > I think it's also worth mentioning that these snapshots were
created
 > > > by the do-release-upgrade script using the package directly,
not as a
 > > > result of the apt configuration. Meaning if you do not want a
snapshot
 > > > taken prior to upgrade, you have to remove the apt-btrfs-snapshot
 > > > package prior to running the upgrade script. You cannot just
update
 > > > /etc/apt/apt.conf.d/80-btrfs-snapshot
 > > Hmm... I could have sworn that it was in the Debian repositories.
 > >
 > > That said, it's kind of stupid that the snapshot is not trivially
 > > optional for a release upgrade.  Yes, that's where it's
arguably the
 > > most important, but it's still kind of stupid to have to remove a
 > > package to get rid of that behavior and then reinstall it again
afterwards.

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Noah Massey]

On Tue, Aug 28, 2018 at 1:25 PM Menion  wrote:
>
> Ok, I have removed the snapshot and the free expected space is here, thank 
> you!
> As a side note: apt-btrfs-snapshot was not installed, but it is
> present in Ubuntu repository and I have used it (and I like the idea
> of automatic snapshot during upgrade)
> This means that the do-release-upgrade does it's own job on BTRFS,
> silently which I believe is not good from the usability perspective,

You are correct. DistUpgradeController.py from python3-distupgrade
imports 'apt_btrfs_snapshot', which I read as coming from
/usr/lib/python3/dist-packages/apt_btrfs_snapshot.py, supplied by
apt-btrfs-snapshot, but I missed the fact that python3-distupgrade
ships its own /usr/lib/python3/dist-packages/DistUpgrade/apt_btrfs_snapshot.py

So now it looks like that cannot be easily disabled, and without the
apt-btrfs-snapshot package scheduling cleanups it's not ever
automatically removed?

> just google it, there is no mention of this behaviour
> Il giorno mar 28 ago 2018 alle ore 19:07 Austin S. Hemmelgarn
>  ha scritto:
> >
> > On 2018-08-28 12:05, Noah Massey wrote:
> > > On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn
> > >  wrote:
> > >>
> > >> On 2018-08-28 11:27, Noah Massey wrote:
> > >>> On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:
> > 
> >  [sudo] password for menion:
> >  ID  gen top level   path
> >  --  --- -   
> >  257 600627  5   /@
> >  258 600626  5   /@home
> >  296 599489  5
> >  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
> >  297 599489  5
> >  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
> >  298 599489  5
> >  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
> > 
> >  So, there are snapshots, right? The time stamp is when I have launched
> >  do-release-upgrade, but it didn't ask anything about snapshot, neither
> >  I asked for it.
> > >>>
> > >>> This is an Ubuntu thing
> > >>> `apt show apt-btrfs-snapshot`
> > >>> which "will create a btrfs snapshot of the root filesystem each time
> > >>> that apt installs/removes/upgrades a software package."
> > >> Not Ubuntu, Debian.  It's just that Ubuntu installs and configures the
> > >> package by default, while Debian does not.
> > >
> > > Ubuntu also maintains the package, and I did not find it in Debian 
> > > repositories.
> > > I think it's also worth mentioning that these snapshots were created
> > > by the do-release-upgrade script using the package directly, not as a
> > > result of the apt configuration. Meaning if you do not want a snapshot
> > > taken prior to upgrade, you have to remove the apt-btrfs-snapshot
> > > package prior to running the upgrade script. You cannot just update
> > > /etc/apt/apt.conf.d/80-btrfs-snapshot
> > Hmm... I could have sworn that it was in the Debian repositories.
> >
> > That said, it's kind of stupid that the snapshot is not trivially
> > optional for a release upgrade.  Yes, that's where it's arguably the
> > most important, but it's still kind of stupid to have to remove a
> > package to get rid of that behavior and then reinstall it again afterwards.

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Menion]

Ok, I have removed the snapshot and the free expected space is here, thank you!
As a side note: apt-btrfs-snapshot was not installed, but it is
present in Ubuntu repository and I have used it (and I like the idea
of automatic snapshot during upgrade)
This means that the do-release-upgrade does it's own job on BTRFS,
silently which I believe is not good from the usability perspective,
just google it, there is no mention of this behaviour
Il giorno mar 28 ago 2018 alle ore 19:07 Austin S. Hemmelgarn
 ha scritto:
>
> On 2018-08-28 12:05, Noah Massey wrote:
> > On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn
> >  wrote:
> >>
> >> On 2018-08-28 11:27, Noah Massey wrote:
> >>> On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:
> 
>  [sudo] password for menion:
>  ID  gen top level   path
>  --  --- -   
>  257 600627  5   /@
>  258 600626  5   /@home
>  296 599489  5
>  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
>  297 599489  5
>  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
>  298 599489  5
>  /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
> 
>  So, there are snapshots, right? The time stamp is when I have launched
>  do-release-upgrade, but it didn't ask anything about snapshot, neither
>  I asked for it.
> >>>
> >>> This is an Ubuntu thing
> >>> `apt show apt-btrfs-snapshot`
> >>> which "will create a btrfs snapshot of the root filesystem each time
> >>> that apt installs/removes/upgrades a software package."
> >> Not Ubuntu, Debian.  It's just that Ubuntu installs and configures the
> >> package by default, while Debian does not.
> >
> > Ubuntu also maintains the package, and I did not find it in Debian 
> > repositories.
> > I think it's also worth mentioning that these snapshots were created
> > by the do-release-upgrade script using the package directly, not as a
> > result of the apt configuration. Meaning if you do not want a snapshot
> > taken prior to upgrade, you have to remove the apt-btrfs-snapshot
> > package prior to running the upgrade script. You cannot just update
> > /etc/apt/apt.conf.d/80-btrfs-snapshot
> Hmm... I could have sworn that it was in the Debian repositories.
>
> That said, it's kind of stupid that the snapshot is not trivially
> optional for a release upgrade.  Yes, that's where it's arguably the
> most important, but it's still kind of stupid to have to remove a
> package to get rid of that behavior and then reinstall it again afterwards.

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Austin S. Hemmelgarn]

On 2018-08-28 12:05, Noah Massey wrote:

On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn
 wrote:


On 2018-08-28 11:27, Noah Massey wrote:

On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:


[sudo] password for menion:
ID  gen top level   path
--  --- -   
257 600627  5   /@
258 600626  5   /@home
296 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
297 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
298 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30

So, there are snapshots, right? The time stamp is when I have launched
do-release-upgrade, but it didn't ask anything about snapshot, neither
I asked for it.


This is an Ubuntu thing
`apt show apt-btrfs-snapshot`
which "will create a btrfs snapshot of the root filesystem each time
that apt installs/removes/upgrades a software package."

Not Ubuntu, Debian.  It's just that Ubuntu installs and configures the
package by default, while Debian does not.


Ubuntu also maintains the package, and I did not find it in Debian repositories.
I think it's also worth mentioning that these snapshots were created
by the do-release-upgrade script using the package directly, not as a
result of the apt configuration. Meaning if you do not want a snapshot
taken prior to upgrade, you have to remove the apt-btrfs-snapshot
package prior to running the upgrade script. You cannot just update
/etc/apt/apt.conf.d/80-btrfs-snapshot

Hmm... I could have sworn that it was in the Debian repositories.

That said, it's kind of stupid that the snapshot is not trivially 
optional for a release upgrade.  Yes, that's where it's arguably the 
most important, but it's still kind of stupid to have to remove a 
package to get rid of that behavior and then reinstall it again afterwards.

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Noah Massey]

On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn
 wrote:
>
> On 2018-08-28 11:27, Noah Massey wrote:
> > On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:
> >>
> >> [sudo] password for menion:
> >> ID  gen top level   path
> >> --  --- -   
> >> 257 600627  5   /@
> >> 258 600626  5   /@home
> >> 296 599489  5
> >> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
> >> 297 599489  5
> >> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
> >> 298 599489  5
> >> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
> >>
> >> So, there are snapshots, right? The time stamp is when I have launched
> >> do-release-upgrade, but it didn't ask anything about snapshot, neither
> >> I asked for it.
> >
> > This is an Ubuntu thing
> > `apt show apt-btrfs-snapshot`
> > which "will create a btrfs snapshot of the root filesystem each time
> > that apt installs/removes/upgrades a software package."
> Not Ubuntu, Debian.  It's just that Ubuntu installs and configures the
> package by default, while Debian does not.

Ubuntu also maintains the package, and I did not find it in Debian repositories.
I think it's also worth mentioning that these snapshots were created
by the do-release-upgrade script using the package directly, not as a
result of the apt configuration. Meaning if you do not want a snapshot
taken prior to upgrade, you have to remove the apt-btrfs-snapshot
package prior to running the upgrade script. You cannot just update
/etc/apt/apt.conf.d/80-btrfs-snapshot

>
> This behavior in general is not specific to Debian either, a lot of
> distributions are either working on or already have this type of
> functionality, because it's the only sane and correct way to handle
> updates short of rebuilding the entire system from scratch.

Yup. Everyone in their own way, plus all the home-brews.

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Austin S. Hemmelgarn]

On 2018-08-28 11:27, Noah Massey wrote:

On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:


[sudo] password for menion:
ID  gen top level   path
--  --- -   
257 600627  5   /@
258 600626  5   /@home
296 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
297 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
298 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30

So, there are snapshots, right? The time stamp is when I have launched
do-release-upgrade, but it didn't ask anything about snapshot, neither
I asked for it.


This is an Ubuntu thing
`apt show apt-btrfs-snapshot`
which "will create a btrfs snapshot of the root filesystem each time
that apt installs/removes/upgrades a software package."
Not Ubuntu, Debian.  It's just that Ubuntu installs and configures the 
package by default, while Debian does not.


This behavior in general is not specific to Debian either, a lot of 
distributions are either working on or already have this type of 
functionality, because it's the only sane and correct way to handle 
updates short of rebuilding the entire system from scratch.



During the do-release-upgrade I got some issues due to the (very) bad
behaviour of the script in remote terminal, then I have fixed
everything manually and now the filesystem is operational in bionic
version
If it is confirmed, how can I remove the unwanted snapshot, keeping
the current "visible" filesystem contents


By default, the package runs a weekly cron job to cleanup old
snapshots. (Defaults to 90d, but you can configure that in
APT::Snapshots::MaxAge) Alternatively, you can cleanup with the
command yourself. Run `sudo apt-btrfs-snapshot list`, and then `sudo
apt-btrfs-snapshot delete `

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Noah Massey]

On Tue, Aug 28, 2018 at 10:59 AM Menion  wrote:
>
> [sudo] password for menion:
> ID  gen top level   path
> --  --- -   
> 257 600627  5   /@
> 258 600626  5   /@home
> 296 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
> 297 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
> 298 599489  5
> /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30
>
> So, there are snapshots, right? The time stamp is when I have launched
> do-release-upgrade, but it didn't ask anything about snapshot, neither
> I asked for it.

This is an Ubuntu thing
`apt show apt-btrfs-snapshot`
which "will create a btrfs snapshot of the root filesystem each time
that apt installs/removes/upgrades a software package."

> During the do-release-upgrade I got some issues due to the (very) bad
> behaviour of the script in remote terminal, then I have fixed
> everything manually and now the filesystem is operational in bionic
> version
> If it is confirmed, how can I remove the unwanted snapshot, keeping
> the current "visible" filesystem contents

By default, the package runs a weekly cron job to cleanup old
snapshots. (Defaults to 90d, but you can configure that in
APT::Snapshots::MaxAge) Alternatively, you can cleanup with the
command yourself. Run `sudo apt-btrfs-snapshot list`, and then `sudo
apt-btrfs-snapshot delete `

~ Noah

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Menion]

[sudo] password for menion:
ID  gen top level   path
--  --- -   
257 600627  5   /@
258 600626  5   /@home
296 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55
297 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:30:08
298 599489  5
/@apt-snapshot-release-upgrade-bionic-2018-08-27_15:33:30

So, there are snapshots, right? The time stamp is when I have launched
do-release-upgrade, but it didn't ask anything about snapshot, neither
I asked for it.
During the do-release-upgrade I got some issues due to the (very) bad
behaviour of the script in remote terminal, then I have fixed
everything manually and now the filesystem is operational in bionic
version
If it is confirmed, how can I remove the unwanted snapshot, keeping
the current "visible" filesystem contents
Sorry, I am still learning BTRFS and I would like to avoid mistakes
Bye
Il giorno mar 28 ago 2018 alle ore 15:47 Chris Murphy
 ha scritto:
>
> On Tue, Aug 28, 2018 at 3:34 AM, Menion  wrote:
> > Hi all
> > I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
> > 4.17.2 with btrfsprogs 4.17.0
> > The root filesystem is BTRFS single created by the Ubuntu Xenial
> > installer (so on kernel 4.4.0) on an internal mmc, located in
> > /dev/mmcblk0p3
> > After the upgrade I have cleaned apt cache and checked the free space,
> > the results were odd, following some checks (shrinked), followed by
> > more comments:
>
> Do you know if you're using Timeshift? I'm not sure if it's enabled by
> default on Ubuntu when using Btrfs, but you may have snapshots.
>
> 'sudo btrfs sub list -at /'
>
> That should show all subvolumes (includes snapshots).
>
>
>
> > [48479.254106] BTRFS info (device mmcblk0p3): 17 enospc errors during 
> > balance
>
> Probably soft enospc errors it was able to work around.
>
>
> --
> Chris Murphy

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Chris Murphy]

On Tue, Aug 28, 2018 at 3:34 AM, Menion  wrote:
> Hi all
> I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
> 4.17.2 with btrfsprogs 4.17.0
> The root filesystem is BTRFS single created by the Ubuntu Xenial
> installer (so on kernel 4.4.0) on an internal mmc, located in
> /dev/mmcblk0p3
> After the upgrade I have cleaned apt cache and checked the free space,
> the results were odd, following some checks (shrinked), followed by
> more comments:

Do you know if you're using Timeshift? I'm not sure if it's enabled by
default on Ubuntu when using Btrfs, but you may have snapshots.

'sudo btrfs sub list -at /'

That should show all subvolumes (includes snapshots).



> [48479.254106] BTRFS info (device mmcblk0p3): 17 enospc errors during balance

Probably soft enospc errors it was able to work around.


-- 
Chris Murphy

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Qu Wenruo]

On 2018/8/28 下午9:07, Menion wrote:
> Ok, thanks for your replay
> This is a root FS, how can I defragment it?

If it's a rootfs, then it's a little strange.

Normally package manager should overwrite the whole file during package
update transaction, thus the extent booking doesn't happen as frequent
as other workload.

One solution is booting from other device, and try defrag.

BTW, is there any snapshots in the fs?

One way to determine it is by btrfs ins dump-tree:

# btrfs ins dump-tree -t root 

Above command can be executed on mounted device, and above root tree
dump doesn't contain confidential info except subvolume names.

If there is no extra subvolumes at all, then try to defrag may make sense.

Thanks,
Qu

> If I try to launch it I get this output:
> 
> menion@Menionubuntu:~$ sudo btrfs filesystem defragment -r /
> ERROR: defrag failed on /bin/bash: Text file busy
> ERROR: defrag failed on /bin/dash: Text file busy
> ERROR: defrag failed on /bin/btrfs: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd-journald: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd-logind: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd-resolved: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd-timesyncd: Text file busy
> ERROR: defrag failed on /lib/systemd/systemd-udevd: Text file busy
> ERROR: defrag failed on /lib/x86_64-linux-gnu/ld-2.27.so: Text file busy
> 
> Bye
> Il giorno mar 28 ago 2018 alle ore 13:54 Qu Wenruo
>  ha scritto:
>>
>>
>>
>> On 2018/8/28 下午5:34, Menion wrote:
>>> Hi all
>>> I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
>>> 4.17.2 with btrfsprogs 4.17.0
>>> The root filesystem is BTRFS single created by the Ubuntu Xenial
>>> installer (so on kernel 4.4.0) on an internal mmc, located in
>>> /dev/mmcblk0p3
>>> After the upgrade I have cleaned apt cache and checked the free space,
>>> the results were odd, following some checks (shrinked), followed by
>>> more comments:
>>>
>>> root@Menionubuntu:/home/menion# df -h
>>> Filesystem  Size  Used Avail Use% Mounted on
>>> ...
>>> /dev/mmcblk0p3   28G   24G  2.7G  90% /
>>>
>>> root@Menionubuntu:/home/menion# btrfs fi usage /usr
>>> Overall:
>>> Device size:  27.07GiB
>>> Device allocated: 25.28GiB
>>> Device unallocated:1.79GiB
>>> Device missing:  0.00B
>>> Used: 23.88GiB
>>> Free (estimated):  2.69GiB  (min: 2.69GiB)
>>> Data ratio:   1.00
>>> Metadata ratio:   1.00
>>> Global reserve:   72.94MiB  (used: 0.00B)
>>>
>>> Data,single: Size:24.00GiB, Used:23.10GiB
>>>/dev/mmcblk0p3 24.00GiB
>>>
>>> Metadata,single: Size:1.25GiB, Used:801.97MiB
>>>/dev/mmcblk0p3  1.25GiB
>>>
>>> System,single: Size:32.00MiB, Used:16.00KiB
>>>/dev/mmcblk0p3 32.00MiB
>>>
>>> Unallocated:
>>>/dev/mmcblk0p3  1.79GiB
>>>
>>> root@Menionubuntu:/home/menion# btrfs fi df /mnt
>>> Data, single: total=24.00GiB, used=23.10GiB
>>> System, single: total=32.00MiB, used=16.00KiB
>>> Metadata, single: total=1.25GiB, used=801.92MiB
>>> GlobalReserve, single: total=72.89MiB, used=0.00B
>>>
>>> The different ways to check the free space are coherent, but if I
>>> check the directories usage on root, surprise:
>>>
>>> root@Menionubuntu:/home/menion# du -x -s -h /*
>>> 17M /bin
>>> 189M/boot
>>> 36K /dead.letter
>>> 0   /dev
>>> 18M /etc
>>> 6.1G/home
>>> 4.0K/initrd.img
>>> 4.0K/initrd.img.old
>>> 791M/lib
>>> 8.3M/lib64
>>> 0   /media
>>> 4.0K/mnt
>>> 0   /opt
>>> du: cannot access '/proc/24660/task/24660/fd/3': No such file or directory
>>> du: cannot access '/proc/24660/task/24660/fdinfo/3': No such file or 
>>> directory
>>> du: cannot access '/proc/24660/fd/3': No such file or directory
>>> du: cannot access '/proc/24660/fdinfo/3': No such file or directory
>>> 0   /proc
>>> 2.9M/root
>>> 2.9M/run
>>> 17M /sbin
>>> 4.0K/snap
>>> 0   /srv
>>> 0   /sys
>>> 0   /tmp
>>> 6.1G/usr
>>> 2.0G/var
>>> 4.0K/vmlinuz
>>> 4.0K/vmlinuz.old
>>> 4.0K/webmin-setup.out
>>>
>>> The computed usage is 15Gb which is what I expected, so there are 9Gb
>>> lost somewhere.
>>> I have run scrub and then full balance with:
>>
>> I think this is related to btrfs CoW and extent booking.
>>
>> One simple example would be:
>>
>> xfs_io -f -c "pwrite 0 128k" -c "sync" -c "pwrite 0 64K" \
>> /mnt/btrfs/file1
>>
>> The result "/mnt/btrfs/file1" will only be sized 128K in du, but it
>> on-disk usage is 128K + 64K.
>>
>> The first 128K is the data written by the first "pwrite" command, it
>> caused a full 128K extent on disk.
>> Then the 2nd pwrite command also created a new 64K extent, 

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Menion]

Ok, thanks for your replay
This is a root FS, how can I defragment it?
If I try to launch it I get this output:

menion@Menionubuntu:~$ sudo btrfs filesystem defragment -r /
ERROR: defrag failed on /bin/bash: Text file busy
ERROR: defrag failed on /bin/dash: Text file busy
ERROR: defrag failed on /bin/btrfs: Text file busy
ERROR: defrag failed on /lib/systemd/systemd: Text file busy
ERROR: defrag failed on /lib/systemd/systemd-journald: Text file busy
ERROR: defrag failed on /lib/systemd/systemd-logind: Text file busy
ERROR: defrag failed on /lib/systemd/systemd-resolved: Text file busy
ERROR: defrag failed on /lib/systemd/systemd-timesyncd: Text file busy
ERROR: defrag failed on /lib/systemd/systemd-udevd: Text file busy
ERROR: defrag failed on /lib/x86_64-linux-gnu/ld-2.27.so: Text file busy

Bye
Il giorno mar 28 ago 2018 alle ore 13:54 Qu Wenruo
 ha scritto:
>
>
>
> On 2018/8/28 下午5:34, Menion wrote:
> > Hi all
> > I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
> > 4.17.2 with btrfsprogs 4.17.0
> > The root filesystem is BTRFS single created by the Ubuntu Xenial
> > installer (so on kernel 4.4.0) on an internal mmc, located in
> > /dev/mmcblk0p3
> > After the upgrade I have cleaned apt cache and checked the free space,
> > the results were odd, following some checks (shrinked), followed by
> > more comments:
> >
> > root@Menionubuntu:/home/menion# df -h
> > Filesystem  Size  Used Avail Use% Mounted on
> > ...
> > /dev/mmcblk0p3   28G   24G  2.7G  90% /
> >
> > root@Menionubuntu:/home/menion# btrfs fi usage /usr
> > Overall:
> > Device size:  27.07GiB
> > Device allocated: 25.28GiB
> > Device unallocated:1.79GiB
> > Device missing:  0.00B
> > Used: 23.88GiB
> > Free (estimated):  2.69GiB  (min: 2.69GiB)
> > Data ratio:   1.00
> > Metadata ratio:   1.00
> > Global reserve:   72.94MiB  (used: 0.00B)
> >
> > Data,single: Size:24.00GiB, Used:23.10GiB
> >/dev/mmcblk0p3 24.00GiB
> >
> > Metadata,single: Size:1.25GiB, Used:801.97MiB
> >/dev/mmcblk0p3  1.25GiB
> >
> > System,single: Size:32.00MiB, Used:16.00KiB
> >/dev/mmcblk0p3 32.00MiB
> >
> > Unallocated:
> >/dev/mmcblk0p3  1.79GiB
> >
> > root@Menionubuntu:/home/menion# btrfs fi df /mnt
> > Data, single: total=24.00GiB, used=23.10GiB
> > System, single: total=32.00MiB, used=16.00KiB
> > Metadata, single: total=1.25GiB, used=801.92MiB
> > GlobalReserve, single: total=72.89MiB, used=0.00B
> >
> > The different ways to check the free space are coherent, but if I
> > check the directories usage on root, surprise:
> >
> > root@Menionubuntu:/home/menion# du -x -s -h /*
> > 17M /bin
> > 189M/boot
> > 36K /dead.letter
> > 0   /dev
> > 18M /etc
> > 6.1G/home
> > 4.0K/initrd.img
> > 4.0K/initrd.img.old
> > 791M/lib
> > 8.3M/lib64
> > 0   /media
> > 4.0K/mnt
> > 0   /opt
> > du: cannot access '/proc/24660/task/24660/fd/3': No such file or directory
> > du: cannot access '/proc/24660/task/24660/fdinfo/3': No such file or 
> > directory
> > du: cannot access '/proc/24660/fd/3': No such file or directory
> > du: cannot access '/proc/24660/fdinfo/3': No such file or directory
> > 0   /proc
> > 2.9M/root
> > 2.9M/run
> > 17M /sbin
> > 4.0K/snap
> > 0   /srv
> > 0   /sys
> > 0   /tmp
> > 6.1G/usr
> > 2.0G/var
> > 4.0K/vmlinuz
> > 4.0K/vmlinuz.old
> > 4.0K/webmin-setup.out
> >
> > The computed usage is 15Gb which is what I expected, so there are 9Gb
> > lost somewhere.
> > I have run scrub and then full balance with:
>
> I think this is related to btrfs CoW and extent booking.
>
> One simple example would be:
>
> xfs_io -f -c "pwrite 0 128k" -c "sync" -c "pwrite 0 64K" \
> /mnt/btrfs/file1
>
> The result "/mnt/btrfs/file1" will only be sized 128K in du, but it
> on-disk usage is 128K + 64K.
>
> The first 128K is the data written by the first "pwrite" command, it
> caused a full 128K extent on disk.
> Then the 2nd pwrite command also created a new 64K extent, which is the
> default data CoW behavior.
> The first half of the original 128K extent is not used by anyone, but it
> still takes space.
>
> Above btrfs extent booking behavior could cause a lot of wasted space
> even there is only one single subvolume without any snapshot.
>
> In that case, instead of balance, defrag should be your friend to free
> up some space.
>
> Thanks,
> Qu
>
> >
> > btrfs scrub start /
> > btrfs balance start /
> > The balance freed 100Mb of space, it was running in background so I
> > have checked dmesg when "btrfs balance status" said that was completed
> >
> > dmesg of balance:
> >
> > [47264.250141] BTRFS info (device mmcblk0p3): relocating block group
> > 37154193408 

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Qu Wenruo]

On 2018/8/28 下午5:34, Menion wrote:
> Hi all
> I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
> 4.17.2 with btrfsprogs 4.17.0
> The root filesystem is BTRFS single created by the Ubuntu Xenial
> installer (so on kernel 4.4.0) on an internal mmc, located in
> /dev/mmcblk0p3
> After the upgrade I have cleaned apt cache and checked the free space,
> the results were odd, following some checks (shrinked), followed by
> more comments:
> 
> root@Menionubuntu:/home/menion# df -h
> Filesystem  Size  Used Avail Use% Mounted on
> ...
> /dev/mmcblk0p3   28G   24G  2.7G  90% /
> 
> root@Menionubuntu:/home/menion# btrfs fi usage /usr
> Overall:
> Device size:  27.07GiB
> Device allocated: 25.28GiB
> Device unallocated:1.79GiB
> Device missing:  0.00B
> Used: 23.88GiB
> Free (estimated):  2.69GiB  (min: 2.69GiB)
> Data ratio:   1.00
> Metadata ratio:   1.00
> Global reserve:   72.94MiB  (used: 0.00B)
> 
> Data,single: Size:24.00GiB, Used:23.10GiB
>/dev/mmcblk0p3 24.00GiB
> 
> Metadata,single: Size:1.25GiB, Used:801.97MiB
>/dev/mmcblk0p3  1.25GiB
> 
> System,single: Size:32.00MiB, Used:16.00KiB
>/dev/mmcblk0p3 32.00MiB
> 
> Unallocated:
>/dev/mmcblk0p3  1.79GiB
> 
> root@Menionubuntu:/home/menion# btrfs fi df /mnt
> Data, single: total=24.00GiB, used=23.10GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=1.25GiB, used=801.92MiB
> GlobalReserve, single: total=72.89MiB, used=0.00B
> 
> The different ways to check the free space are coherent, but if I
> check the directories usage on root, surprise:
> 
> root@Menionubuntu:/home/menion# du -x -s -h /*
> 17M /bin
> 189M/boot
> 36K /dead.letter
> 0   /dev
> 18M /etc
> 6.1G/home
> 4.0K/initrd.img
> 4.0K/initrd.img.old
> 791M/lib
> 8.3M/lib64
> 0   /media
> 4.0K/mnt
> 0   /opt
> du: cannot access '/proc/24660/task/24660/fd/3': No such file or directory
> du: cannot access '/proc/24660/task/24660/fdinfo/3': No such file or directory
> du: cannot access '/proc/24660/fd/3': No such file or directory
> du: cannot access '/proc/24660/fdinfo/3': No such file or directory
> 0   /proc
> 2.9M/root
> 2.9M/run
> 17M /sbin
> 4.0K/snap
> 0   /srv
> 0   /sys
> 0   /tmp
> 6.1G/usr
> 2.0G/var
> 4.0K/vmlinuz
> 4.0K/vmlinuz.old
> 4.0K/webmin-setup.out
> 
> The computed usage is 15Gb which is what I expected, so there are 9Gb
> lost somewhere.
> I have run scrub and then full balance with:

I think this is related to btrfs CoW and extent booking.

One simple example would be:

xfs_io -f -c "pwrite 0 128k" -c "sync" -c "pwrite 0 64K" \
/mnt/btrfs/file1

The result "/mnt/btrfs/file1" will only be sized 128K in du, but it
on-disk usage is 128K + 64K.

The first 128K is the data written by the first "pwrite" command, it
caused a full 128K extent on disk.
Then the 2nd pwrite command also created a new 64K extent, which is the
default data CoW behavior.
The first half of the original 128K extent is not used by anyone, but it
still takes space.

Above btrfs extent booking behavior could cause a lot of wasted space
even there is only one single subvolume without any snapshot.

In that case, instead of balance, defrag should be your friend to free
up some space.

Thanks,
Qu

> 
> btrfs scrub start /
> btrfs balance start /
> The balance freed 100Mb of space, it was running in background so I
> have checked dmesg when "btrfs balance status" said that was completed
> 
> dmesg of balance:
> 
> [47264.250141] BTRFS info (device mmcblk0p3): relocating block group
> 37154193408 flags system
> [47264.592082] BTRFS info (device mmcblk0p3): relocating block group
> 36046897152 flags data
> [47271.499809] BTRFS info (device mmcblk0p3): found 73 extents
> [47272.329921] BTRFS info (device mmcblk0p3): found 60 extents
> [47272.471059] BTRFS info (device mmcblk0p3): relocating block group
> 35778461696 flags metadata
> [47280.530041] BTRFS info (device mmcblk0p3): found 3199 extents
> [47280.735667] BTRFS info (device mmcblk0p3): relocating block group
> 34704719872 flags data
> [47301.460523] BTRFS info (device mmcblk0p3): relocating block group
> 37221302272 flags data
> [47306.038404] BTRFS info (device mmcblk0p3): found 5 extents
> [47306.481371] BTRFS info (device mmcblk0p3): found 5 extents
> [47306.673135] BTRFS info (device mmcblk0p3): relocating block group
> 37187747840 flags system
> [47306.874874] BTRFS info (device mmcblk0p3): found 1 extents
> [47307.073288] BTRFS info (device mmcblk0p3): relocating block group
> 34704719872 flags data
> [47371.059074] BTRFS info (device mmcblk0p3): found 16258 extents
> [47388.191208] BTRFS info (device mmcblk0p3): found 

14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

[Menion]

Hi all
I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel
4.17.2 with btrfsprogs 4.17.0
The root filesystem is BTRFS single created by the Ubuntu Xenial
installer (so on kernel 4.4.0) on an internal mmc, located in
/dev/mmcblk0p3
After the upgrade I have cleaned apt cache and checked the free space,
the results were odd, following some checks (shrinked), followed by
more comments:

root@Menionubuntu:/home/menion# df -h
Filesystem  Size  Used Avail Use% Mounted on
...
/dev/mmcblk0p3   28G   24G  2.7G  90% /

root@Menionubuntu:/home/menion# btrfs fi usage /usr
Overall:
Device size:  27.07GiB
Device allocated: 25.28GiB
Device unallocated:1.79GiB
Device missing:  0.00B
Used: 23.88GiB
Free (estimated):  2.69GiB  (min: 2.69GiB)
Data ratio:   1.00
Metadata ratio:   1.00
Global reserve:   72.94MiB  (used: 0.00B)

Data,single: Size:24.00GiB, Used:23.10GiB
   /dev/mmcblk0p3 24.00GiB

Metadata,single: Size:1.25GiB, Used:801.97MiB
   /dev/mmcblk0p3  1.25GiB

System,single: Size:32.00MiB, Used:16.00KiB
   /dev/mmcblk0p3 32.00MiB

Unallocated:
   /dev/mmcblk0p3  1.79GiB

root@Menionubuntu:/home/menion# btrfs fi df /mnt
Data, single: total=24.00GiB, used=23.10GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.25GiB, used=801.92MiB
GlobalReserve, single: total=72.89MiB, used=0.00B

The different ways to check the free space are coherent, but if I
check the directories usage on root, surprise:

root@Menionubuntu:/home/menion# du -x -s -h /*
17M /bin
189M/boot
36K /dead.letter
0   /dev
18M /etc
6.1G/home
4.0K/initrd.img
4.0K/initrd.img.old
791M/lib
8.3M/lib64
0   /media
4.0K/mnt
0   /opt
du: cannot access '/proc/24660/task/24660/fd/3': No such file or directory
du: cannot access '/proc/24660/task/24660/fdinfo/3': No such file or directory
du: cannot access '/proc/24660/fd/3': No such file or directory
du: cannot access '/proc/24660/fdinfo/3': No such file or directory
0   /proc
2.9M/root
2.9M/run
17M /sbin
4.0K/snap
0   /srv
0   /sys
0   /tmp
6.1G/usr
2.0G/var
4.0K/vmlinuz
4.0K/vmlinuz.old
4.0K/webmin-setup.out

The computed usage is 15Gb which is what I expected, so there are 9Gb
lost somewhere.
I have run scrub and then full balance with:

btrfs scrub start /
btrfs balance start /
The balance freed 100Mb of space, it was running in background so I
have checked dmesg when "btrfs balance status" said that was completed

dmesg of balance:

[47264.250141] BTRFS info (device mmcblk0p3): relocating block group
37154193408 flags system
[47264.592082] BTRFS info (device mmcblk0p3): relocating block group
36046897152 flags data
[47271.499809] BTRFS info (device mmcblk0p3): found 73 extents
[47272.329921] BTRFS info (device mmcblk0p3): found 60 extents
[47272.471059] BTRFS info (device mmcblk0p3): relocating block group
35778461696 flags metadata
[47280.530041] BTRFS info (device mmcblk0p3): found 3199 extents
[47280.735667] BTRFS info (device mmcblk0p3): relocating block group
34704719872 flags data
[47301.460523] BTRFS info (device mmcblk0p3): relocating block group
37221302272 flags data
[47306.038404] BTRFS info (device mmcblk0p3): found 5 extents
[47306.481371] BTRFS info (device mmcblk0p3): found 5 extents
[47306.673135] BTRFS info (device mmcblk0p3): relocating block group
37187747840 flags system
[47306.874874] BTRFS info (device mmcblk0p3): found 1 extents
[47307.073288] BTRFS info (device mmcblk0p3): relocating block group
34704719872 flags data
[47371.059074] BTRFS info (device mmcblk0p3): found 16258 extents
[47388.191208] BTRFS info (device mmcblk0p3): found 16094 extents
[47388.985462] BTRFS info (device mmcblk0p3): relocating block group
31215058944 flags metadata
[47439.164167] BTRFS info (device mmcblk0p3): found 7378 extents
[47440.163793] BTRFS info (device mmcblk0p3): relocating block group
30141317120 flags data
[47593.239048] BTRFS info (device mmcblk0p3): found 15636 extents
[47618.389357] BTRFS info (device mmcblk0p3): found 15634 extents
[47620.020122] BTRFS info (device mmcblk0p3): relocating block group
29012000768 flags data
[47637.708444] BTRFS info (device mmcblk0p3): found 1154 extents
[47639.757342] BTRFS info (device mmcblk0p3): found 1154 extents
[47640.375483] BTRFS info (device mmcblk0p3): relocating block group
27938258944 flags data
[47743.312441] BTRFS info (device mmcblk0p3): found 17009 extents
[47756.928461] BTRFS info (device mmcblk0p3): found 17005 extents
[47757.607346] BTRFS info (device mmcblk0p3): relocating block group
9416212480 flags metadata
[47825.819449] BTRFS info (device mmcblk0p3): found 11503 extents
[47826.465926] BTRFS info (device mmcblk0p3): relocating