Toralf Förster posted on Mon, 22 Dec 2014 15:30:00 +0100 as excerpted:
> I created within a x86 KVM guest at a tmpfs file system under 3.19.0-rc1
> a 257 MB file, created within that abtrfs file system and run the fuzzer
> trinity using that fs for its victim files:
>
> $ mkdir /mnt/ramdisk/btrfs; truncate -s 257M /mnt/ramdisk/btrfs.fs;
> /sbin/mkfs.btrfs /mnt/ramdisk/btrfs.fs; sudo su -c "mount -o
> loop,compress=lzo /mnt/ramdisk/btrfs.fs /mnt/ramdisk/btrfs; chmod 777
> /mnt/ramdisk/btrfs"
> SMALL VOLUME: forcing mixed metadata/data groups
>
> WARNING! - Btrfs v3.14.2 IS EXPERIMENTAL WARNING! - see
> http://btrfs.wiki.kernel.org before using
FWIW, that's an old userspace. Current btrfs userspace is btrfs-progs
v3.17.3, last I checked (last nite). Your kernel is current, at least,
which is the big thing for operational checks.
> Turning ON incompat feature 'mixed-bg':
> mixed data and metadata block groups
> Turning ON incompat feature 'extref':
> increased hardlink limit per file to 65536
> Created a data/metadata chunk of size 8388608 fs
> created label (null) on /mnt/ramdisk/btrfs.fs
> nodesize 4096 leafsize 4096 sectorsize 4096 size 257.00MiB
> Btrfs v3.14.2
[Whitespacing to make the below readable.]
> $ D=/mnt/ramdisk/btrfs
> while [[ : ]]; do
> cd ~
> sudo rm -rf $D/t3 &&
> mkdir $D/t3 ||
> break
> cd $D/t3
> mkdir -p v1/v2
> for i in $(seq 0 99); do
> touch v1/v2/f$i
> mkdir v1/v2/d$i
> done
> trinity -C 2 -N 100000 -V $D/t3/v1/v2 -q
> echo
> echo " done"
> echo
> sleep 4
> done
>
> After a while I got :
>
> $ sudo rm -rf /mnt/ramdisk/btrfs/t3/
> rm: cannot remove ‘/mnt/ramdisk/btrfs/t3/v1/v2/d63’:
> No space left on device
> rm: cannot remove ‘/mnt/ramdisk/btrfs/t3/v1/v2/d10’:
> No space left on device
> rm: cannot remove ‘/mnt/ramdisk/btrfs/t3/v1/v2/f98’:
> No space left on device
[etc]
So where's the unexpected problem?
Btrfs is a COW-based (Copy On Write) filesystem, including metadata,
which means to do an rm requires available space to write the new copy of
the directory without the rmed file, before it deletes the existing copy
of the directory that has the existing file you're trying to rm.
Apparently you filled all the space up, so there's no place to write that
new copy of the directory, which means the rm fails.
If you had run the appropriate btrfs commands as suggested on the wiki
[1], namely the output from...
btrfs fi show /mnt/ramdisk/btrfs
and...
btrfs fi df /mnt/ramdisk/btrfs
..., then we'd have confirmation of the problem. Presumably the show
output will list the same value for total and used space on the device
line, meaning all available space has been chunk-allocated, and the
(btrfs) df output will list most of that as Data+Metadata (due to the
mixed-bg chunk default because the filesystem is under 1 GiB in size,
otherwise it'd be separate data and metadata chunks), DUP mode
(duplicated so it'll hold less than half that due to the system and
global-reserve reservations), again with total and used reasonably near
equivalent.
FWIW, here's the output for my (NOT full) 256 MiB /bt (basically /boot,
$>> is the last line of my prompt):
$>>sudo btrfs fi sh /bt
Label: 'bt0238gcn1+35l0' uuid: d6539322-0834-4eeb-928d-a13eb32dcbb2
Total devices 1 FS bytes used 67.06MiB
devid 1 size 256.00MiB used 192.00MiB path /dev/sda3
Btrfs v3.17.3
$>>sudo btrfs fi df /bt
System, DUP: total=16.00MiB, used=4.00KiB
Data+Metadata, DUP: total=80.00MiB, used=67.06MiB
GlobalReserve, single: total=4.00MiB, used=0.00B
$>>
Total Chunk-allocation (from show): 192 MiB of 256 MiB used (so 64 MiB
still unallocated).
Of that 192 MiB (from (btrfs) df):
System: 16 MiB (dup mode so double):
32 MiB
Data+Metadata: 80 MiB allocated (dup mode so doubled):
160 MiB
160+32=192 MiB allocated.
(The global reserve will probably show up as unknown on yours, given your
older userspace with a new kernel. That obviously isn't included in the
total given in show.)
Of that 80 MiB data+metadata chunk allocation, 67.06 MiB is actually used
by data and metadata (matching between show and df), so there's just
under 13 MiB of free space in (each copy of the pair-copies of) the
current mixed-mode chunks before another pair of data+metadata chunks
must be allocated.
Yours will probably look much closer to something like this:
Show:
Total devices 1 FS bytes used 240 MiB
devid 1 size 257 MiB used 257 MiB
Df:
System, DUP: total=16.00MiB, Used=x.xxKiB
Data+Metadata, DUP: total=240.00MiB, Used=241.00MiB
Unknown: .....
IOW: All space chunk-allocated, all data/metadata chunk-space used. No
place to put the new copy of the directory table with that rm done, so it
can't be written.
However, I've actually run into a corner-case occasionally here (when I
redo my /boot, which I've not done for a couple kernels so with a bit of
luck the bug is gone now) where there was still unallocated space that
could be allocated to new chunks, but btrfs was for some reason failing
to actually allocate those new chunks, and the existing chunks being
full, I was getting ENOSPC. It's /possible/ you're running into that bug
as well. In my case, I was able to copy over (from backup) certain
(small) sized files but not others (bigger), and by trying different copy
order, etc, I eventually got it to trigger a chunk-allocate which gave me
the room to copy the others. I'm not sure how you'd trigger the
allocation if getting the ENOSPC on a rm, however.
How to get out of that bind? Try /truncating/ one or more files. This
can often be done where a normal rm ENOSPCs, with a bit of luck and
perhaps a few more file truncates, freeing enough space to do that rm
properly.
echo > /path/to/file
The details are on the wiki (see the questions on space in the FAQ).
Which I'd suggest you spend some time reading if you're going to be
dealing with btrfs, as it'll answer a bunch of questions. As I said, an
ENOSPC on file rm, when the filesystem is full, isn't entirely unusual or
unexpected because btrfs is COW-based, and thus requires space to write
the new copy when it changes something, including when it does a rm. If
you're entirely out of space, that rm will fail, and that's entirely
expected. The idea is not to run THAT close to out of space in the first
place, and in the event that you do, knowing how to get out of the hole
you dug yourself into, by using truncate or temporarily adding a device.
It's on the wiki! =:^)
---
[1] Btrfs wiki home page, suitable for bookmarking or memorizing:
https://btrfs.wiki.kernel.org
What info to provide when asking a support question? See the bottom
section of the page here:
https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
