nilfs2 weird issue - snapshots are gone, cleanerd not running

[Piotr Szymaniak]

Hi.

I've upgraded nilfs-utils (running Gentoo) on 29 july. Today I ran out
of space on my / and found that nilfs_cleanerd isn't working. When I
start it from the command line it exits instantly. Also, all previous
checkpoints on / (also on two other mountpoints on different machine)
are gone.

What I did? Downgraded nilfs-utils to 2.1.1, remounted mountpoints. On
the second machine it's runnig fine (cleaned _all_ checkpoints), on the
first one with disk space issue it exits just like 2.1.3.

Here are some fs details. Machine with disk space issues, rootfs:
    CNO        DATE     TIME  MODE  FLG     NBLKINC       ICNT
 147688  2012-07-09 08:38:14   cp    -        11075     242915
 147689  2012-07-09 08:38:14   cp    -           60     242895
(…)
 148999  2012-07-09 09:13:46   cp    -           60     242888
 149000  2012-07-09 09:19:45   cp    -           44     242888

Filesystem      Size  Used Avail Use% Mounted on
rootfs           24G   13G   11G  56% /

mount shows:
/dev/sda2 on / type nilfs2 (rw,noatime,nodiratime,gcpid=15356)

There's no nilfs_cleanerd with pid 15356.


Second machine rootfs:
   CNO        DATE     TIME  MODE  FLG     NBLKINC       ICNT
 92246  2012-07-09 08:16:58   cp    -          118      44669
(…)
 92439  2012-07-09 09:19:14   cp    -           29      44668
 92440  2012-07-09 09:19:46   cp    -           33      44668

Filesystem         Size  Used Avail Use% Mounted on
rootfs             3.7G  888M  2.6G  26% /

(it should be around 3G used)

Second machine second mountpoint:
   CNO        DATE     TIME  MODE  FLG     NBLKINC       ICNT
  1496  2012-07-09 03:31:23   cp    -         8837     132766
  1497  2012-07-09 03:31:26   cp    -          468     132766
  1498  2012-07-09 03:41:27   cp    -         1474     132765

(this fs should containt *all* 1498 checkpoints)

Filesystem         Size  Used Avail Use% Mounted on
/dev/dm-2          117G   58G   54G  76% /mnt/home_backup

(in this one it should be around 100G of used space)

mount:
/dev/dm-2 on /mnt/home_backup type nilfs2 (rw,gcpid=13135)
/dev/sda3 on / type nilfs2 (rw,noatime,nodiratime,gcpid=1363)

Both cleaners running (the second mountpoint - /mnt/home_backup - is under
heavy load and I suppose it will end with around 20G used space).

Where to go from this point? How to debug nilfs_cleanerd issue?


Piotr Szymaniak.
-- 
 Wybudowaliśmy dla siebie klatkę - cywilizację - gdyż byliśmy zdolni do
myślenia,  a  teraz musimy myśleć,  ponieważ  jesteśmy  zamknięci w tej
klatce.
  -- William Wharton, "Birdy"

signature.asc
Description: Digital signature