Re: Simple fsck.aufs python script

[OmegaPhil]

Hi Alex,

Thanks for your changes and responses on
https://gist.github.com/Alex131089/560a1386dd756059e21b - it looks like
I can't subscribe to gists and I don't get emailed when mentioned, just
bringing to your attention that I did see your messages and did respond :)

I have used the script a bit more, and have fixed some bugs - see
attached patches - basically some error code reporting was wrong, and
the script treated a symlinked directory as a directory, subsequently
calling rmtree on it and failing as a symlink is a file.

I have 475 examples of the 'File has whiteout but is visible ?!'
warning/error on my main aufs volume. In my case I have 4 writable
branches, looking into the first example the whiteout exists on the 2nd
branch and the whiteouted file exists both on the 1st and 3rd branches -
therefore the whiteout only hides the 3rd branch file but lets the 1st
branch one through.

I think this is valid aufs behaviour, but I agree its invalid for me
(infact the whole whiteout idea is since I'm using 4 writeable branches
to represent one 'whole' writeable disk, if a file gets deleted it
should be purged everywhere). Shall I add a switch to go ahead and
delete visible files (and then the whiteout) when they 'should' have
been whiteouted from this perspective?
From eb24fdc4ce45496adca0eb3f04ed5db528076fa7 Mon Sep 17 00:00:00 2001
From: OmegaPhil 
Date: Sun, 17 Jul 2016 12:39:51 +0100
Subject: [PATCH 1/2] Error objects do not have message attributes

---
 fsck.aufs.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fsck.aufs.py b/fsck.aufs.py
index 2564932..6181350 100755
--- a/fsck.aufs.py
+++ b/fsck.aufs.py
@@ -70,9 +70,9 @@ def main():
 		fsck.start_checks()
 		
 	except ValueError as e:
-		print('ValueError: {}'.format(e.message))
+		print('ValueError: {}'.format(e))
 	except EnvironmentError as e:
-		print('EnvironmentError: {}'.format(e.message))
+		print('EnvironmentError: {}'.format(e))
 	except IOError as e:
 		print('IOError: {}'.format(e.strerror))
 
@@ -445,7 +445,7 @@ class AufsCheck:
 		print('-> {} successfully deleted'.format(fullpath))
 except OSError as e:
 	remove_whiteouted_success = False
-	print(' /!\\ Deletion of {} failed : '.format(fullpath, e.message))
+	print(' /!\\ Deletion of {} failed : '.format(fullpath, e))
 		# Remove the whiteout if it become unnecessarily (ie. we removed all path whiteouted)
 		if remove_whiteouted_success:
 			if not dry_run:
-- 
2.8.1

From 1d89d8c7348deed8583c0b7ad670eae8de2fc356 Mon Sep 17 00:00:00 2001
From: OmegaPhil 
Date: Thu, 21 Jul 2016 18:57:49 +0100
Subject: [PATCH 2/2] Fix deletion of symlinked directories, fix some error
 reporting

---
 fsck.aufs.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/fsck.aufs.py b/fsck.aufs.py
index 6181350..f7b3af1 100755
--- a/fsck.aufs.py
+++ b/fsck.aufs.py
@@ -437,7 +437,8 @@ class AufsCheck:
 try:
 	if not dry_run:
 		fullpath = os.path.join(branch, filepath)
-		if os.path.isdir(fullpath):
+		# Make sure symlinks to directories are treated as normal files
+		if not os.path.islink(fullpath) and os.path.isdir(fullpath):
 			shutil.rmtree(fullpath)
 		else:
 			os.remove(fullpath)
@@ -445,7 +446,7 @@ class AufsCheck:
 		print('-> {} successfully deleted'.format(fullpath))
 except OSError as e:
 	remove_whiteouted_success = False
-	print(' /!\\ Deletion of {} failed : '.format(fullpath, e))
+	print(' /!\\ Deletion of {} failed : {} '.format(fullpath, e))
 		# Remove the whiteout if it become unnecessarily (ie. we removed all path whiteouted)
 		if remove_whiteouted_success:
 			if not dry_run:
@@ -453,7 +454,7 @@ class AufsCheck:
 			if self.args.verbose:
 print('-> {} successfully deleted'.format(whiteout_path))
 		else:
-			print(' -> {} left because of previours error'.format(whiteout_path))
+			print(' -> {} left because of previous error'.format(whiteout_path))
 	
 	
 	@staticmethod
-- 
2.8.1



signature.asc
Description: OpenPGP digital signature
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev

Re: Simple fsck.aufs python script

[Alexandre Levavasseur]

sf...@users.sourceforge.net> wrote:
>
> Hello Alexandre,

Hello Junjiro,

late answer, sorry, I've been busy.

> Alexandre Levavasseur:
> > The result is here : https://gist.github.com/Alex131089/560a1386dd756059e21b
> >  .
> >
> > It does the same as fsck.mergerfs for the permissions (but using another
> > method to find branches), and it checks for useless whiteouts (hoping I'm
> > not misunderstanding the thing). While it's working, it has only been
> > tested on 2 RW branches, the whiteout part has a beginning (I didn't check
> > all possibilities) of support for non-RW but permissions are not RW/RO
> > aware.
>
> There is a script called auchk.
>
> (from README in aufs-util.git)
> --
> o /usr/bin/auchk
>   Similar to generic fsck. Checks whether a branch is healthy or not
>   from aufs's point of view.
> --
>
> (from aufs manual)
> --
> If a sudden accident such like a power failure happens during aufs is
> performing, and regular fsck for branch filesystems is completed after
> the disaster, you need to extra fsck for aufs writable branches. It is
> necessary to check whether the whiteout remains incorrectly or not,
> eg. the real filename and the whiteout for it under the same parent
> directory. If such whiteout remains, aufs cannot handle the file
> correctly.
> To check the consistency from the aufs' point of view, you can use a
> simple shell script called /sbin/auchk. Its purpose is a fsck tool for
> aufs, and it checks the illegal whiteout, the remained
> pseudo\-links and the remained aufs\-temp files. If they are found, the
> utility reports you and asks whether to delete or not.
> It is recommended to execute /sbin/auchk for every writable branch
> filesystem before mounting aufs if the system experienced crash.
> --
>
> Its purpose is very similar to yours, but what to check is different.

the name "fsck.aufs" is probably badly chosen, as it does only "high
level" verifications which are probably not strictly necessary to
aufs.
My goal is not to verify one/each branch, but the branches as a whole
(assuming they are clean to aufs / aufschk), to try to "optimize" what
can be.

> I don't think your check_consistancy() is meaningful because users can
> change the file's mode, uid or gid anytime (and anywhere).

This can cause permission issues with 2 RW branches, doesn't it ?

> To remove the whiteout-ed entries on the lower branch, there is a script
> called aubrsync in the same git tree. Also you may try aumvdown.

But from what I understand, aubrsync will try to move all files from
one branch to the other one, and aumvdown has to be called on each
file where I detect a whiteout ?
I use aufs only to union 2 FS where both have the same priority (both
RW), so aubrsync doens't seems applicable here ?
And for aumvdown I need to detect the whiteouts first.

> > [1]: 2 RW branches on 3.2 kernel (OMV actually), 1 root cron script running
> > a sed -i command => leaves me 1 sed temporary file + it's whiteout each
> > time .. I suppose something's wrong but I can't figure what (or maybe it
> > is/was an aufs issue ?), or I just don't get everything.
>
> It looks more interesting to me.
> Would you post the details following the README file?

Sure :)
But I might have found the problem very recently : OMV applied ACL to
one of the FS, I removed them and since I no more have those
whiteouts. But as I updated the system in the meanwhile, it may also
be that.
ACL aren't supported from what I read ? Is there any warning message ?

> --
> When you have any problems or strange behaviour in aufs, please let me
> know with:
> - /proc/mounts (instead of the output of mount(8))
> - /sys/module/aufs/*
> - /sys/fs/aufs/* (if you have them)
> - /debug/aufs/* (if you have them)
> - linux kernel version
>   if your kernel is not plain, for example modified by distributor,
>   the url where i can download its source is necessary too.
> - aufs version which was printed at loading the module or booting the
>   system, instead of the date you downloaded.
> - configuration (define/undefine CONFIG_AUFS_xxx)
> - kernel configuration or /proc/config.gz (if you have it)
> - behaviour which you think to be incorrect
> - actual operation, reproducible one is better
> - mailto: aufs-users at lists.sourceforge.net
> --

Attached, with the script cron-launched.

>
>
> J. R. Okajima

By the way, I don't mean to add more to this script as it does what I
need, it's only to share it.

Thank you,
-- 
Alexandre L.
root@omv:/boot# cat /export/Medias/\!Dépot/Aria/\!ZeScript.sh
#!/bin/bash

# root@omv:/export/Medias/!Dépot/Aria# ls -la
# 

Re: Simple fsck.aufs python script

[sfjro]

Hello Alexandre,

Alexandre Levavasseur:
> The result is here : https://gist.github.com/Alex131089/560a1386dd756059e21b
>  .
>
> It does the same as fsck.mergerfs for the permissions (but using another
> method to find branches), and it checks for useless whiteouts (hoping I'm
> not misunderstanding the thing). While it's working, it has only been
> tested on 2 RW branches, the whiteout part has a beginning (I didn't check
> all possibilities) of support for non-RW but permissions are not RW/RO
> aware.

There is a script called auchk.

(from README in aufs-util.git)
--
o /usr/bin/auchk
  Similar to generic fsck. Checks whether a branch is healthy or not
  from aufs's point of view.
--

(from aufs manual)
--
If a sudden accident such like a power failure happens during aufs is
performing, and regular fsck for branch filesystems is completed after
the disaster, you need to extra fsck for aufs writable branches. It is
necessary to check whether the whiteout remains incorrectly or not,
eg. the real filename and the whiteout for it under the same parent
directory. If such whiteout remains, aufs cannot handle the file
correctly.
To check the consistency from the aufs' point of view, you can use a
simple shell script called /sbin/auchk. Its purpose is a fsck tool for
aufs, and it checks the illegal whiteout, the remained
pseudo\-links and the remained aufs\-temp files. If they are found, the
utility reports you and asks whether to delete or not.
It is recommended to execute /sbin/auchk for every writable branch
filesystem before mounting aufs if the system experienced crash.
--

Its purpose is very similar to yours, but what to check is different.
I don't think your check_consistancy() is meaningful because users can
change the file's mode, uid or gid anytime (and anywhere).

To remove the whiteout-ed entries on the lower branch, there is a script
called aubrsync in the same git tree. Also you may try aumvdown.


> [1]: 2 RW branches on 3.2 kernel (OMV actually), 1 root cron script running
> a sed -i command => leaves me 1 sed temporary file + it's whiteout each
> time .. I suppose something's wrong but I can't figure what (or maybe it
> is/was an aufs issue ?), or I just don't get everything.

It looks more interesting to me.
Would you post the details following the README file?

--
When you have any problems or strange behaviour in aufs, please let me
know with:
- /proc/mounts (instead of the output of mount(8))
- /sys/module/aufs/*
- /sys/fs/aufs/* (if you have them)
- /debug/aufs/* (if you have them)
- linux kernel version
  if your kernel is not plain, for example modified by distributor,
  the url where i can download its source is necessary too.
- aufs version which was printed at loading the module or booting the
  system, instead of the date you downloaded.
- configuration (define/undefine CONFIG_AUFS_xxx)
- kernel configuration or /proc/config.gz (if you have it)
- behaviour which you think to be incorrect
- actual operation, reproducible one is better
- mailto: aufs-users at lists.sourceforge.net
--


J. R. Okajima

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140

Simple fsck.aufs python script

[Alexandre Levavasseur]

   Hi,Â
   by seeing mergerfs fsck's script
   ([1]https://github.com/trapexit/mergerfs-tools/blob/master/src/fsck.mergerfs
   ), I had the idea to make the same for AUFS (especially since I had issues
   with permissions and have issues with whiteout files[1]).
   The result is here
   : [2]https://gist.github.com/Alex131089/560a1386dd756059e21b .
   It does the same as fsck.mergerfs for the permissions (but using another
   method to find branches), and it checks for useless whiteouts (hoping I'm
   not misunderstanding the thing). While it's working, it has only been tested
   on 2 RW branches, the whiteout part has a beginning (I didn't check all
   possibilities) of support for non-RW but permissions are not RW/RO aware.
   If it can be useful for someone :)
   [1]: 2 RW branches on 3.2 kernel (OMV actually), 1 root cron script running
   a sed -i command => leaves me 1 sed temporary file + it's whiteout each time
   .. I suppose something's wrong but I can't figure what (or maybe it is/was
   an aufs issue ?), or I just don't get everything.
   Regards,
   --
   Alexandre

References

   1. https://github.com/trapexit/mergerfs-tools/blob/master/src/fsck.mergerfs
   2. https://gist.github.com/Alex131089/560a1386dd756059e21b
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140