Hello again, thank you so much for your support!
On Mon, Aug 26, 2013 at 7:18 PM, <sf...@users.sourceforge.net> wrote: > Steffen Dettmer: >> Wouldn't this just delay the problem a little bit (until some log >> file consumes some more bytes)? > > It depends upon how you use aufs. > If the disk space pressure is caused by a few big files, then > expanding /rw will not be effective (as you wrote). I assume the "most-likely" cause for us could be that some log message appearing systematically and frequently (without having automatic supression) resulting in multiple log files typically 5 MB each, but of course there are measures against and the file system should never run full. Except those nasty exceptional exceptions :) >> command like "rsync --delete /rw /ro", but I had to check) works >> fine, but remountro often starts to moan about orphaned inodes in >> ext4 and no more remoutrw are possible without reboot - which >> seems to be a known problem. So currently we never remountro >> after syncing. Not sure if this is good... > > Just out of curious, is it a known problem "of ext4"? Yes, someone cannot safely remountrw/remountro/remountrw an ext4 (or ext3, I had slightly different test result with ext2 IIRC), this was discussed at linux-ext4 mailing list: http://patchwork.ozlabs.org/patch/113594/ which suggests to use umount/mount instead of remount, which is a bit problematic when it comes to be the root file system :) > In order to move files between aufs branches, I'd suggest you to try > "aubrsync" utility in aufs-util.git. It handles all whiteoutes and > pseudo-links well, but the tool is for the shutdown-time only. With shutdown-time sync we had no issues (someone made a script bases on some AUFS example, but simplified it a lot, I'm not sure about the details), it just worked as expected. It was a bit unclear why the reboot after the sync was required and requirements changed not to reboot after each sync. With first (probably over-simplified tests), it seemed to work well, until we met the ext remountro/remountrw issue and now the disk full issue. The latter is a bit disturbing, because during my tests I reached a state where reboot was failing: software detected full file system and rebooted by calling /sbin/reboot but some of the reboot helper processes hang, something like: umountroot: No space left on device reboot: No space left on device startpar: service(s) returned failure [...] reboot... Give root parrword for maintenance. (this was on simulator, otherwise I hadn't seen it :)). Bad on embedded device :) > For a running system, a new utility "aumvdown" is recommended. > But I am still working on it and not completed yet. > Additionally aumvdown is for aufs3.9 and later... As we are Wheezy-based (Debian 7), this seems not to be an option at the moment. > Anyway, back on topic :) > >> Would it be recommended to store XINO files on it's own tmpfs, >> for example? > > Does it mean > - mount another tmpfs > - put XINO files on that tmpfs > and separating them from /rw? Yes, exactly. Or maybe ram block device with other file system, if needed. > It is a good approach to identify the cause of your problem, but I am > not sure whether it will be a good solution for you. mmm... ok... What alternatives could be considered? >> So I "simulated" a full disk by using: >> >> # while true; do dd if=/dev/zero of=large.log.$((n++)) bs=1M count=10 ; done >> >> but I assume it has an effect on XINO files whether there are a >> few big or a lot of small files? > > The size of XINO file is depending the number of files basically. > It means still we are not sure that the cause of the problem is XINO or > not... Do the debug files tell something helpful: xi0: 1, 48x4096 61440 xi1: 1, 216x4096 122340 xib: 8x4096 4096 xigen: 16x4096 8192 what is causing the issue? But anyway, I think we should be able to handle any disk-full situation, at least reboot should work. Regards, Steffen ------------------------------------------------------------------------------ Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
