Re: rsync on branches causes filesystem errors

[bl4]

sf...@users.sourceforge.net wrote:
Here is my current guess.
- qemu or linux-2.6.26 may NOT flush ext2/3 at remounting readonly, but
  they may flush at unmounting time.
- your /dev/hda1 is not unmounted. it is remounted ro at shutdown time
  too.
- contrastingly /dev/hdb can be unmounted, and it can be clean.

But I have to say that my option becomes stronger a little.
Don't you agree?


It's not certain that your test shows the same problem. To find out, I 
would need to umount /ro and fsck. Which means / must be unmounted 
first. Even if it's possible, I would go insane trying to understand 
things like pivot_root.

For testing, /var can be put on a separate partition and mounted as 
aufs. It is always written to and it can be unmounted on shutdown.

Meanwhile I found 2 ways to improve work in qemu: convert root disk 
image to raw format so that it can be mounted as loopback on the host, 
and a virtual serial port with input and output for the kernel 
redirected on the terminal from which qemu was run, which means 
scrolling, copy and paste is possible.

I've attached 0_serial.diff and 1_var.diff. Most important parts are 
/etc/fstab:

/dev/hdb /varro ext3 noauto,ro 0 0
none /varrw tmpfs noauto,rw 0 0
none /var aufs noauto,dirs=/varrw=rw:/varro=ro 0 0

and /etc/init.d/aufs-var-sync:

fsck -nf /dev/hdb
sleep 2
auplink /var flush
mount -o remount,rw /varro
rsync --exclude=".wh.*" --exclude=lost+found -aHSx --devices --specials 
/varrw/ /varro/
cd /varrw
find . -name ".wh.*" -and -not -name ".wh..wh.*" | sed 
's/^\.\//\/varro\//' | sed 's/\/\.wh\./\//' | /usr/bin/xargs -d "\n" rm -rf
cd /
mount -o remount,ro /varro
sleep 2
fsck -nf /dev/hdb
sleep 2
umount /var
umount /varrw
umount /varro
sleep 2
fsck -nf /dev/hdb
busybox sh


This is what I did on the host to set it up:

qemu-img convert -O raw debian_lenny_aufs.vmdk debian_lenny_aufs.raw
mount -o loop,offset=32256 debian_lenny_aufs.raw /mnt/0
dd if=/dev/zero bs=192M count=1 of=var.ext
mkfs -t ext3 -F var.ext
mount -o loop var.ext /mnt/1
cp -a /mnt/0/var/* /mnt/1/
umount /mnt/1
cd /mnt/0
patch -p0 < ~/0_serial.diff
patch -p0 < ~/1_var.diff
mkdir varro
mkdir varrw
chmod +x etc/init.d/aufs-var-mount
chmod +x etc/init.d/aufs-var-sync
cd /mnt/0/etc/rcS.d
ln -s ../init.d/aufs-var-mount S36aufs-var-mount
cd /mnt/0/etc/rc0.d
ln -s ../init.d/aufs-var-sync S38aufs-var-sync
cd /mnt/0/etc/rc6.d
ln -s ../init.d/aufs-var-sync S38aufs-var-sync
cd ~
umount /mnt/0

I started qemu:

qemu -serial stdio -hda debian_lenny_aufs.raw -hdb var.ext

selected the new bootloader entry and switched back to the terminal.

debian:~# cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec 0 0
none /proc proc rw,nosuid,nodev,noexec 0 0
udev /dev tmpfs rw,size=10240k,mode=755 0 0
/dev/hda1 / ext3 rw,errors=continue,data=ordered 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,gid=5,mode=620 0 0
/dev/hdb /varro ext3 ro,errors=continue,data=ordered 0 0
none /varrw tmpfs rw 0 0
none /var aufs 
rw,si=cc26a83b,xino=/varrw/.aufs.xino,br:/varrw=rw:/varro=ro 0 0

output from shutdown:

fsck 1.41.3 (12-Oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
Warning!  /dev/hdb is mounted.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/hdb: 1279/49152 files (0.4% non-contiguous), 82606/196608 blocks
[   21.561310] EXT3 FS on hdb, internal journal
fsck 1.41.3 (12-Oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
Warning!  /dev/hdb is mounted.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix? no

Inode 14342 was part of the orphaned inode list.  IGNORED.
Inode 14343 was part of the orphaned inode list.  IGNORED.
Inode 14345 was part of the orphaned inode list.  IGNORED.
Inode 14347 was part of the orphaned inode list.  IGNORED.
Inode 14352 was part of the orphaned inode list.  IGNORED.
Inode 14353 was part of the orphaned inode list.  IGNORED.
Inode 14354 was part of the orphaned inode list.  IGNORED.
Inode 14362 was part of the orphaned inode list.  IGNORED.
Deleted inode 14370 has zero dtime.  Fix? no

Inode 14371 was part of the orphaned inode list.  IGNORED.
Inode 14375 was part of the orphaned inode list.  IGNORED.
Inode 14376 was part of the orphaned inode list.  IGNORED.
Inode 16387 was part of the orphaned inode list.  IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(58373--58413) -(58455--58972) -(66567--66571)
Fix? no

Inode bitmap differences:  -(14342--14343) -14345 -14347 -(14352--14354) 
-14362 -(14370--14371) -(14375--14376) -16387
Fix? no 



/dev/hdb: ********** WARNING: Filesystem still has errors **********

/dev/hdb: 1292/49152 files (0.4% non-contiguous), 83257/196608 blocks
fsck 1.41.3 (12-Oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix? no

Inode 14342 was part of the orphaned inode list.  IGNORED.
Inode 14343 was part of the orphaned inode list.  IGNORED.
Inode 14345 was part of the orphaned inode list.  IGNORED.
Inode 14347 was part of the orphaned inode list.  IGNORED.
Inode 14352 was part of the orphaned inode list.  IGNORED.
Inode 14353 was part of the orphaned inode list.  IGNORED.
Inode 14354 was part of the orphaned inode list.  IGNORED.
Inode 14362 was part of the orphaned inode list.  IGNORED.
Deleted inode 14370 has zero dtime.  Fix? no

Inode 14371 was part of the orphaned inode list.  IGNORED.
Inode 14375 was part of the orphaned inode list.  IGNORED.
Inode 14376 was part of the orphaned inode list.  IGNORED.
Inode 16387 was part of the orphaned inode list.  IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(58373--58413) -(58455--58972) -(66567--66571)
Fix? no

Inode bitmap differences:  -(14342--14343) -14345 -14347 -(14352--14354) 
-14362 -(14370--14371) -(14375--14376) -16387
Fix? no


/dev/hdb: ********** WARNING: Filesystem still has errors **********

/dev/hdb: 1292/49152 files (0.4% non-contiguous), 83257/196608 blocks


Umount did not fix anything. In the example you posted, it looks like 
some other problem.


By the way, I could not find /etc/default/auplink. So I had to create it
manually.


Does it matter if the scripts call auplink manually?


--
bl4
--- boot/grub/menu.lst.0	2010-03-19 09:31:15.000000000 +0100
+++ boot/grub/menu.lst	2010-03-20 09:25:31.000000000 +0100
@@ -121,12 +121,12 @@
 
 title		Debian GNU/Linux, kernel 2.6.26-2-686
 root		(hd0,0)
-kernel		/boot/vmlinuz-2.6.26-2-686 root=/dev/hda1 rw
+kernel		/boot/vmlinuz-2.6.26-2-686 console=ttyS0,9600n8 root=/dev/hda1 rw
 initrd		/boot/initrd.img-2.6.26-2-686
 
 title		Debian GNU/Linux, kernel 2.6.26-2-686 aufs
 root		(hd0,0)
-kernel		/boot/vmlinuz-2.6.26-2-686 root=/dev/hda1 ro aufs
+kernel		/boot/vmlinuz-2.6.26-2-686 console=ttyS0,9600n8 root=/dev/hda1 ro aufs
 initrd		/boot/initrd.img-2.6.26-2-686
 
 ### END DEBIAN AUTOMAGIC KERNELS LIST
--- etc/inittab.0	2010-03-20 09:25:48.000000000 +0100
+++ etc/inittab	2010-03-20 09:26:02.000000000 +0100
@@ -60,7 +60,7 @@
 
 # Example how to put a getty on a serial line (for a terminal)
 #
-#T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
+T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
 #T1:23:respawn:/sbin/getty -L ttyS1 9600 vt100
 
 # Example how to put a getty on a modem line.
--- boot/grub/menu.lst.1	2010-03-20 09:25:31.000000000 +0100
+++ boot/grub/menu.lst	2010-03-20 09:30:49.000000000 +0100
@@ -129,4 +129,9 @@
 kernel		/boot/vmlinuz-2.6.26-2-686 console=ttyS0,9600n8 root=/dev/hda1 ro aufs
 initrd		/boot/initrd.img-2.6.26-2-686
 
+title		Debian GNU/Linux, kernel 2.6.26-2-686 auvar
+root		(hd0,0)
+kernel		/boot/vmlinuz-2.6.26-2-686 console=ttyS0,9600n8 root=/dev/hda1 rw auvar
+initrd		/boot/initrd.img-2.6.26-2-686
+
 ### END DEBIAN AUTOMAGIC KERNELS LIST
--- etc/fstab.1	2010-03-20 09:32:39.000000000 +0100
+++ etc/fstab	2010-03-20 09:34:38.000000000 +0100
@@ -1 +1,4 @@
 proc /proc proc defaults 0 0
+/dev/hdb /varro ext3 noauto,ro 0 0
+none /varrw tmpfs noauto,rw 0 0
+none /var aufs noauto,dirs=/varrw=rw:/varro=ro 0 0
--- /dev/null	2010-03-20 09:09:32.681410877 +0100
+++ etc/init.d/aufs-var-mount	2010-03-20 09:39:25.000000000 +0100
@@ -0,0 +1,6 @@
+#!/bin/sh
+if grep -q 'auvar' /proc/cmdline; then
+  mount /varro
+  mount /varrw
+  mount /var
+fi
--- /dev/null	2010-03-20 09:09:32.681410877 +0100
+++ etc/init.d/aufs-var-sync	2010-03-20 10:12:46.000000000 +0100
@@ -0,0 +1,21 @@
+#!/bin/sh
+if grep -q 'auvar' /proc/cmdline; then
+  fsck -nf /dev/hdb
+  sleep 2
+  auplink /var flush
+  mount -o remount,rw /varro
+  rsync --exclude=".wh.*" --exclude=lost+found -aHSx --devices --specials /varrw/ /varro/
+  cd /varrw
+  find . -name ".wh.*" -and -not -name ".wh..wh.*" | sed 's/^\.\//\/varro\//' | sed 's/\/\.wh\./\//' | /usr/bin/xargs -d "\n" rm -rf
+  cd /
+  mount -o remount,ro /varro
+  sleep 2
+  fsck -nf /dev/hdb
+  sleep 2
+  umount /var
+  umount /varrw
+  umount /varro
+  sleep 2
+  fsck -nf /dev/hdb
+  busybox sh
+fi
------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev