Hi Paul, I received this bug report through Debian's BTS. You may read the full log at http://bugs.debian.org/557817, should you need to.
I'm not quite sure what the issue is. At first I thought it might be because he was connecting to localhost, but apparently that wasn't the issue. Could you have a look? (in the interest of keeping progress visible, please make sure to keep the bug address, [email protected], in Cc) Regards, ----- Forwarded message from martin f krafft <[email protected]> ----- Date: Tue, 24 Nov 2009 17:55:47 +0100 From: martin f krafft <[email protected]> To: Debian Bug Tracking System <[email protected]> Subject: Bug#557817: nbd has I/O errors with md-over-dmcrypt-over-nbd X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on samba.grep.be X-Spam-Level: X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW autolearn=ham version=3.2.5 Reply-To: martin f krafft <[email protected]>, [email protected] Message-ID: <[email protected]> User-Agent: Mutt/1.5.20 (2009-06-14) Package: nbd-client Version: 1:2.9.14-1 Severity: normal I am trying to investigate a remote backup solution: nbd-server gives me a block device from remote, which I locally crypt with dmcrypt, and then use as a component of a RAID1. The backups are written to the RAID1, and by rsync'ing the RAID, I effectively generate an encrypted live-backup remotely. Unfortunatly, that is only the theory, for in practice, the dmcrypt+nbd combination doesn't seem to work (it might well be a dmcrypt problem, though not likely): % dd if=/dev/zero of=nbd-file bs=1M count=1 seek=1023 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00447344 s, 234 MB/s % nbd-server 127.0.0.1:12345 `pwd`/nbd-file 1024M -C/dev/null % netstat -natp | grep :12345 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 16385/nbd-server % sudo nbd-client localhost 12345 /dev/nbd0 Negotiation: ..size = 1048576KB bs=1024, sz=1048576 % echo foo > key % sudo cryptsetup luksFormat /dev/nbd0 ./key WARNING! ======== This will overwrite data on /dev/nbd0 irrevocably. Are you sure? (Type uppercase yes): YES % grep nbd /var/log/syslog | tail Nov 24 17:24:37 piper nbd_server[16385]: connect from 127.0.0.1, assigned file is /home/madduck/.tmp/cdt.hPOcvv/nbd-file Nov 24 17:24:37 piper nbd_server[16385]: Can't open authorization file /etc/nbd-server/allow (No such file or directory). Nov 24 17:24:37 piper nbd_server[16385]: Authorized client Nov 24 17:24:37 piper nbd_server[16405]: Starting to serve Nov 24 17:24:37 piper nbd_server[16405]: Size of exported file/device is 1073741824 Nov 24 17:24:37 piper kernel: [537982.733434] nbd0: unknown partition table % sudo cryptsetup luksOpen -d ./key /dev/nbd0 cryptnbd Key slot 0 unlocked. % dd if=/dev/zero of=loop-file bs=1M count=1 seek=1023 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00413988 s, 253 MB/s % sudo losetup -vf ./loop-file Loop device is /dev/loop0 % sudo mdadm -Cl1 -n2 -ayes /dev/md99 /dev/mapper/cryptnbd /dev/loop0 mdadm: array /dev/md99 started. % cat /proc/mdstat Personalities : [raid1] md99 : active (auto-read-only) raid1 loop0[1] dm-7[0] 1047936 blocks [2/2] [UU] resync=PENDING unused devices: <none> So far so good. Now as soon as I write anything to the array, it will start sync'ing. And this is when the I/O errors start, but the RAID syncs fine actually. % tail -f /var/log/syslog | grep nbd& [1] 17024 17025 % dd if=/dev/zero of=/dev/md99 bs=1 count=1 dd: opening `/dev/md99': Permission denied % sudo dd if=/dev/zero of=/dev/md99 bs=1 count=1 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0993314 s, 0.0 kB/s Nov 24 17:41:45 piper kernel: [539010.683028] end_request: I/O error, dev nbd0, sector 0 [???] (34 in between) Nov 24 17:42:41 piper kernel: [539066.182929] md: md99: resync done. Nov 24 17:42:41 piper kernel: [539066.212929] end_request: I/O error, dev nbd0, sector 0 Nov 24 17:42:41 piper kernel: [539066.435511] end_request: I/O error, dev nbd0, sector 0 Nov 24 17:42:43 piper mdadm[2599]: RebuildFinished event detected on md device /dev/md99 Nov 24 17:42:43 piper kernel: [539068.098878] RAID1 conf printout: Nov 24 17:42:43 piper kernel: [539068.098883] --- wd:2 rd:2 Nov 24 17:42:43 piper kernel: [539068.098887] disk 0, wo:0, o:1, dev:dm-7 Nov 24 17:42:43 piper kernel: [539068.098891] disk 1, wo:0, o:1, dev:loop0 Nov 24 17:42:43 piper kernel: [539068.099206] end_request: I/O error, dev nbd0, sector 0 % cat /proc/mdstat Personalities : [raid1] md99 : active raid1 loop0[1] dm-7[0] 1047936 blocks [2/2] [UU] 976510912 blocks [2/2] [UU] unused devices: <none> Subsequently, I can read the entire array (I tried with dd) without any further I/O errors, but when I write to it, the I/O errors re-appear. The problem only appears when dmcrypt is in between nbd and md, not if I use /dev/nbd0 directly for the RAID1. Thus this problem might well be with dmcrypt, but it's unlikely. I tried the same setup with a loop device instead of nbd (loop0 directly, dmcrypt-on-loop1, RAID1 across loop0 and dmcrypt device), and there were no I/O errors. -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.31-1-amd64 (SMP w/1 CPU core) Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages nbd-server depends on: ii adduser 3.111 add and remove users and groups ii debconf [debconf-2.0] 1.5.28 Debian configuration management sy ii libc6 2.10.1-7 GNU C Library: Shared libraries ii libglib2.0-0 2.22.2-2 The GLib library of C routines ii ucf 3.0025 Update Configuration File: preserv nbd-server recommends no packages. nbd-server suggests no packages. -- debconf information excluded -- .''`. martin f. krafft <[email protected]> Related projects: : :' : proud Debian developer http://debiansystem.info `. `'` http://people.debian.org/~madduck http://vcs-pkg.org `- Debian - when you have better things to do than fixing systems ----- End forwarded message ----- -- <Lo-lan-do> Home is where you have to wash the dishes. -- #debian-devel, Freenode, 2004-09-22 -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected]
