On 26/01/16 05:50, sf...@users.sourceforge.net wrote: > Sorry for my late reply. > > OmegaPhil: >> Suddenly the VM doesn't error anymore in the particular test I set up, >> so back on the server, I fiddled with the rsync init.d script and ran >> the daemon via 'strace -fv'. One EINVAL hit in the resulting file, here >> is it with some context: > > Thanks. > The trace shows that > - pid1293 opens "/omega1-home/" > - confirms the dir is aufs > - and then issues ioctl(2). > > It is good. > And the readdir(3) call by rsync is converted to ioctl(AUFS_CTL_RDU_INO) > or ioctl(AUFS_CTL_RDU) by libau.so, and it gets EINVAL. Still I don't > know why. But I'd ask you one more thing. Give strace '-e raw=ioctl'. It > will give us the raw parameters which will be helpful to debug libau.so > and aufs kernel module. > > Also I'd suggest you to try a tiny test for libau. Try > $ LD_PRELOAD=libau.so LIBAU=all strace -o /tmp/s -e raw=ioctl ls > /your/aufs/dir > and see the differences between those two strace outputs, one is by your > rsync and the other is by ls.
Here is the raw ioctl trace:
========================================================================
[pid 10412] chdir("/omega1-home/") = 0
[pid 10412] socketpair(PF_LOCAL, SOCK_STREAM, 0, [4, 6]) = 0
[pid 10412] fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
[pid 10412] fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 10412] fcntl(6, F_GETFL) = 0x2 (flags O_RDWR)
[pid 10412] fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 10412] clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f563688c9d0)
= 10413
Process 10413 attached
[pid 10412] close(6 <unfinished ...>
[pid 10413] set_robust_list(0x7f563688c9e0, 24 <unfinished ...>
[pid 10412] <... close resumed> ) = 0
[pid 10413] <... set_robust_list resumed> ) = 0
[pid 10412] lstat(".", <unfinished ...>
[pid 10413] close(4 <unfinished ...>
[pid 10412] <... lstat resumed> {st_dev=makedev(0, 34), st_ino=13,
st_mode=S_IFDIR|0755, st_nlink=4, st_uid=0, st_gid
=0, st_blksize=4096, st_blocks=0, st_size=66,
st_atime=2016/01/25-21:10:15, st_mtime=2014/09/13-12:11:23, st_ctime=20
15/01/07-07:46:30}) = 0
[pid 10413] <... close resumed> ) = 0
[pid 10412] openat(AT_FDCWD, ".",
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC <unfinished ...>
[pid 10413] select(6, [5], [], [5], {60, 0} <unfinished ...>
[pid 10412] <... openat resumed> ) = 6
[pid 10412] brk(0x557f96466000) = 0x557f96466000
[pid 10412] fstatfs(6, {f_type=0x61756673, f_bsize=4096,
f_blocks=3418641366, f_bfree=654794207, f_bavail=651742965,
f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=242, f_frsize=4096}) = 0
[pid 10412] ioctl(0x6, 0xc0404100, 0x7ffff4878360) = -1 (errno 22)
[pid 10412] sendto(3, "<28>Jan 26 19:19:39 rsyncd[10412"..., 104,
MSG_NOSIGNAL, NULL, 0) = 104
[pid 10412] fstatfs(6, {f_type=0x61756673, f_bsize=4096,
f_blocks=3418641366, f_bfree=654794207, f_bavail=651742965, f_files=0,
f_ffree=0, f_fsid={0, 0}, f_namelen=242, f_frsize=4096}) = 0
[pid 10412] futex(0x7f5635ab50a8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 10412] close(6) = 0
[pid 10412] lstat(".", {st_dev=makedev(0, 34), st_ino=13,
st_mode=S_IFDIR|0755, st_nlink=4, st_uid=0, st_gid=0, st_blksize=4096,
st_blocks=0, st_size=66, st_atime=2016/01/25-21:10:15,
st_mtime=2014/09/13-12:11:23, st_ctime=2015/01/07-07:46:30}) = 0
========================================================================
See 'errno 22'.
Sorry, was obsessing about rsync - ls also fails in the same way with
the environment variables used...
========================================================================
stat("/mnt/backups/omega1-home", {st_mode=S_IFDIR|0755, st_size=66,
...}) = 0
openat(AT_FDCWD, "/mnt/backups/omega1-home",
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fstatfs(3, {f_type=0x61756673, f_bsize=4096, f_blocks=3418641366,
f_bfree=654792161, f_bavail=651740917, f_files=0, f
_ffree=0, f_fsid={0, 0}, f_namelen=242, f_frsize=4096}) = 0
ioctl(0x3, 0xc0404100, 0x7ffd38fe6220) = -1 (errno 22)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2492, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fc076108000
read(4, "# Locale name alias data base.\n#"..., 4096) = 2492
read(4, "", 4096) = 0
close(4) = 0
munmap(0x7fc076108000, 4096) = 0
========================================================================
> ----------------------------------------------------------------------
>
>> Minor update - rsync/aufs has suddenly decided to fail all over the
>> place with this supposed kernel memory problem, so even SSH+rsync is
>> somewhat unusable currently.
>
> Do you mean the problem is not related to a specific kernel version?
>
>
> J. R. Okajima
That was referring to the previous kernel memory allocation failures,
suddenly they take over everything rather than just a few directories
(when I converted my backup script to go from rsync daemon to normal
SSH+rsync due to the invalid argument issue). Very suspicious. I'm
currently running on 4.3 in order to do aufs testing.
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
