Re: unable to write via nfs to aufs of zfs branches
Pui Yong Ng: - I'm unable to write via nfs to an aufs union of zfs branches. The file is created, but nothing is written to the file, and nfs reports errno -38 (ENOSYS). - Writing directly to the aufs union (ie, bypass the nfs mount) works. - Reading via the nfs mount seems to work. - Writing via nfs to aufs union of EXT4 branches works. - Writing via nfs to zfs (ie, without aufs union) works. Could you check your kernel syslog? If my guess is correct, you have messages like this. [ cut here ] WARNING: at /.../fs/aufs/f_op.c:XXX au_do_aio.part.12+0x71/0x80 [aufs]() ::: Pid: 2431, comm: nfsd Tainted: PW O 3.2.41aufsD+ #447 Call Trace: [8105bcff] warn_slowpath_common+0x7f/0xc0 [8105bd5a] warn_slowpath_null+0x1a/0x20 [a02e95a1] au_do_aio.part.12+0x71/0x80 [aufs] [a02eb95d] aufs_aio_write+0x40d/0x420 [aufs] [a02eb550] ? aufs_splice_write+0x3b0/0x3b0 [aufs] [8113c21a] do_sync_readv_writev+0xda/0x120 [8113b904] ? rw_verify_area+0xe4/0x190 [8113c5ab] do_readv_writev+0xdb/0x1f0 [81540c1a] ? __mutex_lock_common.isra.3+0x48a/0x560 [a02e99ce] ? aufs_open_nondir+0x18e/0x1b0 [aufs] [8113c6f5] vfs_writev+0x35/0x60 [a0034a1b] nfsd_vfs_write.isra.9+0xeb/0x3f0 [nfsd] [8108939c] ? __validate_process_creds+0x10c/0x150 [a0035c95] ? nfsd_open+0x175/0x250 [nfsd] [a0037528] nfsd_write+0xf8/0x110 [nfsd] [a003e9c6] nfsd3_proc_write+0xe6/0x190 [nfsd] [a0030aeb] nfsd_dispatch+0xeb/0x230 [nfsd] [81509873] svc_process+0x4d3/0x8f0 [a00300d1] nfsd+0xd1/0x180 [nfsd] [a003] ? 0xa002 [8108193c] kthread+0x8c/0xa0 [8153fa00] ? schedule+0x30/0x60 [81549ef4] kernel_thread_helper+0x4/0x10 [810818b0] ? flush_kthread_worker+0xa0/0xa0 [81549ef0] ? gs_change+0x13/0x13 ---[ end trace a451d90c404007c0 ]--- These messages are produced by aufs, fs/aufs/f_op.c:au_do_aio():WARN_ON_ONCE(1). J. R. Okajima -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: unable to write via nfs to aufs of zfs branches
On 30 Sep 2013 14:32, [1]sf...@users.sourceforge.net wrote: Pui Yong Ng: - I'm unable to write via nfs to an aufs union of zfs branches. The file is created, but nothing is written to the file, and nfs reports errno -38 (ENOSYS). - Writing directly to the aufs union (ie, bypass the nfs mount) works. - Reading via the nfs mount seems to work. - Writing via nfs to aufs union of EXT4 branches works. - Writing via nfs to zfs (ie, without aufs union) works. Could you check your kernel syslog? If my guess is correct, you have messages like this. [] These messages are produced by aufs, fs/aufs/f_op.c:au_do_aio():WARN_ON_ONCE(1). Hi Okajima, You're right. Sorry I didn't notice this pertinent info - I think it is because it was only produced once and I only checked the syslog after various other log entries went in. What do these logs indicate? Thanks, Py References 1. mailto:sf...@users.sourceforge.net -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: unable to write via nfs to aufs of zfs branches
Pui Yong Ng: You're right. Sorry I didn't notice this pertinent info - I think it is because it was only produced once and I only checked the syslog after various other log entries went in. What do these logs indicate? It means zfs doesn't implement aio_write() file operation, and aufs doesn't support such filesystem. As far as I know, ALL filesystems in mainline implement aio_write() and aio_read() operations. I think this is my first time to meet a filesystem which doesn't have aio_write(). So our current situation is correct and expected result. Of course I know it is not good enough for you and I am considering how to solve. If you know anything about a zfs plan to implement aio_read/_write, please let me know. J. R. Okajima -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: unable to write via nfs to aufs of zfs branches
On 30 Sep 2013 15:39, [1]sf...@users.sourceforge.net wrote: It means zfs doesn't implement aio_write() file operation, and aufs doesn't support such filesystem. [] If you know anything about a zfs plan to implement aio_read/_write, please let me know. Indeed. Armed with this diagnosis, I've found that lack of async io support is a known issue ([2]https://github.com/zfsonlinux/zfs/issues/223), and is appatently unassigned and targetting milestone 0.8.0, so this looks pretty hopeless for me. I appreciate it if you have any ideas :) otherwise, perhaps I will try non-nfs network file systems. Thanks, Py References 1. mailto:sf...@users.sourceforge.net 2. https://github.com/zfsonlinux/zfs/issues/223 -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: aufs-mounts suddenly become readonly
Florian Klink: I use Arch Linux (amd64) using the zen-kernel (tree on zen-kernel.org, linus' kernel plus some nice additional features like UKSM and aufs3). The package is called linux-zen in Arch Linux. Now I am installing Arch Linux... I hope the problem will automatically go away, when aufs supports user namespaces (as the remount will then happen only inside the namespace of the machine). I therefore mailed you a git-compatible patch of Bhushan Jain's changes a minute ago. Does it mean that FS_USERNS_MOUNT solved the problem? Are you sure? J. R. Okajima -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: aufs-mounts suddenly become readonly
[1]sf...@users.sourceforge.net: Florian Klink: I use Arch Linux (amd64) using the zen-kernel (tree on zen-kernel.org, linus' kernel plus some nice additional features like UKSM and aufs3). The package is called linux-zen in Arch Linux. Now I am installing Arch Linux... Just to test the kernel? You can get the location of the git tree and the default config from here: [2]https://aur.archlinux.org/packages/linux-zen/?setlang=en I hope the problem will automatically go away, when aufs supports user namespaces (as the remount will then happen only inside the namespace of the machine). I therefore mailed you a git-compatible patch of Bhushan Jain's changes a minute ago. Does it mean that FS_USERNS_MOUNT solved the problem? Are you sure? I tested the same on a kernel without enabled user namespaces and a loopback ext4 fs mounted on the location of the container root (no aufs involved), it also got remounted read-only. So it doesn't seem to be really a bug of aufs. But I'm not yet sure how enabled FS_USERNS_MOUNT of the root filesystem will affect a remount of it inside an lxc container... I will build a kernel with enabled user namespaces, aufs FS_USERNS_MOUNT patch applied and will let you know how this one works. Florian J. R. Okajima References 1. mailto:sf...@users.sourceforge.net 2. https://aur.archlinux.org/packages/linux-zen/?setlang=en -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: aufs-mounts suddenly become readonly
I tested it on a kernel with the FS_USERNS_MOUNT and USER_NS enabled, the problem persists. So it doesn't look like a bug of aufs, but still thanks for your help! Florian Am 30.09.2013 11:46, schrieb Florian Klink: [[1]1]sf...@users.sourceforge.net: Florian Klink: I use Arch Linux (amd64) using the zen-kernel (tree on zen-kernel.org, linus' kernel plus some nice additional features like UKSM and aufs3). The package is called linux-zen in Arch Linux. Now I am installing Arch Linux... Just to test the kernel? You can get the location of the git tree and the default config from here: [2][2]https://aur.archlinux.org/packages/linux-zen/?setlang=en I hope the problem will automatically go away, when aufs supports user namespaces (as the remount will then happen only inside the namespace of the machine). I therefore mailed you a git-compatible patch of Bhushan Jain's changes a minute ago. Does it mean that FS_USERNS_MOUNT solved the problem? Are you sure? I tested the same on a kernel without enabled user namespaces and a loopback ext4 fs mounted on the location of the container root (no aufs involved), it also got remounted read-only. So it doesn't seem to be really a bug of aufs. But I'm not yet sure how enabled FS_USERNS_MOUNT of the root filesystem will affect a remount of it inside an lxc container... I will build a kernel with enabled user namespaces, aufs FS_USERNS_MOUNT patch applied and will let you know how this one works. Florian J. R. Okajima References 1. [3]mailto:sf...@users.sourceforge.net 2. [4]https://aur.archlinux.org/packages/linux-zen/?setlang=en -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register [5]http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk References 1. mailto:1]sf...@users.sourceforge.net 2. https://aur.archlinux.org/packages/linux-zen/?setlang=en 3. mailto:sf...@users.sourceforge.net 4. https://aur.archlinux.org/packages/linux-zen/?setlang=en 5. http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk
Re: aufs-mounts suddenly become readonly
Florian Klink: Just to test the kernel? You can get the location of the git tree and the default config from here: https://aur.archlinux.org/packages/linux-zen/?setlang=en To find out who runs remount,ro. I don't understand why you focused FS_USERNS_MOUNT. J. R. Okajima -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134791iu=/4140/ostg.clktrk
Re: aufs-mounts suddenly become readonly
sf...@users.sourceforge.net: Florian Klink: Just to test the kernel? You can get the location of the git tree and the default config from here: https://aur.archlinux.org/packages/linux-zen/?setlang=en To find out who runs remount,ro. I don't understand why you focused FS_USERNS_MOUNT. J. R. Okajima I think, it's the init system that did the remount-ro (a remount,ro of / is done when you shutdown). I now updated to a newer version, (of systemd) and the problem went magically away - seems like systemd now properly detects when it's inside a container and doesn't remount,ro anymore ;-) About FS_USERNS_MOUNT: I thought that there is probably some /dev/root that's mounted inside the container to / by the container root user during startup, and by allowing a user mount inside the namespace, the remount would probably be only effective inside the container namespace. But that was a red herring... But I still think that enabling FS_USERNS_MOUNT is a good idea. I don't really see a reason why a user should not be able to create a union mount of two directories he's able to access on their own anyway. For sure, there are some use cases for this, like the following: user has a big, read only volume (blu ray?) of a lot of small files, and wants to change some of them, but still see the result as a whole without having to copy all whole files to a read-write directory. Florian -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134791iu=/4140/ostg.clktrk
Re: unable to write via nfs to aufs of zfs branches
Pui Yong Ng: I appreciate it if you have any ideas :) otherwise, perhaps I will try non-nfs network file systems. Here is a patch for a new aufs mount option noaiop. - apply the patch (to aufs source files) and rebuild your aufs module. - when you mount aufs with zfs branches, add a mount option noaiop. Since this is a dirty workaround for a non-mainline filesystem (currently), it won't be included in aufs release. J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134791iu=/4140/ostg.clktrk
Re: Linux 3.11-rc7: loop_backing_file() is not defined
sf...@users.sourceforge.net: Florian Klink: The error message also appears when both aufs and loop are compiled as a module. Hmm... I am afraid we should consider all combinations of these. - aufs (is/is not) a module - the source files are from (aufs3-linux/aufs3-standalone).git + the system when you build aufs3-standalone.git is running the (host/target) kernel - loop (is/is not) a module Hopefull this patch supports all cases. J. R. Okajima a.patch.bz2 Description: BZip2 compressed data -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60134791iu=/4140/ostg.clktrk
