Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Wed, Jul 18, 2012 at 01:56:04AM -0400, Rajesh Amaravathi wrote: As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I think its not really necessary since we can mount other exports with nolock option. If we take out NLM or disable NLM at the kernel level, then every time we need NLM from kernel, we need to recompile the kernel/have a secondary kernel with NLM and reboot, much tedious than simply killing Gluster/fuse NFS and after kernel NLM's work is done, restart Gluster/fuse NFS. My $0.02 :) Since there's good reason to want locking with Gluster NFS, wouldn't the answer be to add code to kernel that would allow the kernel's locking to be turned off and on in the standard way - a file called something like kernel_nfs_locking that would hold either a 0 or 1? Obviously the kernel's NFS support was built on the assumption that no one with kernel NFS available would want to run userland NFS. Gluster shows that assumption is wrong. So wouldn't it be sensible for someone on the Gluster team to be submitting kernel patches to fix this oversight? Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Thu, Jul 19, 2012 at 05:16:24AM -0400, Krishna Srinivas wrote: It was pretty confusing to read this thread. Hope I can clarify the questions here. Thanks. I was confused. The other discussion in this thread was related to NLM which has been implemented in 3.3.0. This is to support locking calls from the NFS clients to support fcntl() locking for the applications running on nfs client. NLM server is implemented in glusterfs as well as kernel. NLM server implemented in kernel is used by kernel-nfsd as well as kernel-nfs-client. Hence if you have an nfs mount point, the kernel-nfs-client automatically starts kernel NLM server. So if glusterfs-nfs process is already running on a system (and hence it also runs its own NLM server) and if you try to do mount -t nfs someserver:/export /mnt/nfs on the same system it fails as kernel-nfs-client won't be able to start kernel-NLM-server (because glusterfs NLM server would have already registered with portmapper for NLM service and hence kernel-NLM-server registration with portmapper fails). Workaround is mount -t nfs -o nolock someserver:/export /mnt/nfs if you really want to have an nfs mount on the same machine where glusterfs-nfs process is running. So, if you want to run both at once, only one can lock. Is the architecture of NLM such that there could never be a single NLM server for both Gluster and kernel (whether that single server be Gluster's or kernel's)? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
This is another reason why it there *really* should be a way with the CLI to have an option that GlusterNFS be enabled on only one machine in a cluster (I've asked for that before). Our cluster has a private network. Inside the cluster private network I use native glusterfs mounts. Outside the private network, I use an nfs mount exported by a gateway machine. For us it is only necessary to have GlusterNFS running on that gateway machine. Having the glusterfs file system mounted nolock I can live with but but it is intolerable if the home directories from outside the cluster that are mounted with automount in the gluster require nolock. I might be able to tolerate this it if this is the situation is only on the gateway machine, not every machine in the cluster. In the the past I used a hack to create a peer that had only Kernel NFS by making sure that the Kernel NFS came up first on boot. Is that still true? Bill Sebok Computer Software Manager, Univ. of Maryland, Astronomy Internet: w...@astro.umd.eduURL: http://furo.astro.umd.edu/ On Wed, Jul 18, 2012 at 01:56:04AM -0400, Rajesh Amaravathi wrote: Let me elucidate with an example: host1: GlusterNfs server, say vol host2: Kernel Nfs export: say, export Assuming host1 and host2 are not peers, i.e, host2 does NOT have any Gluster Nfs servers running, Lets assume for some reason, export needs to be mounted on host1. This is not possible *with locking* since glusterNfs already has the portmap registration. Even if it is mounted, then the kernel nfs client starts the kernel NLM v4, which will override the portmap registration of NLMv4 on host1, so Gluster NFS' NLM implementation wouldn't work. However, you can mount export on host1 with -o nolock option, since the kernel nfs client then does not attempt to spawn its own NLM. Also, the only way host1 can mount vol via nfs is by specifying nolock option, else the same conflict arises. As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I think its not really necessary since we can mount other exports with nolock option. If we take out NLM or disable NLM at the kernel level, then every time we need NLM from kernel, we need to recompile the kernel/have a secondary kernel with NLM and reboot, much tedious than simply killing Gluster/fuse NFS and after kernel NLM's work is done, restart Gluster/fuse NFS. My $0.02 :) Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: Gluster General Discussion List gluster-users@gluster.org Sent: Tuesday, July 17, 2012 5:38:48 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Sorry my question was too vague. What I meant to ask is if it is possible, since there is a conflict between the locking requests from the kernel's NFS and from Gluster/fuse's NFS, that the kernel might be compiled so with some or all of its NFS support disabled, so that then Gluster/fuse NFS-locking would work. Perhaps longer term there needs to be a way to have the kernel shut its NFS locking attempts off, just if there is a userland NFS such as Gluster's running. Meanwhile can enough of NFS be taken out of a custom kernel to allow Gluster to lock? Thanks, Whit On Tue, Jul 17, 2012 at 08:03:03AM -0400, Rajesh Amaravathi wrote: it should be possible to mount another kernel export with -o nolock option and compile kernel on it. I'm just guessing when you mount with nolock option, we are mounting for mostly read purposes and not for critical writes. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: David Coulson da...@davidcoulson.net, Gluster General Discussion List gluster-users@gluster.org Sent: Monday, July 16, 2012 9:56:28 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
it should be possible to mount another kernel export with -o nolock option and compile kernel on it. I'm just guessing when you mount with nolock option, we are mounting for mostly read purposes and not for critical writes. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: David Coulson da...@davidcoulson.net, Gluster General Discussion List gluster-users@gluster.org Sent: Monday, July 16, 2012 9:56:28 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Sorry my question was too vague. What I meant to ask is if it is possible, since there is a conflict between the locking requests from the kernel's NFS and from Gluster/fuse's NFS, that the kernel might be compiled so with some or all of its NFS support disabled, so that then Gluster/fuse NFS-locking would work. Perhaps longer term there needs to be a way to have the kernel shut its NFS locking attempts off, just if there is a userland NFS such as Gluster's running. Meanwhile can enough of NFS be taken out of a custom kernel to allow Gluster to lock? Thanks, Whit On Tue, Jul 17, 2012 at 08:03:03AM -0400, Rajesh Amaravathi wrote: it should be possible to mount another kernel export with -o nolock option and compile kernel on it. I'm just guessing when you mount with nolock option, we are mounting for mostly read purposes and not for critical writes. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: David Coulson da...@davidcoulson.net, Gluster General Discussion List gluster-users@gluster.org Sent: Monday, July 16, 2012 9:56:28 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Let me elucidate with an example: host1: GlusterNfs server, say vol host2: Kernel Nfs export: say, export Assuming host1 and host2 are not peers, i.e, host2 does NOT have any Gluster Nfs servers running, Lets assume for some reason, export needs to be mounted on host1. This is not possible *with locking* since glusterNfs already has the portmap registration. Even if it is mounted, then the kernel nfs client starts the kernel NLM v4, which will override the portmap registration of NLMv4 on host1, so Gluster NFS' NLM implementation wouldn't work. However, you can mount export on host1 with -o nolock option, since the kernel nfs client then does not attempt to spawn its own NLM. Also, the only way host1 can mount vol via nfs is by specifying nolock option, else the same conflict arises. As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I think its not really necessary since we can mount other exports with nolock option. If we take out NLM or disable NLM at the kernel level, then every time we need NLM from kernel, we need to recompile the kernel/have a secondary kernel with NLM and reboot, much tedious than simply killing Gluster/fuse NFS and after kernel NLM's work is done, restart Gluster/fuse NFS. My $0.02 :) Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: Gluster General Discussion List gluster-users@gluster.org Sent: Tuesday, July 17, 2012 5:38:48 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Sorry my question was too vague. What I meant to ask is if it is possible, since there is a conflict between the locking requests from the kernel's NFS and from Gluster/fuse's NFS, that the kernel might be compiled so with some or all of its NFS support disabled, so that then Gluster/fuse NFS-locking would work. Perhaps longer term there needs to be a way to have the kernel shut its NFS locking attempts off, just if there is a userland NFS such as Gluster's running. Meanwhile can enough of NFS be taken out of a custom kernel to allow Gluster to lock? Thanks, Whit On Tue, Jul 17, 2012 at 08:03:03AM -0400, Rajesh Amaravathi wrote: it should be possible to mount another kernel export with -o nolock option and compile kernel on it. I'm just guessing when you mount with nolock option, we are mounting for mostly read purposes and not for critical writes. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Whit Blauvelt whit.glus...@transpect.com To: Rajesh Amaravathi raj...@redhat.com Cc: David Coulson da...@davidcoulson.net, Gluster General Discussion List gluster-users@gluster.org Sent: Monday, July 16, 2012 9:56:28 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Try 3.3.0 - 3.2.6 has issues with NFS in general (memory leaks, etc). -Original Message- From: Tomasz Chmielewski man...@wpkg.org Date: Thursday, 12 July 2012 5:56 PM To: Gluster General Discussion List gluster-users@gluster.org Subject: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Hi, are NFS mounts made on a single server (i.e. where glusterd is running) supposed to be stable (with gluster 3.2.6)? I'm using the following line in /etc/fstab: localhost:/sites /var/ftp/sites nfs _netdev,mountproto=tcp,nfsvers=3,bg 0 0 The problem is, after some time (~1-6 hours), I'm no longer able to access this mount. dmesg says: [49609.832274] nfs: server localhost not responding, still trying [49910.639351] nfs: server localhost not responding, still trying [50211.446433] nfs: server localhost not responding, still trying What's worse, whenever this happens, *all* other servers in the cluster (it's a 10-server distributed volume) will destabilise - their load average will grow, and eventually their gluster mount becomes unresponsive, too (other servers use normal gluster mounts). At this point, I have to kill all gluster processes, start glusterd again, mount (on servers using gluster mount). Is it expected behaviour with gluster and NFS mounts on localhost? Can it be caused by some kind of deadlock? Any workarounds? -- Tomasz Chmielewski http://www.ptraveler.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On 07/13/2012 02:59 PM, James Kahn wrote: Try 3.3.0 - 3.2.6 has issues with NFS in general (memory leaks, etc). Upgrading to 3.3.0 would be quite a big adventure to me (production site, lots of traffic etc.). But I guess it would be justified, if it really fixes this bug. The issue was reported earlier, but I don't see any references it was fixed in 3.3.0: Deadlock happens when writing a file big enough to fill the filesystem cache and kernel is trying to flush it to free some memory for glusterfsd which needs memory to commit some filesystem blocks to free some memory for glusterfsd... http://gluster.org/pipermail/gluster-users/2011-January/006477.html https://bugzilla.redhat.com/show_bug.cgi?id=GLUSTER-2320 -- Tomasz Chmielewski http://www.ptraveler.com -Original Message- From: Tomasz Chmielewski man...@wpkg.org Date: Thursday, 12 July 2012 5:56 PM To: Gluster General Discussion List gluster-users@gluster.org Subject: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Hi, are NFS mounts made on a single server (i.e. where glusterd is running) supposed to be stable (with gluster 3.2.6)? I'm using the following line in /etc/fstab: localhost:/sites /var/ftp/sites nfs _netdev,mountproto=tcp,nfsvers=3,bg 0 0 The problem is, after some time (~1-6 hours), I'm no longer able to access this mount. dmesg says: [49609.832274] nfs: server localhost not responding, still trying [49910.639351] nfs: server localhost not responding, still trying [50211.446433] nfs: server localhost not responding, still trying What's worse, whenever this happens, *all* other servers in the cluster (it's a 10-server distributed volume) will destabilise - their load average will grow, and eventually their gluster mount becomes unresponsive, too (other servers use normal gluster mounts). At this point, I have to kill all gluster processes, start glusterd again, mount (on servers using gluster mount). Is it expected behaviour with gluster and NFS mounts on localhost? Can it be caused by some kind of deadlock? Any workarounds? -- Tomasz Chmielewski http://www.ptraveler.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Original Message - From: Tomasz Chmielewski man...@wpkg.org To: James Kahn jk...@idea11.com.au Cc: Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 1:51:15 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 07/13/2012 02:59 PM, James Kahn wrote: Try 3.3.0 - 3.2.6 has issues with NFS in general (memory leaks, etc). Upgrading to 3.3.0 would be quite a big adventure to me (production site, lots of traffic etc.). But I guess it would be justified, if it really fixes this bug. The issue was reported earlier, but I don't see any references it was fixed in 3.3.0: Deadlock happens when writing a file big enough to fill the filesystem cache and kernel is trying to flush it to free some memory for glusterfsd which needs memory to commit some filesystem blocks to free some memory for glusterfsd... http://gluster.org/pipermail/gluster-users/2011-January/006477.html https://bugzilla.redhat.com/show_bug.cgi?id=GLUSTER-2320 This is a problem generic to fuse/userspace filesystems. Also, in 3.3, since we have NLM implemented to provide locking for NFS, Its not possible to mount from a system which has glusterd(more precisely, a Gluster NFS process) running since both kernel nfs and gNFS will try to register for NLM v4 with portmapper. -- Tomasz Chmielewski http://www.ptraveler.com -Original Message- From: Tomasz Chmielewski man...@wpkg.org Date: Thursday, 12 July 2012 5:56 PM To: Gluster General Discussion List gluster-users@gluster.org Subject: [Gluster-users] NFS mounts with glusterd on localhost - reliable ornot? Hi, are NFS mounts made on a single server (i.e. where glusterd is running) supposed to be stable (with gluster 3.2.6)? I'm using the following line in /etc/fstab: localhost:/sites /var/ftp/sites nfs _netdev,mountproto=tcp,nfsvers=3,bg 0 0 The problem is, after some time (~1-6 hours), I'm no longer able to access this mount. dmesg says: [49609.832274] nfs: server localhost not responding, still trying [49910.639351] nfs: server localhost not responding, still trying [50211.446433] nfs: server localhost not responding, still trying What's worse, whenever this happens, *all* other servers in the cluster (it's a 10-server distributed volume) will destabilise - their load average will grow, and eventually their gluster mount becomes unresponsive, too (other servers use normal gluster mounts). At this point, I have to kill all gluster processes, start glusterd again, mount (on servers using gluster mount). Is it expected behaviour with gluster and NFS mounts on localhost? Can it be caused by some kind of deadlock? Any workarounds? -- Tomasz Chmielewski http://www.ptraveler.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On 07/13/2012 05:08 PM, Rajesh Amaravathi wrote: The issue was reported earlier, but I don't see any references it was fixed in 3.3.0: Deadlock happens when writing a file big enough to fill the filesystem cache and kernel is trying to flush it to free some memory for glusterfsd which needs memory to commit some filesystem blocks to free some memory for glusterfsd... http://gluster.org/pipermail/gluster-users/2011-January/006477.html https://bugzilla.redhat.com/show_bug.cgi?id=GLUSTER-2320 This is a problem generic to fuse/userspace filesystems. Also, in 3.3, since we have NLM implemented to provide locking for NFS, Its not possible to mount from a system which has glusterd(more precisely, a Gluster NFS process) running since both kernel nfs and gNFS will try to register for NLM v4 with portmapper. Hmm, this is sad. gluster/fuse mounts perform very poor for me. On a 10-server distributed gluster setup, when put to a lot of load, I'm able to serve (each server has 1xGbit NIC dedicated for gluster traffic and 1xGbit NIC dedicated for external traffic, webservers): - around 100 Mbit/s with gluster/fuse mounts to glusterd on localhost, - around 300 Mbit/s with NFS mounts to glusterd on localhost, - around 500 Mbit/s with NFS mounts to glusterd on localhost, and fsc/cachefilesd. Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! -- Tomasz Chmielewski http://www.ptraveler.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On 07/13/2012 05:46 PM, David Coulson wrote: On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. OK, but it deadlocks when we write, unless we use sync option, which hurts the performance. As mentioned by Rajesh Amaravathi: This is a problem generic to fuse/userspace filesystems. Mid term, are there any plans to deal with this problem? I understand the fix may need to be partially done on the kernel side, but it would be very beneficial to glusterfs. -- Tomasz Chmielewski http://www.ptraveler.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
I hope you do realize that two NLM implementations of the same version cannot operate simultaneously in the same machine. I really look forward to a solution to make this work, that'd be something. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Rajesh Amaravathi raj...@redhat.com Cc: Tomasz Chmielewski man...@wpkg.org, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 5:28:04 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? Was that introduced by the same person who thought that binding to sequential ports down from 1024 was a good idea? Considering how hard RedHat was pushing Gluster at the Summit a week or two ago, it seems like they're making it hard for people to really implement it in any capacity other than their Storage Appliance product. Luckily I don't need locking yet, but I suppose RedHat will be happy when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) David On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: Actually, if you want to mount *any* nfs volumes(of Gluster) OR exports (of kernel-nfs-server), you cannot do it with locking on a system where a glusterfs(nfs process) is running(since 3.3.0). However, if its ok to mount without locking, then you should be able to do it on localhost. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: David Coulson da...@davidcoulson.net To: Tomasz Chmielewski man...@wpkg.org Cc: Rajesh Amaravathi raj...@redhat.com, Gluster General Discussion List gluster-users@gluster.org Sent: Friday, July 13, 2012 3:16:38 PM Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not? On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: Killing the option to use NFS mounts on localhost is certainly quite the opposite to my performance needs! He was saying you can't run kernel NFS server and gluster NFS server at the same time, on the same host. There is nothing stopping you from mounting localhost:/volume on all your boxes. That is exactly how our 3.2.5 and 3.3.0 environments access volumes for the performance reasons you identified. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
