Re: [CentOS] [Solved] Excessive NFS operations
Maybe it is updating the access time on each read or something that causes the activity. It is either re-reading the files or checking mod times to determin if the local cached copy is valid. Either way, lots of traffic. And this was hundreds of ops/second? I need to ppoint out that the retieval script was running on a RHEL3 machine. The filer NFS load was registered from a CentOS5 machine, and the only connection between the two were one or more vnc sessions. I cannot explain what happened, other than the observation that the excessive NFS access from the CentOS machine stopped when we disabled that screensaver on RHEL3. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
At Fri, 2 Oct 2009 13:11:19 +0100 CentOS mailing list centos@centos.org wrote: lheck...@users.sourceforge.net writes: [...] A single CentOS 5.2 x86_64 machine here is overloading our NetApp filer with excessive NFS getattr, lookup and access operations. The weird thing is that the number of these operations increases over time. I have an mrtg graph (which I didn't want to attach here) showing e.g. 200 NFS Ops on Monday, measured with filer-mrtg, going up to, e.g. 1200 in a straight line within days. nfsstat -l on the filer proves beyond doubt that the load is caused by this particular machine. dstat shows me which NFS operations are causing it. Thanks for all the replies. I believe we have found the culprit. First, updating the CentOS kernel did not help. I am now 99% certain that the problem was caused by the XScreenSaver Phosphor screensaver running in one or more vnc sessions to RHEL3 machines on the CentOS5 desktop. The screensaver was customised to run a perl script in the user's account that generates random quotes. In any case, disabling this screensaver under RHEL3 appears to have solved our problem, with about 5 days' worth of monitoring data to support this. This is definitely a weird interaction, as neither the screensaver nor its components actually run on the CentOS machine. I have not checked whether any other activities in a vnc session cause similar behaviour. Where does the screensaver's data files (eg where are the quotes stored) live? If on the CentOS machine, then it is simply that the screensaver is making lots of NFS I/O operations. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
This is definitely a weird interaction, as neither the screensaver nor its components actually run on the CentOS machine. I have not checked whether any other activities in a vnc session cause similar behaviour. Where does the screensaver's data files (eg where are the quotes stored) live? If on the CentOS machine, then it is simply that the screensaver is making lots of NFS I/O operations. On the filer. But this retrieval script runs on the RHEL3 box(es). ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
At Fri, 2 Oct 2009 16:55:51 +0100 CentOS mailing list centos@centos.org wrote: This is definitely a weird interaction, as neither the screensaver nor its components actually run on the CentOS machine. I have not checked whether any other activities in a vnc session cause similar behaviour. Where does the screensaver's data files (eg where are the quotes stored) live? If on the CentOS machine, then it is simply that the screensaver is making lots of NFS I/O operations. On the filer. But this retrieval script runs on the RHEL3 box(es). The retrieval script is/was hitting on the file server to fetch the quotes. It probably was doing something dumb and not caching the datafile. This file access was beating on your file server. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
Robert Heller wrote: This is definitely a weird interaction, as neither the screensaver nor its components actually run on the CentOS machine. I have not checked whether any other activities in a vnc session cause similar behaviour. Where does the screensaver's data files (eg where are the quotes stored) live? If on the CentOS machine, then it is simply that the screensaver is making lots of NFS I/O operations. On the filer. But this retrieval script runs on the RHEL3 box(es). The retrieval script is/was hitting on the file server to fetch the quotes. It probably was doing something dumb and not caching the datafile. This file access was beating on your file server. Seems odd that caching wouldn't just happen naturally in the nfs client. Maybe it is updating the access time on each read or something that causes the activity. -- lesmikes...@gmail.com lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
At Fri, 02 Oct 2009 12:57:16 -0500 CentOS mailing list centos@centos.org wrote: Robert Heller wrote: This is definitely a weird interaction, as neither the screensaver nor its components actually run on the CentOS machine. I have not checked whether any other activities in a vnc session cause similar behaviour. Where does the screensaver's data files (eg where are the quotes stored) live? If on the CentOS machine, then it is simply that the screensaver is making lots of NFS I/O operations. On the filer. But this retrieval script runs on the RHEL3 box(es). The retrieval script is/was hitting on the file server to fetch the quotes. It probably was doing something dumb and not caching the datafile. This file access was beating on your file server. Seems odd that caching wouldn't just happen naturally in the nfs client. I am not sure if it even makes sense to cache NFS files on a nfs client -- how does the client know that the file might not have changed on the server? At the very least it has to check the file mod times on the server to be sure its local cache is valid. Maybe it is updating the access time on each read or something that causes the activity. It is either re-reading the files or checking mod times to determin if the local cached copy is valid. Either way, lots of traffic. -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] [Solved] Excessive NFS operations
Robert Heller wrote: Seems odd that caching wouldn't just happen naturally in the nfs client. I am not sure if it even makes sense to cache NFS files on a nfs client -- how does the client know that the file might not have changed on the server? At the very least it has to check the file mod times on the server to be sure its local cache is valid. Pretty much all filesytems cache, and would be unusably slow if they didn't. File attributes should be only a few seconds on NFS, but that should be enough to avoid killing your server. Maybe it is updating the access time on each read or something that causes the activity. It is either re-reading the files or checking mod times to determin if the local cached copy is valid. Either way, lots of traffic. And this was hundreds of ops/second? -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
