Re: Need Advice for Tuning NFS to place nice with a Netapp
Nicole, On 02/08/06, N. Harrington [EMAIL PROTECTED] wrote: Hi I have several web servers that are attached to a Netapp (network appliance) unit via NFS-3. A few servers are 5.5 and a few are 6.1 for comparison testing. All seem to have lousy performance. We have a similar setup and it runs smoothly. Can you define lousy performance ? Can you give more details on your network? Are you using Gig ethernet? And over what medium? Can you also try just copying a 100MB file from the filer to one of the web servers and record the time? Are you running nfsiod? When going through the issues with Netapp, the reasons given were that we have too many GettAdr/Lookup requests compared to actual reads. So all the NFS IOPS are being used up by these requests. As soon as the webservers get busy, requests pile up. I have tried everything I can think of. The web servers are even mounted read only with no help. My current mount options are: filer:/vol/fvol31/home/13/13 nfs ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0 Mounting noatime for web servers is a good idea but... your noatime option has no effect on NFS mounts (check out the mount man page). You need vol options no_atime_update on the NetApp. Any advice for sysctl tunes or anything else would be much appreciatted! Thanks Nicole One last thing - are you female?! In a UNIX newsgroup?! Frem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need Advice for Tuning NFS to place nice with a Netapp
--- Freminlins [EMAIL PROTECTED] wrote: Nicole, On 02/08/06, N. Harrington [EMAIL PROTECTED] wrote: Hi I have several web servers that are attached to a Netapp (network appliance) unit via NFS-3. A few servers are 5.5 and a few are 6.1 for comparison testing. All seem to have lousy performance. We have a similar setup and it runs smoothly. Cool! Can you share with me what sort of settings you use on your boxes? sysctl/kerneltunes/mount options? It has taken me a over a month to even get to speak to someone high enough up he food chain at Netapp to not say FreeBSD - that's a version of Linux right? Can you define lousy performance ? The web server replies (using either Apache and Lighthttpd) seem to max out at about 17mb/s. Response time for the web server will rise gradually, then suddenly become 10-20seconds for a reply. Much like a backup on a highway. They claim that the netapp unit is spending too much time dealing with file information IOPS than actual transfer of files. However even on a non in-use server, if I make a request for a file, that heavy file access seems normal. IE: GtAttr Lookup Rdlink Read Write Rename Access Rddir 0 0 0 0 0 0 0 0 248 160 0 4 0 0236 0 0 0 0 0 0 0 0 0 Can you give more details on your network? Are you using Gig ethernet? And over what medium? Yes, 4X GigE from the filer via a Vif and trunking on the switch. A nice 10Gb ready HP unit. I have asked if using the Vif and trunking could have any effects but been assured it should not. It does mean I cannot use jumbo frames. But since web pages and images are small, I don't think there would be any benefit. Can you also try just copying a 100MB file from the filer to one of the web servers and record the time? 9907187 bytes/sec for a 16M file. It will transfer in nanotime. So, I believe that eliminates network performance as an issue. Are you running nfsiod? Yes, I show 4 instances running. When going through the issues with Netapp, the reasons given were that we have too many GettAdr/Lookup requests compared to actual reads. So all the NFS IOPS are being used up by these requests. As soon as the webservers get busy, requests pile up. I have tried everything I can think of. The web servers are even mounted read only with no help. My current mount options are: filer:/vol/fvol31/home/13/13 nfs ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0 Mounting noatime for web servers is a good idea but... your noatime option has no effect on NFS mounts (check out the mount man page). You need vol options no_atime_update on the NetApp. Hmm. Drat. We have some web servers that do nothing but send out data, but some that are used for uploading and file manipulation. I will have to make sure that global of an option will not effect what they do. Any advice for sysctl tunes or anything else would be much appreciatted! Thanks Nicole One last thing - are you female?! In a UNIX newsgroup?! Yup :) Oh, and yes, I do play the drums :) Frem. Thanks for your assistance!! Nicole The Large Print Giveth And The Small Print Taketh Away -- Anon __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need Advice for Tuning NFS to place nice with a Netapp
On 02/08/06, N. Harrington [EMAIL PROTECTED] wrote: Cool! Can you share with me what sort of settings you use on your boxes? sysctl/kerneltunes/mount options? This may be a disappointment to you but... I didn't have to do anything :-( All I have is rw on the client. It has taken me a over a month to even get to speak to someone high enough up he food chain at Netapp to not say FreeBSD - that's a version of Linux right? It depends who you speak to. There are people at NetApp who know about FreeBSD. The web server replies (using either Apache and Lighthttpd) seem to max out at about 17mb/s. Response time for the web server will rise gradually, then suddenly become 10-20seconds for a reply. Much like a backup on a highway. They claim that the netapp unit is spending too much time dealing with file information IOPS than actual transfer of files. However even on a non in-use server, if I make a request for a file, that heavy file access seems normal. IE: GtAttr Lookup Rdlink Read Write Rename Access Rddir 0 0 0 0 0 0 0 0 248 160 0 4 0 0236 0 0 0 0 0 0 0 0 0 I would dispute NetApp's claim. NetApp filers are very capable at doing NFS operations. Static files tend to sit usefully in the buffer cache on web servers. So unless you are doing something really odd with your web servers I would tend to disagree with NetApp. I've just looked on one of our POP3 servers (mounting NetApps). POP3 causes far more random access than our web servers. As such it doesn't sit in the buffer cache very long. We have much higher figures than yours and absolutely no performance problems. Yes, 4X GigE from the filer via a Vif and trunking on the switch. A nice 10Gb ready HP unit. I have asked if using the Vif and trunking could have any effects but been assured it should not. It does mean I cannot use jumbo frames. But since web pages and images are small, I don't think there would be any benefit. There is nothing wrong in theory with that setup. But is may not be what you want. Try it with just one GigE interface. You're right - you probably don't need jumbo frames. Which interface does the HP unit have? Also, have a look at netstat -in. Are there any IErrs or OErrs or Coll? Paste the results here! 9907187 bytes/sec for a 16M file. It will transfer in nanotime. So, I believe that eliminates network performance as an issue. Well, not really. The figure above is showing 10MB a second. That's not quite Fast Ethernet speed. If you are pushing 17mb (I guess that's megabits) that's not really a problem though. I've just tested this on the same POP3 server above, using dd to write a file onto a NetApp and I get 10889359 a second. And this machine is busy. Also, it is mounting the NetApp over Fast Ethernet. Hmm. Drat. We have some web servers that do nothing but send out data, but some that are used for uploading and file manipulation. I will have to make sure that global of an option will not effect what they do. It is a per volume option. And frankly I've never seen much use for atime. It's useful sometimes, but not a lot. Can you also put in the output of nfsstat -W -c 2. Maybe it's best to put this up on the web somewhere as it's wide, and it's not easy to read in email. Let it run for a minute or so, and if possible do two runs. One during the OK time, the other during the problem time. I would go back to basics. One GigE interface. Just rw mount options, and start testing. By testing I mean measuring. NFS tuning is fiddly. I've been using NetApps with FreeBSD for 5 years. It is a good combination. Can you also post the output of sysctl -a|grep nfs. But don't start fiddling with them yet! One last thing - are you female?! In a UNIX newsgroup?! Yup :) Oh, and yes, I do play the drums :) Oh gawd. Whatever next? :-) Thanks for your assistance!! Nicole Frem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need Advice for Tuning NFS to place nice with a Netapp
On 8/1/2006 11:05 PM, N. Harrington wrote: My current mount options are: filer:/vol/fvol31/home/13/13 nfs ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0 for the same performance reason, mine is: netapp1:/vol/vol0/export /export nfs rw,-r=16384,-w=16384,-L 0 0 might be worth a try to add the -w. -- Jeremy Kister http://jeremy.kister.net./ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Need Advice for Tuning NFS to place nice with a Netapp
--- Jeremy Kister [EMAIL PROTECTED] wrote: On 8/1/2006 11:05 PM, N. Harrington wrote: My current mount options are: filer:/vol/fvol31/home/13/13 nfs ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0 for the same performance reason, mine is: netapp1:/vol/vol0/export /export nfs rw,-r=16384,-w=16384,-L 0 0 might be worth a try to add the -w. Jeremy Kister http://jeremy.kister.net./ Hi Jeremy Thanks for the suggestion. Would using the -w do anything for me since I am mounting read only? I will try 16384 again however, as they could not settle on wether 16k or 32k would be best. Thanks Nicole The Large Print Giveth And The Small Print Taketh Away -- Anon __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
