[nfs-discuss] svc_cots_kdup no slots free
Hello Spencer,
Monday, June 19, 2006, 5:47:21 PM, you wrote:
Hello Spencer,
Thursday, June 15, 2006, 11:48:57 PM, you wrote:
SS On Thu, Robert Milkowski wrote:
Hello nfs-discuss,
Sometimes on nfs servers I get below messages and then I have
performance problems. What are they about and what can I do about
this?
rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots
free
last message repeated 700353 times
SS This is an error from the NFS server when it attempts to place
SS a request into the duplicate request cache and finds that
SS all of the ones in the duplicate request table are in progress.
SS The maximum for the duplicate request cache is 1024.
SS The most effective way to increase the value is /etc/system
SS and the variable would be: rpcmod:cotsmaxdupreqs
I set it to 8192 and see some problems with nfsd+zfs - like almost all
threads hanging in ZFS and no actual IOs are happening while I can
issue IOs to the same zfs filesystems locally (see nfsd threads hang
in ZFS on zfs-discuss list).
I did set rpcmod:cotsmaxdupreqs=8192
rpcmod:maxdupreqs=8192
Maybe it's too high?
SS Are you still receiving the original error message your reported
SS after making these changes? (the no slots free error)
No more errors - thanks for hint.
SS If not, then the problem is in the local filesystem.
SS What you have done is just allowed the NFS server to use the
SS threads that it can create and not reject with error the incoming
SS requests.
ok, thank you.
--
Best regards,
Robertmailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
[nfs-discuss] svc_cots_kdup no slots free
Hello Spencer, Thursday, June 15, 2006, 11:48:57 PM, you wrote: SS On Thu, Robert Milkowski wrote: Hello nfs-discuss, Sometimes on nfs servers I get below messages and then I have performance problems. What are they about and what can I do about this? rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots free last message repeated 700353 times SS This is an error from the NFS server when it attempts to place SS a request into the duplicate request cache and finds that SS all of the ones in the duplicate request table are in progress. SS The maximum for the duplicate request cache is 1024. SS The most effective way to increase the value is /etc/system SS and the variable would be: rpcmod:cotsmaxdupreqs And when it finds that all requests are 'in progress' - then what? Thank you for info. btw: it looks like A LOT of messages are being sent in that case to syslog and that is perhaps one of the things which kills performance and makes whole server unresponsive. -- Best regards, Robertmailto:rmilkowski at task.gda.pl http://milek.blogspot.com
[nfs-discuss] svc_cots_kdup no slots free
FYI - this reminds me of a discussion I had once regarding bug 4321293 (Unable to exit svc_cots_kdup() all cache marked DUP_INPROGRESS). http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=4321293 That bug is fixed but someone thought they were still hitting it, turned out they were just hitting the limit you are seeing. The cache is dynamically sized up to the limit of cotsmaxdupreqs (connection orientated, eg TCP) and maxdupreqs (connectionless, eg UDP). The current behaviour is the expected effect of the fix to bug 4321293 in that previously it would 'soft hang', now it just logs the error and returns an error to the client. If the NFS server is stuck processing incoming requests so that it never replies then the incoming requests will fill up the dup request cache. In other words, if there's some other problem that is preventing the NFS server from replying to requests then you will also see this behaviour. In that case the dup syslog messages are a side effect of the underlying problem, ie the NFS server not being able to process requests at all, or at least fast enough. -- Peter
[nfs-discuss] svc_cots_kdup no slots free
Hello nfs-discuss, Sometimes on nfs servers I get below messages and then I have performance problems. What are they about and what can I do about this? rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots free last message repeated 700353 times -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
[nfs-discuss] svc_cots_kdup no slots free
On Thu, Robert Milkowski wrote: Hello nfs-discuss, Sometimes on nfs servers I get below messages and then I have performance problems. What are they about and what can I do about this? rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots free last message repeated 700353 times This is an error from the NFS server when it attempts to place a request into the duplicate request cache and finds that all of the ones in the duplicate request table are in progress. The maximum for the duplicate request cache is 1024. The most effective way to increase the value is /etc/system and the variable would be: rpcmod:cotsmaxdupreqs Spencer
[nfs-discuss] svc_cots_kdup no slots free
On Jun 15, 2006, at 5:32 PM, Robert Milkowski wrote: And when it finds that all requests are 'in progress' - then what? When the non-idempotent reply can not be placed in the duplicate request cache, we call svcerr_systemerr() which places SYSTEM_ERR in the reply ar_stat; This has the effect to cause the client to re-drive the call, in the hopes that the duplicate request cache congestion has passed. Thank you for info. btw: it looks like A LOT of messages are being sent in that case to syslog and that is perhaps one of the things which kills performance and makes whole server unresponsive. The unresponsiveness is most likely to be due to the clients having to re-drive the requests. Increasing the size of the duplicate request cache as Spencer suggests and see if that helps. -- Robert.
