Hi Pavel, both clients and server were updated (we always have a consistent environment), and we came from u6, to which we updated directly from a vanilla Osol 2009.06 before. The Readmes don't list any NFS patches there, so I suspect that our IDR30 patch carries unwanted changes from Solaris u8 into Opensolaris U7 which trigger this problem. Since we don't use Solaris 10, I cannot confirm that 10u7 did not have that problem, I just concluded it from the initial post here (but that conclusion maybe wrong, I admit). We currently snoop the problem and catched some clues (maybe): Shortly before, we see CB_NULL and NULL4 exchanges, seemingly as a result of a client renewal (we do not catch everything before), both server and client seem to check their partners callback capabilities:
imksunxxx -> imksunyyy NFS C 4 (setclientid ) PUTROOTFH GETATTR 400 0 SETCLIENTID Prog=1073741826 ID=tcp Addr=192.168.2.151.156.34 CBID=1073741... imksunyyy -> imksunxxx NFS R 4 (setclientid ) NFS4_OK PUTROOTFH NFS4_OK GETATTR NFS4_OK SETCLIENTID NFS4_OK CL=3144b6920ee CFV=00002AE100000314 imksunxxx -> imksunyyy NFS C 4 (sclntid_conf) SETCLIENTID_CONFIRM CL=3144b6920ee CFV=00002AE100000314 imksunyyy -> imksunxxx NFS R 4 (sclntid_conf) NFS4_OK SETCLIENTID_CONFIRM NFS4_OK imksunyyy -> imksunxxx NFS C CB_NULL imksunxxx -> imksunyyy NFS R CB_NULL imksunxxx -> imksunyyy TCP D=2049 S=772 Ack=2315081869 Seq=1684524332 Len=0 Win=64074 Options=<nop,nop,tstamp 449054982 188906496> imksunyyy -> imksunxxx TCP D=39970 S=46007 Ack=2765004476 Seq=2079880454 Len=0 Win=64074 Options=<nop,nop,tstamp 188906503 449054975> imksunxxx -> imksunyyy TCP D=2049 S=35143 Syn Seq=3666179320 Len=0 Win=64057 Options=<mss 1460,nop,nop,tstamp 449055009 0,nop,wscale 4,nop,nop,sackOK> imksunyyy -> imksunxxx TCP D=35143 S=2049 Syn Ack=3666179321 Seq=3064768973 Len=0 Win=64074 Options=<nop,nop,tstamp 188906530 449055009,mss 1460,nop,wscale 4,nop,nop,sackOK> imksunxxx -> imksunyyy TCP D=2049 S=35143 Ack=3064768974 Seq=3666179321 Len=0 Win=64074 Options=<nop,nop,tstamp 449055009 188906530> imksunxxx -> imksunyyy NFS C NULL4 imksunyyy -> imksunxxx TCP D=35143 S=2049 Ack=3666179365 Seq=3064768974 Len=0 Win=64074 Options=<nop,nop,tstamp 188906530 449055009> imksunyyy -> imksunxxx NFS R NULL4 imksunxxx -> imksunyyy TCP D=2049 S=35143 Ack=3064769002 Seq=3666179365 Len=0 Win=64074 Options=<nop,nop,tstamp 449055009 188906530> imksunxxx -> imksunyyy TCP D=2049 S=35143 Fin Ack=3064769002 Seq=3666179365 Len=0 Win=64074 Options=<nop,nop,tstamp 449055009 188906530> imksunyyy -> imksunxxx TCP D=35143 S=2049 Ack=3666179366 Seq=3064769002 Len=0 Win=64074 Options=<nop,nop,tstamp 188906530 449055009> imksunyyy -> imksunxxx TCP D=35143 S=2049 Fin Ack=3666179366 Seq=3064769002 Len=0 Win=64074 Options=<nop,nop,tstamp 188906530 449055009> imksunxxx -> imksunyyy TCP D=2049 S=35143 Ack=3064769003 Seq=3666179366 Len=0 Win=64074 Options=<nop,nop,tstamp 449055009 188906530> ****** here starts the reopen mess, clients hang ********** imksunxxx -> imksunyyy NFS C 4 (reopen ) PUTFH FH=92CA OPEN OT=NC SQ=21955 CT=P DT=N AC=R DN=N OO=0670 GETFH GETATTR 10011a b0a23a imksunyyy -> imksunxxx NFS R 4 (reopen ) NFS4ERR_NO_GRACE PUTFH NFS4_OK OPEN NFS4ERR_NO_GRACE imksunxxx -> imksunyyy NFS C 4 (reopen ) PUTFH FH=95AC OPEN start_version OT=NC SQ=21956 CT=N AC=R DN=N OO=0670 GETFH GETATTR 10011a b0a23a imksunyyy -> imksunxxx NFS R 4 (reopen ) NFS4_OK PUTFH NFS4_OK OPEN NFS4_OK ST=150F:10979 RF=PL DT=N GETFH NFS4_OK FH=92CA GETATTR NFS4_OK imksunxxx -> imksunyyy NFS C 4 (reopen ) PUTFH FH=92CA OPEN OT=NC SQ=21955 CT=P DT=N AC=R DN=N OO=07D1 GETFH GETATTR 10011a b0a23a imksunyyy -> imksunxxx NFS R 4 (reopen ) NFS4ERR_NO_GRACE PUTFH NFS4_OK OPEN NFS4ERR_NO_GRACE Another one here, this time after the reopen: imksunzzz -> local-yyy NFS C 4 (reopen ) PUTFH FH=92CA OPEN OT=NC SQ=42131 CT=P DT=N AC=R DN=N OO=065A GETFH GETATTR 10011a b0a23a local-yyy -> imksunzzz NFS R 4 (reopen ) NFS4ERR_NO_GRACE PUTFH NFS4_OK OPEN NFS4ERR_NO_GRACE imksunzzz -> local-yyy NFS C 4 (reopen ) PUTFH FH=95AC OPEN start_version OT=NC SQ=42132 CT=N AC=R DN=N OO=065A GETFH GETATTR 10011a b0a23a local-yyy -> imksunzzz NFS R 4 (reopen ) NFS4_OK PUTFH NFS4_OK OPEN NFS4_OK ST=1B1B:21067 RF=PL DT=N GETFH NFS4_OK FH=92CA GETATTR NFS4_OK imksunzzz -> local-yyy NFS C 4 (open ) PUTFH FH=95AC OPEN start_version OT=NC SQ=1 CT=N AC=R DN=N OO=0F54 GETFH GETATTR 10011a b0a23a local-yyy -> imksunzzz NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID imksunzzz -> local-yyy NFS C 4 (setclientid ) PUTROOTFH GETATTR 400 0 SETCLIENTID Prog=1073741826 ID=tcp Addr=192.168.2.157.145.153 CBID=107374... local-yyy -> imksunzzz NFS R 4 (setclientid ) NFS4_OK PUTROOTFH NFS4_OK GETATTR NFS4_OK SETCLIENTID NFS4_OK CL=3154b6920ee CFV=0000524A00000315 imksunzzz -> local-yyy NFS C 4 (sclntid_conf) SETCLIENTID_CONFIRM CL=3154b6920ee CFV=0000524A00000315 local-yyy -> imksunzzz NFS R 4 (sclntid_conf) NFS4_OK SETCLIENTID_CONFIRM NFS4_OK local-yyy -> imksunzzz NFS C CB_NULL imksunzzz -> local-yyy NFS R CB_NULL and a third one with a couple of NFS4ERR_STALE_CLIENTID: imksunwww -> local-uuu NFS C 4 (setclientid ) PUTROOTFH GETATTR 400 0 SETCLIENTID Prog=1073741825 ID=tcp Addr=192.168.2.158.211.129 CBID=107374... local-uuu -> imksunwww NFS R 4 (setclientid ) NFS4_OK PUTROOTFH NFS4_OK GETATTR NFS4_OK SETCLIENTID NFS4_OK CL=784b2b88bf CFV=000017B600000078 imksunwww -> local-uuu NFS C 4 (sclntid_conf) SETCLIENTID_CONFIRM CL=784b2b88bf CFV=000017B600000078 local-uuu -> imksunwww NFS R 4 (sclntid_conf) NFS4_OK SETCLIENTID_CONFIRM NFS4_OK local-uuu -> imksunwww NFS C CB_NULL imksunwww -> local-uuu NFS R CB_NULL imksunwww -> local-uuu TCP D=2049 S=1014 Ack=2856458498 Seq=2503417192 Len=0 Win=64074 Options=<nop,nop,tstamp 419650675 417669232> local-uuu -> imksunwww TCP D=54145 S=44819 Ack=696198650 Seq=3514023629 Len=0 Win=64074 Options=<nop,nop,tstamp 417669239 419650669> imksunwww -> local-uuu NFS C 4 (reopen ) PUTFH FH=D915 OPEN OT=NC SQ=12381 CT=P DT=N AC=R DN=N OO=0053 GETFH GETATTR 10011a b0a23a local-uuu -> imksunwww NFS R 4 (reopen ) NFS4ERR_NO_GRACE PUTFH NFS4_OK OPEN NFS4ERR_NO_GRACE imksunwww -> local-uuu NFS C 4 (reopen ) PUTFH FH=CDFC OPEN sge_shepherd OT=NC SQ=12382 CT=N AC=R DN=N OO=0053 GETFH GETATTR 10011a b0a23a local-uuu -> imksunwww NFS R 4 (reopen ) NFS4_OK PUTFH NFS4_OK OPEN NFS4_OK ST=1B1B:6070 RF=PL DT=N GETFH NFS4_OK FH=D915 GETATTR NFS4_OK imksunwww -> local-uuu NFS C 4 (reopen ) PUTFH FH=D8C0 OPEN OT=NC SQ=12163 CT=P DT=N AC=R DN=N OO=005A GETFH GETATTR 10011a b0a23a local-uuu -> imksunwww NFS R 4 (reopen ) NFS4ERR_NO_GRACE PUTFH NFS4_OK OPEN NFS4ERR_NO_GRACE imksunwww -> local-uuu NFS C 4 (reopen ) PUTFH FH=CDFC OPEN sge_execd OT=NC SQ=12164 CT=N AC=R DN=N OO=005A GETFH GETATTR 10011a b0a23a local-uuu -> imksunwww NFS R 4 (reopen ) NFS4_OK PUTFH NFS4_OK OPEN NFS4_OK ST=1583:6071 RF=PL DT=N GETFH NFS4_OK FH=D8C0 GETATTR NFS4_OK imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0C0D GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0C5C GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0C1B GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0C69 GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0FF5 GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0C70 GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0CAA GETFH GETATTR 10011a b0a23a imksunwww -> local-uuu NFS C 4 (open ) PUTFH FH=C7A4 OPEN settings.sh OT=NC SQ=1 CT=N AC=R DN=N OO=0FE3 GETFH GETATTR 10011a b0a23a local-uuu -> imksunwww TCP D=1014 S=2049 Ack=2503418560 Seq=2856459314 Len=0 Win=64074 Options=<nop,nop,tstamp 417669242 419650679> local-uuu -> imksunwww TCP D=1014 S=2049 Ack=2503419056 Seq=2856459314 Len=0 Win=64074 Options=<nop,nop,tstamp 417669242 419650679> local-uuu -> imksunwww TCP D=1014 S=2049 Ack=2503419552 Seq=2856459314 Len=0 Win=64074 Options=<nop,nop,tstamp 417669242 419650679> local-uuu -> imksunwww TCP D=1014 S=2049 Ack=2503420048 Seq=2856459314 Len=0 Win=64074 Options=<nop,nop,tstamp 417669242 419650679> local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID imksunwww -> local-uuu TCP D=2049 S=1014 Ack=2856459450 Seq=2503420048 Len=0 Win=64074 Options=<nop,nop,tstamp 419650680 417669242> local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID imksunwww -> local-uuu TCP D=2049 S=1014 Ack=2856459586 Seq=2503420048 Len=0 Win=64074 Options=<nop,nop,tstamp 419650680 417669242> local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID imksunwww -> local-uuu TCP D=2049 S=1014 Ack=2856459722 Seq=2503420048 Len=0 Win=64074 Options=<nop,nop,tstamp 419650680 417669242> local-uuu -> imksunwww NFS R 4 (open ) NFS4ERR_STALE_CLIENTID PUTFH NFS4_OK OPEN NFS4ERR_STALE_CLIENTID imksunwww -> local-uuu TCP D=2049 S=1014 Ack=2856459858 Seq=2503420048 Len=0 Win=64074 Options=<nop,nop,tstamp 419650680 417669242> -- This message posted from opensolaris.org
