On Fri, Oct 21, 2016 at 10:56 PM, Nick Fisk <[email protected]> wrote: > > -----Original Message----- > > From: ceph-users [mailto:[email protected]] On Behalf > Of Haomai Wang > > Sent: 21 October 2016 15:40 > > To: Nick Fisk <[email protected]> > > Cc: [email protected] > > Subject: Re: [ceph-users] Ceph and TCP States > > > > > > > > On Fri, Oct 21, 2016 at 10:31 PM, Nick Fisk <mailto:[email protected]> > wrote: > > > -----Original Message----- > > > From: ceph-users [mailto:mailto:[email protected]] On > Behalf Of Haomai Wang > > > Sent: 21 October 2016 15:28 > > > To: Nick Fisk <mailto:[email protected]> > > > Cc: mailto:[email protected] > > > Subject: Re: [ceph-users] Ceph and TCP States > > > > > > > > > > > > On Fri, Oct 21, 2016 at 10:19 PM, Nick Fisk <mailto:mailto: > [email protected]> wrote: > > > Hi, > > > > > > I'm just testing out using a Ceph client in a DMZ behind a FW from the > main Ceph cluster. One thing I have noticed is that if the > > > state table on the FW is emptied maybe by restarting it or just > clearing the state table...etc. Then the Ceph client will hang for a > > > long time as the TCP session can no longer pass through the FW and > just gets blocked instead. > > > > > > This "FW" is linux firewall or hardware FW? > > > > PFSense running on dedicated HW. Eventually they will be in a HA pair so > states should persist, but trying to work around this for now. > > Bit annoying having CephFS lock hard for 15 minutes even though the > network connection only went down for a few seconds. > > > > hmm, I'm not familiar with this fw. And from my view, whether RST > packet sent is decided by FW. But I think you can try > > "/proc/sys/net/ipv4/tcp_keepalive_time", if FW reset tcp session, tcp > keepalive should detect and send a rst. > > Yeah I think that’s where the problem lies. Most Firewalls tend to > silently drop denied packets without sending RST's, so Ceph effectively > just thinks that its experiencing packet loss and will never retry until > the 15 minute timeout period is up. Am I right in thinking I can't tune > down this parameter for a CephFS kernel client as it doesn't use the > ceph.conf file? >
I think cephfs kernel client doesn't have any timeout behavior. 15mins timeout is triggered by server side I guess. So you can turn down this in server side. Keep in mind that a very low value will cause frequency connection lossy. > > > > > > > > > > > > I believe this behaviour can be adjusted by the "ms tcp read timeout" > setting to limit its impact, but wondering if anybody has any > > > other ideas. I'm also thinking of experimenting with either stateless > FW rules for Ceph or getting the FW to send back RST packets > > > instead of silently dropping packets. > > > > > > hmm, I think it depends on FW > > > > > > > > > Thanks, > > > Nick > > > > > > _______________________________________________ > > > ceph-users mailing list > > > mailto:mailto:[email protected] > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > _______________________________________________ > > ceph-users mailing list > > mailto:[email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
