Hello @misc, Just some further information on this.
When I stop relayd and enter the pf rules like relayd does with its anchor, then it's - more or less - working as expected. When I start an upload within an SFTP session and failover, then the session is "stalled" nearly forever. When I set the tcp.established to 60 (instead of 600), then the "state" times out but the SFTP client starts reconnecting after a while (about 2-3 minutes) and the sessions keeps running. So it looks like relayd is "terminating" the session when carp fails over. With relayd and doing a carp failover, I get an Broken pipe. Connection reset by peer immediately. I just want to know, if this is a normal behaviour with this setup. Thanks. Robert > Gesendet: Mittwoch, 12. Oktober 2016 um 14:21 Uhr > Von: "Robert Paschedag" <[email protected]> > An: "Robert Paschedag" <[email protected]> > Cc: [email protected] > Betreff: Aw: Re: pf on carp backup resets connection after failover > > This time it should be better. Again sorry.. > > > Hi all, > > basically, if have exactly this problem already described here > (https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjxwfY) . > But because there is no answer since 2009, I'll give it a try. > > The setup of the 2 servers is also the same as in the other thread > only exception is, that my boxes are behind a "master" firewall > which I do not manage. > > I have 2 OpenBSD 6.0 servers that should just act as a load balancer > for SFTP connections. We use DSR mode because huge files get > downloaded from the SFTP servers and don't want the "load" to > pass completly through the OpenBSD load balancers. > > Everything is working as long as I don't do a failover to the backup system. > In this situation, I see, that the "new" carp master "resets" the connection > of the client. Immediatly opening a new SFTP sessions then works as > expected through the "new" carp master. > > This is my /etc/pf.conf (identical on both). Still testing.. > > # cat /etc/pf.conf > carp_if = "vmx0" > sync_if = "vmx1" > # already allow pfsync and carp protocols > pass quick on $sync_if proto pfsync keep state (no-sync) > pass on $carp_if proto carp keep state (no-sync) > # allow relayd to communicate with pf and set rules > anchor "relayd/*" > > And this is the relayd.conf > > log updates > prefork 5 > fx_vip = "VIP" > table <fxhosts> { > "host1" > "host2" > } > redirect FX-SFTP { > listen on $fx_vip port 22 interface vmx0 > route to <fxhosts> check tcp interface vmx0 > sticky-address > } > > This is the "ruleset" (identical on both) after reloading pf > > # pfctl -a '*' -s rules > pass quick on vmx1 proto pfsync all keep state (no-sync) > pass on vmx0 proto carp all keep state (no-sync) > anchor "relayd/*" all { > anchor "FX-SFTP" all { > pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port = 22 > flags any keep state (sloppy, tcp.established 600) > route-to <FX-SFTP>@vmx0 round-robin sticky-address > } > } > > When the first connection is made, I see the state on the > backup carp machine. But with slightly different content. > > This is on "master" > > all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED > [0 + 1] [946261580 + 2] > age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2, rule 2, sloppy > id: 57fbd5520000a2b4 creatorid: d4cdd00a > > "expires" is 10 minutes (tcp.established 600) and I see the anchor and rule > which generated state > > This in on "backup" > > all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED > [0 + 1] [946261580 + 2] > age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy > id: 57fbd5520000a2b4 creatorid: d4cdd00a > > expires is 1 day (?) and "backup" did not yet see any packes. > > Now, how can I get this to work, so the sessions won't be terminated > in case of a failover. > > Every help will be appreciated. > > Kind regards, > Robert > > > > Gesendet: Mittwoch, 12. Oktober 2016 um 14:18 Uhr > > Von: "Robert Paschedag" <[email protected]> > > An: [email protected] > > Betreff: Re: pf on carp backup resets connection after failover > > > > Sorry for this bad web mailer formatting. I didn't want that.Am 12.10.2016 2:08 nachm. schrieb Robert Paschedag <[email protected]>: > > > > > > Hi all, basically, if have exactly this problem already described here(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjx wfY).But > > > because there is no answer since 2009, I'll give it a try. The setup of > > > the 2 servers is also the same as in the other threadonly exception is, > > > that my boxes are behind a "master" firewallwhich I do not manage. I have > > > 2 OpenBSD 6.0 servers that should just act as a load balancerfor SFTP > > > connections. We use DSR mode because huge files getdownloaded from the > > > SFTP servers and don't want the "load" topass completly through the > > > OpenBSD load balancers. Everything is working as long as I don't do a > > > failover to the backup system.In this situation, I see, that the "new" > > > carp master "resets" the connectionof the client. Immediatly opening a > > > new SFTP sessions then works asexpected through the "new" carp master. > > > This is my /etc/pf.conf (identical on both). Still testing.. # cat > > > /etc/pf.conf > > > carp_if = "vmx0" > > > sync_if = "vmx1"# already allow pfsync and carp protocols > > > pass quick on $sync_if proto pfsync keep state (no-sync) > > > pass on $carp_if proto carp keep state (no-sync)# allow relayd to > > > communicate with pf and set rules > > > anchor "relayd/*" And this is the relayd.conf log updates > > > prefork 5fx_vip = "VIP"table <fxhosts> { > > > "host1" > > > "host2" > > > }redirect FX-SFTP { > > > listen on $fx_vip port 22 interface vmx0 > > > route to <fxhosts> check tcp interface vmx0 > > > sticky-address > > > } > > > This is the "ruleset" (identical on both) after reloading pf # pfctl -a > > > '*' -s rules > > > pass quick on vmx1 proto pfsync all keep state (no-sync) > > > pass on vmx0 proto carp all keep state (no-sync) > > > anchor "relayd/*" all { > > > anchor "FX-SFTP" all { > > > pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port = > > > 22 flags any keep state (sloppy, tcp.established 600) route-to > > > <FX-SFTP>@vmx0 round-robin sticky-address > > > } > > > } When the first connection is made, I see the state on thebackup carp > > > machine. But with slightly different content. This is on "master" all tcp > > > VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED > > > [0 + 1] [946261580 + 2] > > > age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2, > > > rule 2, sloppy id: 57fbd5520000a2b4 creatorid: d4cdd00a "expires" is 10 > > > minutes (tcp.established 600) and I see the anchor and rulewhich > > > generated state This in on "backup" all tcp VIP:22 <- CLIENT:43334 > > > ESTABLISHED:ESTABLISHED > > > [0 + 1] [946261580 + 2] > > > age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy > > > id: 57fbd5520000a2b4 creatorid: d4cdd00a expires is 1 day (?) and > > > "backup" did not yet see any packes. Now, how can I get this to work, so > > > the sessions won't be terminatedin case of a failover. Every help will be > > > appreciated. Kind regards,Robert
