Re: pf on carp backup resets connection after failover
Hello @misc,
Just some further information on this.
When I stop relayd and enter the pf rules like relayd does with
its anchor, then it's - more or less - working as expected.
When I start an upload within an SFTP session and failover, then
the session is "stalled" nearly forever. When I set the
tcp.established to 60 (instead of 600), then the "state" times
out but the SFTP client starts reconnecting after a while
(about 2-3 minutes) and the sessions keeps running.
So it looks like relayd is "terminating" the session when
carp fails over. With relayd and doing a carp failover, I get
an
Broken pipe. Connection reset by peer
immediately.
I just want to know, if this is a normal behaviour with this setup.
Thanks.
Robert
> Gesendet: Mittwoch, 12. Oktober 2016 um 14:21 Uhr
> Von: "Robert Paschedag"
> An: "Robert Paschedag"
> Cc: misc@openbsd.org
> Betreff: Aw: Re: pf on carp backup resets connection after failover
>
> This time it should be better. Again sorry..
>
>
> Hi all,
>
> basically, if have exactly this problem already described here
>
(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjxwfY)
.
> But because there is no answer since 2009, I'll give it a try.
>
> The setup of the 2 servers is also the same as in the other thread
> only exception is, that my boxes are behind a "master" firewall
> which I do not manage.
>
> I have 2 OpenBSD 6.0 servers that should just act as a load balancer
> for SFTP connections. We use DSR mode because huge files get
> downloaded from the SFTP servers and don't want the "load" to
> pass completly through the OpenBSD load balancers.
>
> Everything is working as long as I don't do a failover to the backup
system.
> In this situation, I see, that the "new" carp master "resets" the
connection
> of the client. Immediatly opening a new SFTP sessions then works as
> expected through the "new" carp master.
>
> This is my /etc/pf.conf (identical on both). Still testing..
>
> # cat /etc/pf.conf
> carp_if = "vmx0"
> sync_if = "vmx1"
> # already allow pfsync and carp protocols
> pass quick on $sync_if proto pfsync keep state (no-sync)
> pass on $carp_if proto carp keep state (no-sync)
> # allow relayd to communicate with pf and set rules
> anchor "relayd/*"
>
> And this is the relayd.conf
>
> log updates
> prefork 5
> fx_vip = "VIP"
> table {
> "host1"
> "host2"
> }
> redirect FX-SFTP {
> listen on $fx_vip port 22 interface vmx0
> route to check tcp interface vmx0
> sticky-address
> }
>
> This is the "ruleset" (identical on both) after reloading pf
>
> # pfctl -a '*' -s rules
> pass quick on vmx1 proto pfsync all keep state (no-sync)
> pass on vmx0 proto carp all keep state (no-sync)
> anchor "relayd/*" all {
> anchor "FX-SFTP" all {
> pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port =
22
> flags any keep state (sloppy, tcp.established 600)
> route-to @vmx0 round-robin sticky-address
> }
> }
>
> When the first connection is made, I see the state on the
> backup carp machine. But with slightly different content.
>
> This is on "master"
>
> all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
>[0 + 1] [946261580 + 2]
>age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2, rule
2, sloppy
>id: 57fbd552a2b4 creatorid: d4cdd00a
>
> "expires" is 10 minutes (tcp.established 600) and I see the anchor and rule
> which generated state
>
> This in on "backup"
>
> all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
>[0 + 1] [946261580 + 2]
>age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy
>id: 57fbd552a2b4 creatorid: d4cdd00a
>
> expires is 1 day (?) and "backup" did not yet see any packes.
>
> Now, how can I get this to work, so the sessions won't be terminated
> in case of a failover.
>
> Every help will be appreciated.
>
> Kind regards,
> Robert
>
>
> > Gesendet: Mittwoch, 12. Oktober 2016 um 14:18 Uhr
> > Von: "Robert Paschedag"
> > An: misc@openbsd.org
> > Betreff: Re: pf on carp backup resets connection after failover
> >
> > Sorry for this bad web mailer formatting. I didn't want that.Am 12.10.2016
2:08 nachm. schrieb Robert Paschedag :
> > >
> > > Hi all, basically, if have exactly this problem already described
here(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUj
Re: pf on carp backup resets connection after failover
This time it should be better. Again sorry..
Hi all,
basically, if have exactly this problem already described here
(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjxwfY)
.
But because there is no answer since 2009, I'll give it a try.
The setup of the 2 servers is also the same as in the other thread
only exception is, that my boxes are behind a "master" firewall
which I do not manage.
I have 2 OpenBSD 6.0 servers that should just act as a load balancer
for SFTP connections. We use DSR mode because huge files get
downloaded from the SFTP servers and don't want the "load" to
pass completly through the OpenBSD load balancers.
Everything is working as long as I don't do a failover to the backup system.
In this situation, I see, that the "new" carp master "resets" the connection
of the client. Immediatly opening a new SFTP sessions then works as
expected through the "new" carp master.
This is my /etc/pf.conf (identical on both). Still testing..
# cat /etc/pf.conf
carp_if = "vmx0"
sync_if = "vmx1"
# already allow pfsync and carp protocols
pass quick on $sync_if proto pfsync keep state (no-sync)
pass on $carp_if proto carp keep state (no-sync)
# allow relayd to communicate with pf and set rules
anchor "relayd/*"
And this is the relayd.conf
log updates
prefork 5
fx_vip = "VIP"
table {
"host1"
"host2"
}
redirect FX-SFTP {
listen on $fx_vip port 22 interface vmx0
route to check tcp interface vmx0
sticky-address
}
This is the "ruleset" (identical on both) after reloading pf
# pfctl -a '*' -s rules
pass quick on vmx1 proto pfsync all keep state (no-sync)
pass on vmx0 proto carp all keep state (no-sync)
anchor "relayd/*" all {
anchor "FX-SFTP" all {
pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port =
22
flags any keep state (sloppy, tcp.established 600)
route-to @vmx0 round-robin sticky-address
}
}
When the first connection is made, I see the state on the
backup carp machine. But with slightly different content.
This is on "master"
all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
[0 + 1] [946261580 + 2]
age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2, rule
2, sloppy
id: 57fbd552a2b4 creatorid: d4cdd00a
"expires" is 10 minutes (tcp.established 600) and I see the anchor and rule
which generated state
This in on "backup"
all tcp VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
[0 + 1] [946261580 + 2]
age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy
id: 57fbd552a2b4 creatorid: d4cdd00a
expires is 1 day (?) and "backup" did not yet see any packes.
Now, how can I get this to work, so the sessions won't be terminated
in case of a failover.
Every help will be appreciated.
Kind regards,
Robert
> Gesendet: Mittwoch, 12. Oktober 2016 um 14:18 Uhr
> Von: "Robert Paschedag"
> An: misc@openbsd.org
> Betreff: Re: pf on carp backup resets connection after failover
>
> Sorry for this bad web mailer formatting. I didn't want that.Am 12.10.2016
2:08 nachm. schrieb Robert Paschedag :
> >
> > Hi all, basically, if have exactly this problem already described
here(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjx
wfY).But
> > because there is no answer since 2009, I'll give it a try. The setup of
> > the 2 servers is also the same as in the other threadonly exception is,
> > that my boxes are behind a "master" firewallwhich I do not manage. I have
> > 2 OpenBSD 6.0 servers that should just act as a load balancerfor SFTP
> > connections. We use DSR mode because huge files getdownloaded from the
> > SFTP servers and don't want the "load" topass completly through the
> > OpenBSD load balancers. Everything is working as long as I don't do a
> > failover to the backup system.In this situation, I see, that the "new"
> > carp master "resets" the connectionof the client. Immediatly opening a
> > new SFTP sessions then works asexpected through the "new" carp master.
> > This is my /etc/pf.conf (identical on both). Still testing.. # cat
> > /etc/pf.conf
> > carp_if = "vmx0"
> > sync_if = "vmx1"# already allow pfsync and carp protocols
> > pass quick on $sync_if proto pfsync keep state (no-sync)
> > pass on $carp_if proto carp keep state (no-sync)# allow relayd to
> > communicate with pf and set rules
> > anchor "relayd/*" And this is the relayd.conf log updates
> > prefork 5fx_vip = "VIP"table {
> > "host1"
> > "host2"
> > }redir
Re: pf on carp backup resets connection after failover
Sorry for this bad web mailer formatting. I didn't want that.Am 12.10.2016 2:08
nachm. schrieb Robert Paschedag :
>
> Hi all, basically, if have exactly this problem already described
> here(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjxwfY).But
>
> because there is no answer since 2009, I'll give it a try. The setup of
> the 2 servers is also the same as in the other threadonly exception is,
> that my boxes are behind a "master" firewallwhich I do not manage. I have
> 2 OpenBSD 6.0 servers that should just act as a load balancerfor SFTP
> connections. We use DSR mode because huge files getdownloaded from the
> SFTP servers and don't want the "load" topass completly through the
> OpenBSD load balancers. Everything is working as long as I don't do a
> failover to the backup system.In this situation, I see, that the "new"
> carp master "resets" the connectionof the client. Immediatly opening a
> new SFTP sessions then works asexpected through the "new" carp master.
> This is my /etc/pf.conf (identical on both). Still testing.. # cat
> /etc/pf.conf
> carp_if = "vmx0"
> sync_if = "vmx1"# already allow pfsync and carp protocols
> pass quick on $sync_if proto pfsync keep state (no-sync)
> pass on $carp_if proto carp keep state (no-sync)# allow relayd to
> communicate with pf and set rules
> anchor "relayd/*" And this is the relayd.conf log updates
> prefork 5fx_vip = "VIP"table {
> "host1"
> "host2"
> }redirect FX-SFTP {
> listen on $fx_vip port 22 interface vmx0
> route to check tcp interface vmx0
> sticky-address
> }
> This is the "ruleset" (identical on both) after reloading pf # pfctl -a
> '*' -s rules
> pass quick on vmx1 proto pfsync all keep state (no-sync)
> pass on vmx0 proto carp all keep state (no-sync)
> anchor "relayd/*" all {
> anchor "FX-SFTP" all {
> pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port =
> 22 flags any keep state (sloppy, tcp.established 600) route-to
> @vmx0 round-robin sticky-address
> }
> } When the first connection is made, I see the state on thebackup carp
> machine. But with slightly different content. This is on "master" all tcp
> VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
> [0 + 1] [946261580 + 2]
> age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2,
> rule 2, sloppy id: 57fbd552a2b4 creatorid: d4cdd00a "expires" is 10
> minutes (tcp.established 600) and I see the anchor and rulewhich
> generated state This in on "backup" all tcp VIP:22 <- CLIENT:43334
> ESTABLISHED:ESTABLISHED
> [0 + 1] [946261580 + 2]
> age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy
> id: 57fbd552a2b4 creatorid: d4cdd00a expires is 1 day (?) and
> "backup" did not yet see any packes. Now, how can I get this to work, so
> the sessions won't be terminatedin case of a failover. Every help will be
> appreciated. Kind regards,Robert
pf on carp backup resets connection after failover
Hi all, basically, if have exactly this problem already described
here(https://groups.google.com/forum/#!topic/bit.listserv.openbsd-pf/yZn4EUjxwfY).But
because there is no answer since 2009, I'll give it a try. The setup of
the 2 servers is also the same as in the other threadonly exception is,
that my boxes are behind a "master" firewallwhich I do not manage. I have
2 OpenBSD 6.0 servers that should just act as a load balancerfor SFTP
connections. We use DSR mode because huge files getdownloaded from the
SFTP servers and don't want the "load" topass completly through the
OpenBSD load balancers. Everything is working as long as I don't do a
failover to the backup system.In this situation, I see, that the "new"
carp master "resets" the connectionof the client. Immediatly opening a
new SFTP sessions then works asexpected through the "new" carp master.
This is my /etc/pf.conf (identical on both). Still testing.. # cat
/etc/pf.conf
carp_if = "vmx0"
sync_if = "vmx1"# already allow pfsync and carp protocols
pass quick on $sync_if proto pfsync keep state (no-sync)
pass on $carp_if proto carp keep state (no-sync)# allow relayd to
communicate with pf and set rules
anchor "relayd/*" And this is the relayd.conf log updates
prefork 5fx_vip = "VIP"table {
"host1"
"host2"
}redirect FX-SFTP {
listen on $fx_vip port 22 interface vmx0
route to check tcp interface vmx0
sticky-address
}
This is the "ruleset" (identical on both) after reloading pf # pfctl -a
'*' -s rules
pass quick on vmx1 proto pfsync all keep state (no-sync)
pass on vmx0 proto carp all keep state (no-sync)
anchor "relayd/*" all {
anchor "FX-SFTP" all {
pass in quick on vmx0 on rdomain 0 inet proto tcp from any to VIP port =
22 flags any keep state (sloppy, tcp.established 600) route-to
@vmx0 round-robin sticky-address
}
} When the first connection is made, I see the state on thebackup carp
machine. But with slightly different content. This is on "master" all tcp
VIP:22 <- CLIENT:43334 ESTABLISHED:ESTABLISHED
[0 + 1] [946261580 + 2]
age 00:00:35, expires in 00:09:37, 16:0 pkts, 913:0 bytes, anchor 2,
rule 2, sloppy id: 57fbd552a2b4 creatorid: d4cdd00a "expires" is 10
minutes (tcp.established 600) and I see the anchor and rulewhich
generated state This in on "backup" all tcp VIP:22 <- CLIENT:43334
ESTABLISHED:ESTABLISHED
[0 + 1] [946261580 + 2]
age 00:00:32, expires in 23:59:41, 0:0 pkts, 0:0 bytes, sloppy
id: 57fbd552a2b4 creatorid: d4cdd00a expires is 1 day (?) and
"backup" did not yet see any packes. Now, how can I get this to work, so
the sessions won't be terminatedin case of a failover. Every help will be
appreciated. Kind regards,Robert
