Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
Nope, I tried here misc@/bugs@, Reddit and with Protectli but unfortunately no luck. I might have to try to install -current or wait for the next release and see. On 2020-08-16 19:30, Steve Woodward wrote: Were you able to resolve this issue? On Jun 10, 2020, at 15:59, obs...@loopw.com wrote: I have a small fleet of protectli firewalls, all of them with em nics. Only the units i’ve upgraded to 6.7 are showing interface errors, where 6.6 is definitely not. On Jun 8, 2020, at 5:30 PM, Gabri Tofano wrote: Hi all, I'm sending this e-mail since I have found other users in this mailing-list using the same device without issues. I'm using a "Protectli FW1" with FreeBSD 12.1 amd64 as a firewall which is serving me with great performances and no issues at all. The appliance has 4 Intel Gigabit 82583V Ethernet NIC ports which are working very well under FreeBSD 12.1. I have used PFsense as well prior to FreeBSD and it worked without issues too. I took the decision to move to OpenBSD 6.7 amd64 in order to benefit of the latest pf (and other) features but unfortunately the OS is giving me an issue which I guess is related to the NIC drivers; When I was connected via ssh I felt some glitches meanwhile I was typing/moving around with the editor, so I started to ping the inside interface from my wired connected pc and found out that time to time the appliance is responding with a 100+/200+ ms response (I have cut some 1ms reply to make it shorter): Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=163ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=2ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=3ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=43ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=4ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=257ms TTL=254 With FreeBSD 12.1 is steady at <1/1ms all the time and even under load. As an online gamer as well, I felt the glitches meanwhile playing few online FPS games using OpenBSD 6.7 on the appliance. Looking at the interface statistics on OpenBSD I found out that inbound/outbound errors are present (this has been taken after few minutes of a reinstall to test it again): FRW-FW1# netstat -i NameMtu Network Address Ipkts IfailOpkts Ofail Colls em0 1500xx:xx:xx:xx:xx:xx1317600 2351 466114 0 0 em0 1500 74.215.235/ xxx.xxx.xxx.xxx 1317600 2351 466114 0 0 em1 1500xx:xx:xx:xx:xx:xx39278218 1199871 1 0 em1 1500 172.16.200. 172.16.200.1 39278218 1199871 1 0 em2 1500xx:xx:xx:xx:xx:xx156055 1 0 em2 1500 172.16.103/ 172.16.103.254 156055 1 0 em3*1500xx:xx:xx:xx:xx:xx 0 0 0 0 0 enc0* 0 0 0 0 0 0 pflog0 33136 0 0 0 0 0 Looking at the Cisco 3560G where the ports are connected there are no errors at all. I have also doublechecked the drivers and the firmware installed by fw_update are the following: vmm-firmware-1.11.0p2 inteldrm-firmware-20181218 intel-firmware-20200508v0 I have done multiple reinstall with different OS to make sure that this is related to OpenBSD 6.7 itself and found the following: PFsense 2.4.5: no issues at all FreeBSD 12.1: no issues at all OPNsense: interface errors OpenBSD: interface errors and interface latency spikes I have also swapped the ethernet cables and contacted Protectli which has confirme
Issue with relayd and redirections
Hi All,
I am trying to move to relayd (OpenBSD 6.7) from HAproxy by keeping my
config to serve multiple domains in SSL passthrough but I'm having some
difficulties. If I correctly understand, according to the man page it
looks like that redirections are used for passthrough traffic and relays
for SSL acceleration/Layer 7 proxy.
Here my config with redirections:
ext_if = "172.16.101.35"
lab1_web1 = "172.16.101.31"
lab1_web2 = "172.16.101.32"
interval 3
log state changes
log connection
table {
$lab1_web1 retry 2
}
table {
$lab1_web2 retry 2
}
http protocol "http" {
return error
tcp { backlog 100, nodelay, sack, socket buffer 65536 }
match header log "Host"
match header log "X-Forwarded-For"
match header log "User-Agent"
match header log "Referer"
match url log
match request header set "X-Forwarded-For" \
value "$REMOTE_ADDR"
match request header set "X-Forwarded-By" \
value "$SERVER_ADDR:$SERVER_PORT"
match request header "Host" value "test1.domain.com" \
forward to
match request header "Host" value "test2.domain.com" \
forward to
}
http protocol "https" {
return error
tcp { backlog 100, nodelay, sack, socket buffer 65536 }
match header log "Host"
match header log "X-Forwarded-For"
match header log "User-Agent"
match header log "Referer"
match url log
match request header set "X-Forwarded-For" \
value "$REMOTE_ADDR"
match request header set "X-Forwarded-By" \
value "$SERVER_ADDR:$SERVER_PORT"
pass request header "Host" value "test1.domain.com" \
forward to
pass request header "Host" value "test2.domain.com" \
forward to
tls keypair "test1.domain.com"
tls keypair "test2.domain.com"
}
redirect "http" {
listen on $ext_if port 80
forward to check http "/" code 200
forward to check http "/" code 200
sticky-address
}
redirect "https" {
listen on $ext_if port 443
forward to check http "/" code 200
forward to check http "/" code 200
sticky-address
}
Here when I use the relays instead of redirections in the config:
relay "http" {
listen on $ext_if port 80
protocol "http"
forward to check http "/" code 200
forward to check http "/" code 200
}
relay "https" {
listen on $ext_if port 443
protocol "https"
forward to check https "/" code 200
forward to check https "/" code 200
}
With relays I see relayd listening on port 80 and 443 and I'm able
to reach each individual backend server by pointing to the related
configured domain (just in http as I have not defined any local
certificates for https).
When using redirections, no listening ports are open (I guess due to
relayd using pf nat rules) and I'm unable to reach both backend
servers.
I have added the relayd anchor to pf.conf as following:
anchor "relayd/*"
set skip on lo
block return
pass
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
And here how pf lists what's in the anchor:
#pfctl -a relayd/* -s rules
anchor "http" all {
pass in quick on rdomain 0 inet proto tcp from any to 172.16.101.35 \
port = 80 flags S/SA keep state (tcp.established 600) rdr-to \
port 80 round-robin sticky-address
}
anchor "https" all {
}
I'm sure I'm doing something wrong here but I can't find where.
My goal to use SSL passthrough is to leverage the use of SNI and not
generate additional certificates on the load balancer, but using the
already implemented ones on the backend servers.
Thank you!
Re: Issue with relayd and redirections
After some further troubleshooting, tonight I took some time to sit down and read again the man pages as everything on my config files was looking fine and no errors were showing up in any log. With Brian's help we were leading to the direction that something was wrong with the pf translation itself and so I tested a static rdr-to configuration with pf only in the same environment, and neither this test worked as expected. So I went back to read the pf.conf man page and here comes the rdr-to relevant section: "Redirections cannot reflect packets back through the interface they arrive on, they can only be redirected to hosts connected to different interfaces or to the firewall itself." Focusing on relayd, my oversight was to not going back and read again the pf.conf man page in order to make sure that my box's network configuration was ok, since apparently I got it to work with relays without problems. The next challenge now is to find if there is another way to make this setup working with just 1 network interface and implement relayd redirects for SSL passthrough, or give up. There seems to be few options here that I can think of: - Keep my current configuration with HAproxy - Add another network interface to the box and configure an additional network to it (it might be tricky when deploying a droplet with a direct public IP address) - Migrate to relayd relays and give up with SSL passthrough (with the benefit of SSL offloading if want to implement it) Thank you to the community and the devs for the great work on this OS! Especially on the man pages :) On 2020-07-11 12:58, Gabri Tofano wrote: It isn’t. rdr-to, and by extension redirects, are not natting the source address. Clients connecting through relayd and to the backend will have source addresses not that of the relayd machine but of the original client. Thank you for correcting me on this as it was a bad statement told before getting coffee in the morning :) I’m going to play around on my boxes and try and come up with some options for you. I’ll get back to you later. Thank you for dedicating time in looking to this issue! On 2020-07-11 12:08, Brian Brombacher wrote: On Jul 11, 2020, at 11:20 AM, Gabri Tofano wrote: On 2020-07-11 06:33, Brian Brombacher wrote: On Jul 10, 2020, at 11:42 PM, Gabri Tofano wrote: Does http work with redirects? It wasn’t clear if it did or not in your first post. It doesn't work with http and that is the redirect that I was testing. Indications from your pf anchor rules and the down status above, and the check http attribute on the https forward to directives tell me relayd isn’t liking your check http configuration for port 443. Start by switching to check icmp or check tcp or something else, see if it works, unless you can fix the check http based on logs or otherwise. I changed it to tcp and now the servers are showing as "up": LAB1-LB1# relayctl sh sum Id TypeNameAvlblty Status 1 redirecthttp active 1 table web_servers:80 active (1 hosts) 1 host172.16.101.31 100.00% up 2 table nc_servers:80 active (1 hosts) 2 host172.16.101.32 100.00% up 2 redirecthttps active 3 table web_servers:443 active (1 hosts) 3 host172.16.101.31 100.00% up 4 table nc_servers:443 active (1 hosts) 4 host172.16.101.32 100.00% up However I was hoping to fix the http redirect first and then move to https, but it looks like more of a "general issue" with redirects in my current configuration. Thanks If http redirection isn’t working, I’d be curious from where you’re trying to connect or what router you have configured on the backend hosts. I see you’re relayd box and back ends are on the same network. If you’re trying to connect from another address in 172.16.101.x to your relayd setup, it won’t work reliably. It might also not work reliably or at all, if you are not routing responses through the relayd host. If they are replying direct, any PF scrub normalization, tcp sequence handling, etc., all get lost, among other issues. I hope this is the cause of your issues, otherwise you’re going to need to include more information for your setup, or at a minimum some relayd logs. -Brian I have a layer3 switch doing routing between 2 vlans, relayd and the 2 backend web servers are on the same vlan and the client is on another vlan 172.16.100.x. The relayd VM is configured with only 1 network interface. When the client try to reach the web servers directly everything work fine. When the client i
Re: Issue with relayd and redirections
I have tried to implement the workaround as per man page
but it still doesn't work, here the pf.conf config:
eth0 = "xnf0"
web1 = "172.16.101.31"
anchor "relayd/*"
set skip on lo
block return log
pass log
pass out quick on $eth0 proto tcp to $web1 port 80 \
received-on $eth0 nat-to $eth0
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
I'm trying to gather some useful log on relayd and see if
there's any error but even with "relayctl log verbose"
nothing is showing beside the startup entries
Thank you!
There's a "workaround" also mentioned in pf.conf(5) which also works
with relayd inserted rdr-rules, e.g.
pass out quick on vlan99 proto tcp to 192.168.89.13 received-on vlan99
nat-to 192.168.89.1
vlan99 has 'inet 192.168.89.1/24' and 192.168.89.13 is the relayd rdr
"target".
HTH,
--
pb
On 2020-07-13 01:08, Gabri Tofano wrote:
After some further troubleshooting, tonight I took some time to sit
down and
read again the man pages as everything on my config files was looking
fine and
no errors were showing up in any log. With Brian's help we were leading
to the
direction that something was wrong with the pf translation itself and
so I
tested a static rdr-to configuration with pf only in the same
environment, and
neither this test worked as expected. So I went back to read the
pf.conf man
page and here comes the rdr-to relevant section:
"Redirections cannot reflect packets back through the interface they
arrive on, they can only be redirected to hosts connected to different
interfaces or to the firewall itself."
Focusing on relayd, my oversight was to not going back and read again
the
pf.conf man page in order to make sure that my box's network
configuration was
ok, since apparently I got it to work with relays without problems.
The next challenge now is to find if there is another way to make this
setup
working with just 1 network interface and implement relayd redirects
for SSL
passthrough, or give up. There seems to be few options here that I can
think of:
- Keep my current configuration with HAproxy
- Add another network interface to the box and configure an additional
network to
it (it might be tricky when deploying a droplet with a direct public IP
address)
- Migrate to relayd relays and give up with SSL passthrough (with the
benefit of
SSL offloading if want to implement it)
Thank you to the community and the devs for the great work on this OS!
Especially
on the man pages :)
On 2020-07-11 12:58, Gabri Tofano wrote:
It isn’t. rdr-to, and by extension redirects, are not natting the
source address.
Clients connecting through relayd and to the backend will have source
addresses
not that of the relayd machine but of the original client.
Thank you for correcting me on this as it was a bad statement told
before
getting coffee in the morning :)
I’m going to play around on my boxes and try and come up with some
options for you.
I’ll get back to you later.
Thank you for dedicating time in looking to this issue!
On 2020-07-11 12:08, Brian Brombacher wrote:
On Jul 11, 2020, at 11:20 AM, Gabri Tofano
wrote:
On 2020-07-11 06:33, Brian Brombacher wrote:
On Jul 10, 2020, at 11:42 PM, Gabri Tofano
wrote:
Does http work with redirects? It wasn’t clear if it did or
not in
your first post.
It doesn't work with http and that is the redirect that I was
testing.
Indications from your pf anchor rules and the down
status above, and the check http attribute on the https forward
to
directives tell me relayd isn’t liking your check http
configuration
for port 443.
Start by switching to check icmp or check tcp or something
else, see
if it works, unless you can fix the check http based on logs or
otherwise.
I changed it to tcp and now the servers are showing as "up":
LAB1-LB1# relayctl sh sum
Id TypeNameAvlblty
Status
1 redirecthttp
active
1 table web_servers:80
active (1 hosts)
1 host172.16.101.31 100.00%
up
2 table nc_servers:80
active (1 hosts)
2 host172.16.101.32 100.00%
up
2 redirecthttps
active
3 table web_servers:443
active (1 hosts)
3 host172.16.101.31 100.00%
up
4 table nc_servers:443
active (1 hosts)
4 host172.16.101.32 100.00%
up
However I was hoping to fix the http redirect first and then
move to https, but it
looks like more of a "general issue" with redirects in my
current configuration.
Thanks
If http redirection isn’t working, I’d be cur
Re: Issue with relayd and redirections
I did but still negative. No sessions shown in relayctl so still
thinking it's an issue in pf.
On 2020-07-13 22:51, Brian Brombacher wrote:
On Jul 13, 2020, at 8:30 PM, Gabri Tofano wrote:
I have tried to implement the workaround as per man page
but it still doesn't work, here the pf.conf config:
eth0 = "xnf0"
web1 = "172.16.101.31"
anchor "relayd/*"
set skip on lo
block return log
pass log
pass out quick on $eth0 proto tcp to $web1 port 80 \
received-on $eth0 nat-to $eth0
Try putting this before the anchor. The quick entry in the anchor
that relayd creates takes precedence.
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
I'm trying to gather some useful log on relayd and see if
there's any error but even with "relayctl log verbose"
nothing is showing beside the startup entries
Thank you!
There's a "workaround" also mentioned in pf.conf(5) which also works
with relayd inserted rdr-rules, e.g.
pass out quick on vlan99 proto tcp to 192.168.89.13 received-on
vlan99
nat-to 192.168.89.1
vlan99 has 'inet 192.168.89.1/24' and 192.168.89.13 is the relayd rdr
"target".
HTH,
--
pb
On 2020-07-13 01:08, Gabri Tofano wrote:
After some further troubleshooting, tonight I took some time to sit
down and
read again the man pages as everything on my config files was looking
fine and
no errors were showing up in any log. With Brian's help we were
leading to the
direction that something was wrong with the pf translation itself and
so I
tested a static rdr-to configuration with pf only in the same
environment, and
neither this test worked as expected. So I went back to read the
pf.conf man
page and here comes the rdr-to relevant section:
"Redirections cannot reflect packets back through the interface they
arrive on, they can only be redirected to hosts connected to
different
interfaces or to the firewall itself."
Focusing on relayd, my oversight was to not going back and read again
the
pf.conf man page in order to make sure that my box's network
configuration was
ok, since apparently I got it to work with relays without problems.
The next challenge now is to find if there is another way to make
this setup
working with just 1 network interface and implement relayd redirects
for SSL
passthrough, or give up. There seems to be few options here that I
can think of:
- Keep my current configuration with HAproxy
- Add another network interface to the box and configure an
additional
network to
it (it might be tricky when deploying a droplet with a direct public
IP address)
- Migrate to relayd relays and give up with SSL passthrough (with the
benefit of
SSL offloading if want to implement it)
Thank you to the community and the devs for the great work on this
OS!
Especially
on the man pages :)
On 2020-07-11 12:58, Gabri Tofano wrote:
It isn’t. rdr-to, and by extension redirects, are not natting the
source address.
Clients connecting through relayd and to the backend will have
source addresses
not that of the relayd machine but of the original client.
Thank you for correcting me on this as it was a bad statement told
before
getting coffee in the morning :)
I’m going to play around on my boxes and try and come up with some
options for you.
I’ll get back to you later.
Thank you for dedicating time in looking to this issue!
On 2020-07-11 12:08, Brian Brombacher wrote:
On Jul 11, 2020, at 11:20 AM, Gabri Tofano
wrote:
On 2020-07-11 06:33, Brian Brombacher wrote:
On Jul 10, 2020, at 11:42 PM, Gabri Tofano
wrote:
Does http work with redirects? It wasn’t clear if it did or
not in
your first post.
It doesn't work with http and that is the redirect that I was
testing.
Indications from your pf anchor rules and the down
status above, and the check http attribute on the https
forward to
directives tell me relayd isn’t liking your check http
configuration
for port 443.
Start by switching to check icmp or check tcp or something
else, see
if it works, unless you can fix the check http based on logs
or
otherwise.
I changed it to tcp and now the servers are showing as "up":
LAB1-LB1# relayctl sh sum
Id TypeName
Avlblty Status
1 redirecthttp
active
1 table web_servers:80
active (1 hosts)
1 host172.16.101.31
100.00% up
2 table nc_servers:80
active (1 hosts)
2 host172.16.101.32
100.00% up
2 redirecthttps
active
3 table web_servers:443
active (1 hosts)
3 host172.16.101.31
100.00% up
4 table nc_servers:443
active (1 host
Re: Issue with relayd and redirections
Here: LAB1-LB1$ relayctl sh sum Id TypeName Avlblty Status 1 redirecthttp active 1 table web_servers:80 active (1 hosts) 1 host172.16.101.31 4.87% up 2 table nc_servers:80 active (1 hosts) 2 host172.16.101.32 4.86% up 2 redirecthttps down 3 table web_servers:443empty 3 host172.16.101.31 0.00% down 4 table nc_servers:443 empty 4 host172.16.101.32 0.00% down The low availability is due too the web servers were turned off. Thanks! On 2020-07-10 17:41, Sebastian Benoit wrote: Gabri Tofano(ga...@tofanos.com) on 2020.07.07 15:38:17 -0400: When using redirections, no listening ports are open (I guess due to relayd using pf nat rules) correct and I'm unable to reach both backend servers. show the output of "relayctl sh sum".
Re: Issue with relayd and redirections
Does http work with redirects? It wasn’t clear if it did or not in your first post. It doesn't work with http and that is the redirect that I was testing. Indications from your pf anchor rules and the down status above, and the check http attribute on the https forward to directives tell me relayd isn’t liking your check http configuration for port 443. Start by switching to check icmp or check tcp or something else, see if it works, unless you can fix the check http based on logs or otherwise. I changed it to tcp and now the servers are showing as "up": LAB1-LB1# relayctl sh sum Id TypeNameAvlblty Status 1 redirecthttpactive 1 table web_servers:80 active (1 hosts) 1 host172.16.101.31 100.00% up 2 table nc_servers:80 active (1 hosts) 2 host172.16.101.32 100.00% up 2 redirecthttps active 3 table web_servers:443 active (1 hosts) 3 host172.16.101.31 100.00% up 4 table nc_servers:443 active (1 hosts) 4 host172.16.101.32 100.00% up However I was hoping to fix the http redirect first and then move to https, but it looks like more of a "general issue" with redirects in my current configuration. Thanks
Re: Issue with relayd and redirections
On 2020-07-11 06:33, Brian Brombacher wrote: On Jul 10, 2020, at 11:42 PM, Gabri Tofano wrote: Does http work with redirects? It wasn’t clear if it did or not in your first post. It doesn't work with http and that is the redirect that I was testing. Indications from your pf anchor rules and the down status above, and the check http attribute on the https forward to directives tell me relayd isn’t liking your check http configuration for port 443. Start by switching to check icmp or check tcp or something else, see if it works, unless you can fix the check http based on logs or otherwise. I changed it to tcp and now the servers are showing as "up": LAB1-LB1# relayctl sh sum Id TypeNameAvlblty Status 1 redirecthttpactive 1 table web_servers:80 active (1 hosts) 1 host172.16.101.31 100.00% up 2 table nc_servers:80 active (1 hosts) 2 host172.16.101.32 100.00% up 2 redirecthttps active 3 table web_servers:443 active (1 hosts) 3 host172.16.101.31 100.00% up 4 table nc_servers:443 active (1 hosts) 4 host172.16.101.32 100.00% up However I was hoping to fix the http redirect first and then move to https, but it looks like more of a "general issue" with redirects in my current configuration. Thanks If http redirection isn’t working, I’d be curious from where you’re trying to connect or what router you have configured on the backend hosts. I see you’re relayd box and back ends are on the same network. If you’re trying to connect from another address in 172.16.101.x to your relayd setup, it won’t work reliably. It might also not work reliably or at all, if you are not routing responses through the relayd host. If they are replying direct, any PF scrub normalization, tcp sequence handling, etc., all get lost, among other issues. I hope this is the cause of your issues, otherwise you’re going to need to include more information for your setup, or at a minimum some relayd logs. -Brian I have a layer3 switch doing routing between 2 vlans, relayd and the 2 backend web servers are on the same vlan and the client is on another vlan 172.16.100.x. The relayd VM is configured with only 1 network interface. When the client try to reach the web servers directly everything work fine. When the client is passing through relayd I see the following: - Only SYN packets coming into relayd box which they become retransmissions - The relayd anchor rules do not have the log parameter set so I cannot see passing traffic from the client to the backend servers, but at least no traffic is being blocked. I haven't found a way to manipulate an anchor via pfctl in order to add the log parameter - The web server does not see any traffic reaching out on port 80 beside the http checks from relayd IP address - I have set "log connection" in relayd.conf and then relayctl log verbose but /var/log/daemon unfortunately is not showing much: relayd[84883]: startup relayd[84883]: unused protocol: http relayd[84883]: unused protocol: https relayd[33541]: host 172.16.101.32, check tcp (1ms,tcp connect ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.31, check tcp (2ms,tcp connect ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.31, check http code (3ms,http code ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.32, check http code (3ms,http code ok), state unknown -> up, availability 100.00% relayd[17311]: table http: 1 added, 0 deleted, 0 changed, 0 killed relayd[17311]: table https: 1 added, 0 deleted, 0 changed, 0 killed Since with relayd redirects traffic is being natted, I'm not sure if the issue could be with the fact that relayd is natting on the same interface for the same network subnet, as with relayd relays everything works fine. I'm currently stuck in trying to find a way to log verbose what is happening on relayd as at least the client sessions are seen on relayd itself: LAB1-LB1# relayctl sh redirects Id TypeNameAvlblty Status 1 redirecthttpactive total: 18 sessions last: 0/60s 2/h 18/d sessions average: 0/60s 0/h 0/d sessions 2 redirecthttps active Thank you,
Re: Issue with relayd and redirections
It isn’t. rdr-to, and by extension redirects, are not natting the source address. Clients connecting through relayd and to the backend will have source addresses not that of the relayd machine but of the original client. Thank you for correcting me on this as it was a bad statement told before getting coffee in the morning :) I’m going to play around on my boxes and try and come up with some options for you. I’ll get back to you later. Thank you for dedicating time in looking to this issue! On 2020-07-11 12:08, Brian Brombacher wrote: On Jul 11, 2020, at 11:20 AM, Gabri Tofano wrote: On 2020-07-11 06:33, Brian Brombacher wrote: On Jul 10, 2020, at 11:42 PM, Gabri Tofano wrote: Does http work with redirects? It wasn’t clear if it did or not in your first post. It doesn't work with http and that is the redirect that I was testing. Indications from your pf anchor rules and the down status above, and the check http attribute on the https forward to directives tell me relayd isn’t liking your check http configuration for port 443. Start by switching to check icmp or check tcp or something else, see if it works, unless you can fix the check http based on logs or otherwise. I changed it to tcp and now the servers are showing as "up": LAB1-LB1# relayctl sh sum Id TypeNameAvlblty Status 1 redirecthttp active 1 table web_servers:80 active (1 hosts) 1 host172.16.101.31 100.00% up 2 table nc_servers:80 active (1 hosts) 2 host172.16.101.32 100.00% up 2 redirecthttps active 3 table web_servers:443 active (1 hosts) 3 host172.16.101.31 100.00% up 4 table nc_servers:443 active (1 hosts) 4 host172.16.101.32 100.00% up However I was hoping to fix the http redirect first and then move to https, but it looks like more of a "general issue" with redirects in my current configuration. Thanks If http redirection isn’t working, I’d be curious from where you’re trying to connect or what router you have configured on the backend hosts. I see you’re relayd box and back ends are on the same network. If you’re trying to connect from another address in 172.16.101.x to your relayd setup, it won’t work reliably. It might also not work reliably or at all, if you are not routing responses through the relayd host. If they are replying direct, any PF scrub normalization, tcp sequence handling, etc., all get lost, among other issues. I hope this is the cause of your issues, otherwise you’re going to need to include more information for your setup, or at a minimum some relayd logs. -Brian I have a layer3 switch doing routing between 2 vlans, relayd and the 2 backend web servers are on the same vlan and the client is on another vlan 172.16.100.x. The relayd VM is configured with only 1 network interface. When the client try to reach the web servers directly everything work fine. When the client is passing through relayd I see the following: - Only SYN packets coming into relayd box which they become retransmissions - The relayd anchor rules do not have the log parameter set so I cannot see passing traffic from the client to the backend servers, but at least no traffic is being blocked. I haven't found a way to manipulate an anchor via pfctl in order to add the log parameter - The web server does not see any traffic reaching out on port 80 beside the http checks from relayd IP address - I have set "log connection" in relayd.conf and then relayctl log verbose but /var/log/daemon unfortunately is not showing much: relayd[84883]: startup relayd[84883]: unused protocol: http relayd[84883]: unused protocol: https relayd[33541]: host 172.16.101.32, check tcp (1ms,tcp connect ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.31, check tcp (2ms,tcp connect ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.31, check http code (3ms,http code ok), state unknown -> up, availability 100.00% relayd[33541]: host 172.16.101.32, check http code (3ms,http code ok), state unknown -> up, availability 100.00% relayd[17311]: table http: 1 added, 0 deleted, 0 changed, 0 killed relayd[17311]: table https: 1 added, 0 deleted, 0 changed, 0 killed Since with relayd redirects traffic is being natted, It isn’t. rdr-to, and by extension redirects, are not natting the source address. Clients connecting through relayd and to the backend will have source addresses not that of the relayd machine but of the original client. I'm not sure if the issue could be with t
Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
That's funny because when I received your e-mail I was almost done in installing OpenBSD 6.6. Another user pointed out to me that in the OpenBSD 6.7 release notes there is a statement in regards of the em(4) drivers: "Improvements in the em(4) driver." and so I have gave it a try. It looks like that the system is now stable and latency spikes/interface errors are not present at all even under heavy traffic loads. I am going to update the message in the OpenBSD-bug mailing list and maybe one of the dev can take a look at it now that we have more info. Thank you for sharing your experience! On 2020-06-10 21:59, obs...@loopw.com wrote: I have a small fleet of protectli firewalls, all of them with em nics. Only the units i’ve upgraded to 6.7 are showing interface errors, where 6.6 is definitely not. On Jun 8, 2020, at 5:30 PM, Gabri Tofano wrote: Hi all, I'm sending this e-mail since I have found other users in this mailing-list using the same device without issues. I'm using a "Protectli FW1" with FreeBSD 12.1 amd64 as a firewall which is serving me with great performances and no issues at all. The appliance has 4 Intel Gigabit 82583V Ethernet NIC ports which are working very well under FreeBSD 12.1. I have used PFsense as well prior to FreeBSD and it worked without issues too. I took the decision to move to OpenBSD 6.7 amd64 in order to benefit of the latest pf (and other) features but unfortunately the OS is giving me an issue which I guess is related to the NIC drivers; When I was connected via ssh I felt some glitches meanwhile I was typing/moving around with the editor, so I started to ping the inside interface from my wired connected pc and found out that time to time the appliance is responding with a 100+/200+ ms response (I have cut some 1ms reply to make it shorter): Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=163ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=2ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=3ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=43ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=4ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=257ms TTL=254 With FreeBSD 12.1 is steady at <1/1ms all the time and even under load. As an online gamer as well, I felt the glitches meanwhile playing few online FPS games using OpenBSD 6.7 on the appliance. Looking at the interface statistics on OpenBSD I found out that inbound/outbound errors are present (this has been taken after few minutes of a reinstall to test it again): FRW-FW1# netstat -i NameMtu Network Address Ipkts IfailOpkts Ofail Colls em0 1500xx:xx:xx:xx:xx:xx1317600 2351 466114 0 0 em0 1500 74.215.235/ xxx.xxx.xxx.xxx 1317600 2351 466114 0 0 em1 1500xx:xx:xx:xx:xx:xx39278218 1199871 1 0 em1 1500 172.16.200. 172.16.200.1 39278218 1199871 1 0 em2 1500xx:xx:xx:xx:xx:xx156055 1 0 em2 1500 172.16.103/ 172.16.103.254 156055 1 0 em3*1500xx:xx:xx:xx:xx:xx 0 0 0 0 0 enc0* 0 0 0 0 0 0 pflog0 33136 0 0 0 0 0 Looking at the Cisco 3560G where the ports are connected there are no errors at all. I have also doublechecked the drivers and the firmware installed by fw_update are the following: vmm-firmware-1.11.0p2 inteldrm-firmware-20181218 intel-firmware-20200508v0 I have done mu
Re: Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
After extensive testing the latency spikes shown up again: To the inside interface of the firewall: Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=132ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 And to the firewall's next hop (ISP ONT) at the same time: Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=3ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=3ms TTL=62 Reply from 74.215.235.1: bytes=32 time=242ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=2ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=1ms TTL=62 Reply from 74.215.235.1: bytes=32 time=3ms TTL=62 Reply from 74.215.235.1: bytes=32 time=3ms TTL=62 Interface errors are now showing up just as output: #netstat -i NameMtu Network Address Ipkts IfailOpkts Ofail Colls em0 1500XX:XX:XX:XX:XX:XX22655 041589 0 0 em0 1500 XX.XX.XX.XX XX:XX:XX:XX:XX:XX22655 041589 0 0 em1 1500XX:XX:XX:XX:XX:XX39924 020476 1 0 em1 1500 172.16.200. XX:XX:XX:XX:XX:XX39924 020476 1 0 em2 1500XX:XX:XX:XX:XX:XX 427 0 330 2 0 em2 1500 172.16.103/ XX:XX:XX:XX:XX:XX 427 0 330 2 0 em3*1500XX:XX:XX:XX:XX:XX0 00 0 0 enc0* 00 00 0 0 pflog0 331360 0 1294 0 0 UDP real time traffic is the most affected one as very sensitive and I keep \ having spikes meanwhile playing online. Thank you! Gabri On 2020-06-10 22:46, Gabri Tofano wrote: That's funny because when I received your e-mail I was almost done in installing OpenBSD 6.6. Another user pointed out to me that in the OpenBSD 6.7 release notes there is a statement in regards of the em(4) drivers: "Improvements in the em(4) driver." and so I have gave it a try. It looks like that the system is now stable and latency spikes/interface errors are not present at all even under heavy traffic loads. I am going to update the message in the OpenBSD-bug mailing list and maybe one of the dev can take a look at it now that we have more info. Thank you for sharing your experience! On 2020-06-10 21:59, obs...@loopw.com wrote: I have a small fleet of protectli firewalls, all of them with em nics. Only the units i’ve upgraded to 6.7 are showing interface errors, where 6.6 is definitely not. On Jun 8, 2020, at 5:30 PM, Gabri Tofano wrote: Hi all, I'm sending this e-mail since I have found other users in this mailing-list using the same device without issues. I'm using a "Protectli FW1" with FreeBSD 12.1 amd64 as a firewall which is serving me with great performances and no issues at all. The appliance has 4 Intel Gigabit 82583V Ethernet NIC ports which are working very well under FreeBSD 12.1. I have used PFsense as well prior to FreeBSD and it worked without issues too. I took the decision to move to OpenBSD 6.7 amd64 in order to benefit of the latest pf (and other) features but unfortunately the OS is giving me an issue which I guess is related to the NIC drivers; When I was connected via ssh I felt some glitches meanwhile I was typing/moving around with the editor, so I started to ping the inside interface from my wired connected pc and found out that time to time the appliance is responding with a 100+/200+ ms response (I have cut some 1ms reply
Protectli FW1 with Intel 82583V - Interfaces errors and latency spike issue
Hi all, I'm sending this e-mail since I have found other users in this mailing-list using the same device without issues. I'm using a "Protectli FW1" with FreeBSD 12.1 amd64 as a firewall which is serving me with great performances and no issues at all. The appliance has 4 Intel Gigabit 82583V Ethernet NIC ports which are working very well under FreeBSD 12.1. I have used PFsense as well prior to FreeBSD and it worked without issues too. I took the decision to move to OpenBSD 6.7 amd64 in order to benefit of the latest pf (and other) features but unfortunately the OS is giving me an issue which I guess is related to the NIC drivers; When I was connected via ssh I felt some glitches meanwhile I was typing/moving around with the editor, so I started to ping the inside interface from my wired connected pc and found out that time to time the appliance is responding with a 100+/200+ ms response (I have cut some 1ms reply to make it shorter): Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=163ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=2ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=3ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=43ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time<1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=4ms TTL=254 Reply from 172.16.200.1: bytes=32 time=1ms TTL=254 Reply from 172.16.200.1: bytes=32 time=257ms TTL=254 With FreeBSD 12.1 is steady at <1/1ms all the time and even under load. As an online gamer as well, I felt the glitches meanwhile playing few online FPS games using OpenBSD 6.7 on the appliance. Looking at the interface statistics on OpenBSD I found out that inbound/outbound errors are present (this has been taken after few minutes of a reinstall to test it again): FRW-FW1# netstat -i NameMtu Network Address Ipkts IfailOpkts Ofail Colls em0 1500xx:xx:xx:xx:xx:xx1317600 2351 466114 0 0 em0 1500 74.215.235/ xxx.xxx.xxx.xxx 1317600 2351 466114 0 0 em1 1500xx:xx:xx:xx:xx:xx39278218 1199871 1 0 em1 1500 172.16.200. 172.16.200.1 39278218 1199871 1 0 em2 1500xx:xx:xx:xx:xx:xx156055 1 0 em2 1500 172.16.103/ 172.16.103.254 156055 1 0 em3*1500xx:xx:xx:xx:xx:xx 0 0 0 0 0 enc0* 0 0 0 0 0 0 pflog0 33136 0 0 0 0 0 Looking at the Cisco 3560G where the ports are connected there are no errors at all. I have also doublechecked the drivers and the firmware installed by fw_update are the following: vmm-firmware-1.11.0p2 inteldrm-firmware-20181218 intel-firmware-20200508v0 I have done multiple reinstall with different OS to make sure that this is related to OpenBSD 6.7 itself and found the following: PFsense 2.4.5: no issues at all FreeBSD 12.1: no issues at all OPNsense: interface errors OpenBSD: interface errors and interface latency spikes I have also swapped the ethernet cables and contacted Protectli which has confirmed that this appliance has been tested on OpenBSD (it looks like 6.3). Here the dmesg output: OpenBSD 6.7 (GENERIC.MP) #2: Thu Jun 4 09:55:08 MDT 2020 r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4163854336 (3970MB) avail mem = 4025044992 (3838MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xecea0 (51 entries) bios0: vendor American Megatrends Inc. version "5.6.5" date 10/24/2018 bios0: Protectli FW1 acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5
