Hello, Sorry for the delay in replying I dont get the opportunity to reboot that box often and with the COVID-19 Crisis Im trying to minimise maintenance that impacts customers
i also tried to split the number of vlans across an additional ix(4) parent interface but this did not change the behaviour Please find output of diagnostic commands below #netstat -nl bridge101 netstat -nI bridge101 (after reboot (broken) ngabr# cat netstat-nI-bridge101-afterreboot Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls bridge101 1500 <Link> 51546 0 2813599 0 0 ngabr# cat netstat-nIbridge101-postnetstart (after netstart (fixed)) Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls bridge101 1500 <Link> 5119709 0 8565403 0 0 ########################################################################### #netstat -s ngabr# cat netstat-s-afterreboot (broken) ip: 3999 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 malformed fragments dropped 0 fragments dropped after timeout 0 packets reassembled ok 237 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 3092 packets not forwardable 0 redirects sent 10977 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 11122 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 fragment floods 0 packets with ip length > max ip packet size 0 tunneling packets that can't find gif 0 datagrams with bad address in header 3656 input datagrams software-checksummed 2837975 output datagrams software-checksummed 3335 multicast packets which we don't join icmp: 225 calls to icmp_error 0 errors not generated because old message was icmp 0 errors not generated because of rate limitation Output packet histogram: destination unreachable: 225 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length 0 echo requests to broadcast/multicast rejected 0 message responses generated igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent ipencap: 0 total input packets 0 total output packets 0 packets shorter than header shows 0 packets dropped due to policy 0 packets with possibly spoofed local addresses 0 packets were dropped due to full output queue 0 input bytes 0 output bytes 0 protocol family mismatches 0 attempts to use tunnel with unspecified endpoint(s) tcp: 54 packets sent 24 data packets (2022 bytes) 0 data packets (0 bytes) retransmitted 0 fast retransmitted packets 18 ack-only packets (18 delayed) 0 URG only packets 0 window probe packets 0 window update packets 12 control packets 0 packets software-checksummed 51 packets received 21 acks (for 1956 bytes) 6 duplicate acks 0 acks for unsent data 0 acks for old data 30 packets (21161 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some duplicate data (0 bytes duplicated) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 1 discarded for bad checksum 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 52 packets software-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 6 connection requests 0 connection accepts 6 connections established (including accepts) 6 connections closed (including 0 drops) 0 connections drained 0 embryonic connections dropped 27 segments updated rtt (of 30 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 12 correct data packet header predictions 3 PCB cache misses 3 dropped due to no socket 0 ECN connections accepted 0 ECE packets received 0 CWR packets received 0 CE packets received 0 ECT packets sent 0 ECE packets sent 0 CWR packets sent cwr by fastrecovery: 0 cwr by timeout: 0 cwr by ecn: 0 0 bad connection attempts 0 SYN packets dropped due to queue or memory full 0 SYN cache entries added 0 hash collisions 0 completed 0 aborted (no space to build PCB) 0 timed out 0 dropped due to overflow 0 dropped due to bucket overflow 0 dropped due to RST 0 dropped due to ICMP unreachable 0 SYN,ACKs retransmitted 0 duplicate SYNs received for entries already in the cache 0 SYNs dropped (no route or no space) 0 SYN cache seeds with new random 293 hash bucket array size in current SYN cache 0 entries in current SYN cache, limit is 10255 0 longest bucket length in current SYN cache, limit is 105 0 uses of current SYN cache left 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options received 0 SACK options sent 0 SACK options dropped udp: 186 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 6 with no checksum 64 input packets software-checksummed 0 output packets software-checksummed 0 dropped due to no socket 160 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 0 dropped due to full socket buffers 26 delivered 26 datagrams output 0 missed PCB cache ipsec: 0 input IPsec packets 0 output IPsec packets 0 input bytes 0 output bytes 0 input bytes, decompressed 0 output bytes, uncompressed 0 packets dropped on input 0 packets dropped on output 0 packets that failed crypto processing 0 packets for which no XFORM was set in TDB received 0 packets for which no TDB was found esp: 0 input ESP packets 0 output ESP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets with bad encryption received 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad payload size or padding received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 output packets could not be sent 0 input UDP encapsulated ESP packets 0 output UDP encapsulated ESP packets 0 UDP packets for non-encapsulating TDB received 0 raw ESP packets for encapsulating TDB received 0 input bytes 0 output bytes ah: 0 input AH packets 0 output AH packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad authenticator length received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 output packets could not be sent 0 input bytes 0 output bytes etherip: 0 packets shorter than header shows 0 packets were dropped due to full output queue 0 packets were dropped because of no interface/bridge information 0 packets dropped due to policy 0 packets dropped for other reasons 0 input ethernet-in-IP packets 0 output ethernet-in-IP packets 0 input bytes 0 output bytes ipcomp: 0 input IPCOMP packets 0 output IPCOMP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed (de)compression processing 0 output packets could not be sent 0 packets less than minimum compression length 0 input bytes 0 output bytes carp: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 transitions to master pfsync: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 0 stale states 0 failed state lookup/inserts 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error divert: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors pflow: 0 flows sent 0 packets sent 0 send failed due to mbuf memory error 0 send error ip6: 11 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 4 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 11 multicast packets which we don't join Input packet histogram: ICMP6: 11 Mbuf statistics: 0 one mbufs 0 one ext mbufs 0 two or more ext mbufs 0 tunneling packets that can't find gif 0 packets discarded due to too many headers 0 failures of source address selection 0 forward cache hit 0 forward cache miss divert6: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors icmp6: 0 calls to icmp6_error 0 errors not generated because old message was icmp6 or so 0 errors not generated because of rate limitation Output packet histogram: multicast listener report: 4 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output ################################################################# ngabr# cat netstat-s-postnetstart working ip: 8928 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 malformed fragments dropped 0 fragments dropped after timeout 0 packets reassembled ok 490 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 7270 packets not forwardable 0 redirects sent 19894 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 20397 output packets discarded due to no route 22 output datagrams fragmented 22 fragments created 0 datagrams that can't be fragmented 0 fragment floods 0 packets with ip length > max ip packet size 0 tunneling packets that can't find gif 0 datagrams with bad address in header 8029 input datagrams software-checksummed 15579679 output datagrams software-checksummed 7953 multicast packets which we don't join icmp: 659 calls to icmp_error 0 errors not generated because old message was icmp 0 errors not generated because of rate limitation Output packet histogram: destination unreachable: 659 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length 0 echo requests to broadcast/multicast rejected 0 message responses generated igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent ipencap: 0 total input packets 0 total output packets 0 packets shorter than header shows 0 packets dropped due to policy 0 packets with possibly spoofed local addresses 0 packets were dropped due to full output queue 0 input bytes 0 output bytes 0 protocol family mismatches 0 attempts to use tunnel with unspecified endpoint(s) tcp: 108 packets sent 48 data packets (4044 bytes) 0 data packets (0 bytes) retransmitted 0 fast retransmitted packets 36 ack-only packets (36 delayed) 0 URG only packets 0 window probe packets 0 window update packets 24 control packets 0 packets software-checksummed 100 packets received 44 acks (for 3960 bytes) 12 duplicate acks 0 acks for unsent data 0 acks for old data 60 packets (42320 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some duplicate data (0 bytes duplicated) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 1 discarded for bad checksum 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 101 packets software-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 12 connection requests 0 connection accepts 12 connections established (including accepts) 12 connections closed (including 0 drops) 0 connections drained 0 embryonic connections dropped 56 segments updated rtt (of 60 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 24 correct data packet header predictions 4 PCB cache misses 4 dropped due to no socket 0 ECN connections accepted 0 ECE packets received 0 CWR packets received 0 CE packets received 0 ECT packets sent 0 ECE packets sent 0 CWR packets sent cwr by fastrecovery: 0 cwr by timeout: 0 cwr by ecn: 0 0 bad connection attempts 0 SYN packets dropped due to queue or memory full 0 SYN cache entries added 0 hash collisions 0 completed 0 aborted (no space to build PCB) 0 timed out 0 dropped due to overflow 0 dropped due to bucket overflow 0 dropped due to RST 0 dropped due to ICMP unreachable 0 SYN,ACKs retransmitted 0 duplicate SYNs received for entries already in the cache 0 SYNs dropped (no route or no space) 0 SYN cache seeds with new random 293 hash bucket array size in current SYN cache 0 entries in current SYN cache, limit is 10255 0 longest bucket length in current SYN cache, limit is 105 0 uses of current SYN cache left 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options received 0 SACK options sent 0 SACK options dropped udp: 390 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 14 with no checksum 135 input packets software-checksummed 0 output packets software-checksummed 0 dropped due to no socket 342 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 0 dropped due to full socket buffers 48 delivered 48 datagrams output 0 missed PCB cache ipsec: 0 input IPsec packets 0 output IPsec packets 0 input bytes 0 output bytes 0 input bytes, decompressed 0 output bytes, uncompressed 0 packets dropped on input 0 packets dropped on output 0 packets that failed crypto processing 0 packets for which no XFORM was set in TDB received 0 packets for which no TDB was found esp: 0 input ESP packets 0 output ESP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets with bad encryption received 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad payload size or padding received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 output packets could not be sent 0 input UDP encapsulated ESP packets 0 output UDP encapsulated ESP packets 0 UDP packets for non-encapsulating TDB received 0 raw ESP packets for encapsulating TDB received 0 input bytes 0 output bytes ah: 0 input AH packets 0 output AH packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad authenticator length received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 output packets could not be sent 0 input bytes 0 output bytes etherip: 0 packets shorter than header shows 0 packets were dropped due to full output queue 0 packets were dropped because of no interface/bridge information 0 packets dropped due to policy 0 packets dropped for other reasons 0 input ethernet-in-IP packets 0 output ethernet-in-IP packets 0 input bytes 0 output bytes ipcomp: 0 input IPCOMP packets 0 output IPCOMP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed (de)compression processing 0 output packets could not be sent 0 packets less than minimum compression length 0 input bytes 0 output bytes carp: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 transitions to master pfsync: 0 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 0 stale states 0 failed state lookup/inserts 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error divert: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors pflow: 0 flows sent 0 packets sent 0 send failed due to mbuf memory error 0 send error ip6: 841 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 4 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 839 multicast packets which we don't join Input packet histogram: UDP: 4 ICMP6: 837 Mbuf statistics: 0 one mbufs 0 one ext mbufs 0 two or more ext mbufs 0 tunneling packets that can't find gif 0 packets discarded due to too many headers 0 failures of source address selection 0 forward cache hit 0 forward cache miss divert6: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors icmp6: 0 calls to icmp6_error 0 errors not generated because old message was icmp6 or so 0 errors not generated because of rate limitation Output packet histogram: multicast listener report: 4 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output On Fri, 20 Mar 2020 at 10:55, Tom Smyth <tom.sm...@wirelessconnect.eu> wrote: > > Hi Stefan, > I have attached the hostname.bridge101, > and the output of ifconfig bridge101 (both after reboot and after the > restart of the interface > > The server is in production so I'll run those commands you requested, > I will build up a current box > to drop in and test also Thanks > Tom Smyth > > On Fri, 20 Mar 2020 at 07:54, Stefan Sperling <s...@stsp.name> wrote: > > > > On Fri, Mar 20, 2020 at 01:50:42AM +0000, Tom Smyth wrote: > > > >Synopsis: <bridge with alot of ports does not forward frames after > > > >reboot > > > > >Category: <Network > > > > >Environment: > > > System : OpenBSD 6.6 > > > Details : OpenBSD 6.6 (GENERIC.MP) #7: Thu Mar 12 11:55:22 MDT 2020 > > > r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > > > > > Architecture: OpenBSD.amd64 > > > Machine : amd64 > > > >Description: > > > <if you configure a bridge with hostname.bridge and add 90 vlans as > > > protected ports > > > the bridge will not forward frames however if you run sh > > > /etc/netstart bridgex it does > > > start to forward > > > ifconfig bridge101 after reboot compared with ifconfig bridge101 after > > > the restart of the interface > > > using sh /etc/netstart appear very similar, the vlans appear to be > > > members of the bridge fine > > > both the bridge appears to learn mac addresses on ports both after > > > reboot and after manual restart > > > of interface) > > > the only difference that I observed was the interface index > > > after reboot bridge index was 6 > > > after restarting the interface the bridge index was 98 > > > > >How-To-Repeat: > > > <create a hostname.bridge101 file add 90 vlans as protected ports > > > reboot the machine > > > > >Fix: > > > <restarting the bridge interface using sh /etc/netstart bridge101 worked > > > I have added sh /etc/netstart bridge101 to /etc/rc.local > > > > Tom, could you share your hostname.bridge101 file? > > That might make it easier for others to reproduce the issue. > > > > Is there any obvious difference in the counters displayed by > > netstat -nI bridge101 or netstat -s in the working vs non-working states? > > > > And does it also happen on -current? > > > > -- > Kindest regards, > Tom Smyth. -- Kindest regards, Tom Smyth.