Hello,
Sorry for the delay in replying I dont get the opportunity to reboot
that box often and with the COVID-19 Crisis
Im trying to minimise maintenance that impacts customers
i also tried to split the number of vlans across an additional ix(4)
parent interface but this did not change the behaviour
Please find output of diagnostic commands below
#netstat -nl bridge101
netstat -nI bridge101 (after reboot (broken)
ngabr# cat netstat-nI-bridge101-afterreboot
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
bridge101 1500 <Link> 51546 0 2813599
0 0
ngabr# cat netstat-nIbridge101-postnetstart (after netstart (fixed))
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
bridge101 1500 <Link> 5119709 0 8565403
0 0
###########################################################################
#netstat -s
ngabr# cat netstat-s-afterreboot (broken)
ip:
3999 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
237 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
3092 packets not forwardable
0 redirects sent
10977 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
11122 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 fragment floods
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
3656 input datagrams software-checksummed
2837975 output datagrams software-checksummed
3335 multicast packets which we don't join
icmp:
225 calls to icmp_error
0 errors not generated because old message was icmp
0 errors not generated because of rate limitation
Output packet histogram:
destination unreachable: 225
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 echo requests to broadcast/multicast rejected
0 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ipencap:
0 total input packets
0 total output packets
0 packets shorter than header shows
0 packets dropped due to policy
0 packets with possibly spoofed local addresses
0 packets were dropped due to full output queue
0 input bytes
0 output bytes
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
54 packets sent
24 data packets (2022 bytes)
0 data packets (0 bytes) retransmitted
0 fast retransmitted packets
18 ack-only packets (18 delayed)
0 URG only packets
0 window probe packets
0 window update packets
12 control packets
0 packets software-checksummed
51 packets received
21 acks (for 1956 bytes)
6 duplicate acks
0 acks for unsent data
0 acks for old data
30 packets (21161 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some duplicate data (0 bytes duplicated)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
1 discarded for bad checksum
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
52 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
6 connection requests
0 connection accepts
6 connections established (including accepts)
6 connections closed (including 0 drops)
0 connections drained
0 embryonic connections dropped
27 segments updated rtt (of 30 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
12 correct data packet header predictions
3 PCB cache misses
3 dropped due to no socket
0 ECN connections accepted
0 ECE packets received
0 CWR packets received
0 CE packets received
0 ECT packets sent
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 0
cwr by timeout: 0
cwr by ecn: 0
0 bad connection attempts
0 SYN packets dropped due to queue or memory full
0 SYN cache entries added
0 hash collisions
0 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 SYN cache seeds with new random
293 hash bucket array size in current SYN cache
0 entries in current SYN cache, limit is 10255
0 longest bucket length in current SYN cache, limit is 105
0 uses of current SYN cache left
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options received
0 SACK options sent
0 SACK options dropped
udp:
186 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
6 with no checksum
64 input packets software-checksummed
0 output packets software-checksummed
0 dropped due to no socket
160 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
26 delivered
26 datagrams output
0 missed PCB cache
ipsec:
0 input IPsec packets
0 output IPsec packets
0 input bytes
0 output bytes
0 input bytes, decompressed
0 output bytes, uncompressed
0 packets dropped on input
0 packets dropped on output
0 packets that failed crypto processing
0 packets for which no XFORM was set in TDB received
0 packets for which no TDB was found
esp:
0 input ESP packets
0 output ESP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets with bad encryption received
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad payload size or padding received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input UDP encapsulated ESP packets
0 output UDP encapsulated ESP packets
0 UDP packets for non-encapsulating TDB received
0 raw ESP packets for encapsulating TDB received
0 input bytes
0 output bytes
ah:
0 input AH packets
0 output AH packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad authenticator length received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input bytes
0 output bytes
etherip:
0 packets shorter than header shows
0 packets were dropped due to full output queue
0 packets were dropped because of no interface/bridge information
0 packets dropped due to policy
0 packets dropped for other reasons
0 input ethernet-in-IP packets
0 output ethernet-in-IP packets
0 input bytes
0 output bytes
ipcomp:
0 input IPCOMP packets
0 output IPCOMP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed (de)compression processing
0 output packets could not be sent
0 packets less than minimum compression length
0 input bytes
0 output bytes
carp:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master
pfsync:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
0 failed state lookup/inserts
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
divert:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
pflow:
0 flows sent
0 packets sent
0 send failed due to mbuf memory error
0 send error
ip6:
11 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
0 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
4 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
11 multicast packets which we don't join
Input packet histogram:
ICMP6: 11
Mbuf statistics:
0 one mbufs
0 one ext mbufs
0 two or more ext mbufs
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
0 forward cache hit
0 forward cache miss
divert6:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
icmp6:
0 calls to icmp6_error
0 errors not generated because old message was icmp6 or so
0 errors not generated because of rate limitation
Output packet histogram:
multicast listener report: 4
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
#################################################################
ngabr# cat netstat-s-postnetstart working
ip:
8928 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
490 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
7270 packets not forwardable
0 redirects sent
19894 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
20397 output packets discarded due to no route
22 output datagrams fragmented
22 fragments created
0 datagrams that can't be fragmented
0 fragment floods
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
8029 input datagrams software-checksummed
15579679 output datagrams software-checksummed
7953 multicast packets which we don't join
icmp:
659 calls to icmp_error
0 errors not generated because old message was icmp
0 errors not generated because of rate limitation
Output packet histogram:
destination unreachable: 659
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 echo requests to broadcast/multicast rejected
0 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ipencap:
0 total input packets
0 total output packets
0 packets shorter than header shows
0 packets dropped due to policy
0 packets with possibly spoofed local addresses
0 packets were dropped due to full output queue
0 input bytes
0 output bytes
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
108 packets sent
48 data packets (4044 bytes)
0 data packets (0 bytes) retransmitted
0 fast retransmitted packets
36 ack-only packets (36 delayed)
0 URG only packets
0 window probe packets
0 window update packets
24 control packets
0 packets software-checksummed
100 packets received
44 acks (for 3960 bytes)
12 duplicate acks
0 acks for unsent data
0 acks for old data
60 packets (42320 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some duplicate data (0 bytes duplicated)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
1 discarded for bad checksum
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
101 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
12 connection requests
0 connection accepts
12 connections established (including accepts)
12 connections closed (including 0 drops)
0 connections drained
0 embryonic connections dropped
56 segments updated rtt (of 60 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
24 correct data packet header predictions
4 PCB cache misses
4 dropped due to no socket
0 ECN connections accepted
0 ECE packets received
0 CWR packets received
0 CE packets received
0 ECT packets sent
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 0
cwr by timeout: 0
cwr by ecn: 0
0 bad connection attempts
0 SYN packets dropped due to queue or memory full
0 SYN cache entries added
0 hash collisions
0 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 SYN cache seeds with new random
293 hash bucket array size in current SYN cache
0 entries in current SYN cache, limit is 10255
0 longest bucket length in current SYN cache, limit is 105
0 uses of current SYN cache left
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options received
0 SACK options sent
0 SACK options dropped
udp:
390 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
14 with no checksum
135 input packets software-checksummed
0 output packets software-checksummed
0 dropped due to no socket
342 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
48 delivered
48 datagrams output
0 missed PCB cache
ipsec:
0 input IPsec packets
0 output IPsec packets
0 input bytes
0 output bytes
0 input bytes, decompressed
0 output bytes, uncompressed
0 packets dropped on input
0 packets dropped on output
0 packets that failed crypto processing
0 packets for which no XFORM was set in TDB received
0 packets for which no TDB was found
esp:
0 input ESP packets
0 output ESP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets with bad encryption received
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad payload size or padding received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input UDP encapsulated ESP packets
0 output UDP encapsulated ESP packets
0 UDP packets for non-encapsulating TDB received
0 raw ESP packets for encapsulating TDB received
0 input bytes
0 output bytes
ah:
0 input AH packets
0 output AH packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets that failed verification received
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 possibly replayed packets received
0 packets with bad authenticator length received
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed crypto processing
0 output packets could not be sent
0 input bytes
0 output bytes
etherip:
0 packets shorter than header shows
0 packets were dropped due to full output queue
0 packets were dropped because of no interface/bridge information
0 packets dropped due to policy
0 packets dropped for other reasons
0 input ethernet-in-IP packets
0 output ethernet-in-IP packets
0 input bytes
0 output bytes
ipcomp:
0 input IPCOMP packets
0 output IPCOMP packets
0 packets from unsupported protocol families
0 packets shorter than header shows
0 packets dropped due to policy
0 packets for which no TDB was found
0 input packets that failed to be processed
0 packets for which no XFORM was set in TDB received
0 packets were dropped due to full output queue
0 packets where counter wrapping was detected
0 packets attempted to use an invalid TDB
0 packets got larger than max IP packet size
0 packets that failed (de)compression processing
0 output packets could not be sent
0 packets less than minimum compression length
0 input bytes
0 output bytes
carp:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 transitions to master
pfsync:
0 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
0 failed state lookup/inserts
0 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
divert:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
pflow:
0 flows sent
0 packets sent
0 send failed due to mbuf memory error
0 send error
ip6:
841 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (duplicates or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
0 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
4 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
839 multicast packets which we don't join
Input packet histogram:
UDP: 4
ICMP6: 837
Mbuf statistics:
0 one mbufs
0 one ext mbufs
0 two or more ext mbufs
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
0 forward cache hit
0 forward cache miss
divert6:
0 total packets received
0 dropped due to no socket
0 dropped due to full socket buffers
0 packets output
0 errors
icmp6:
0 calls to icmp6_error
0 errors not generated because old message was icmp6 or so
0 errors not generated because of rate limitation
Output packet histogram:
multicast listener report: 4
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
On Fri, 20 Mar 2020 at 10:55, Tom Smyth <[email protected]> wrote:
>
> Hi Stefan,
> I have attached the hostname.bridge101,
> and the output of ifconfig bridge101 (both after reboot and after the
> restart of the interface
>
> The server is in production so I'll run those commands you requested,
> I will build up a current box
> to drop in and test also Thanks
> Tom Smyth
>
> On Fri, 20 Mar 2020 at 07:54, Stefan Sperling <[email protected]> wrote:
> >
> > On Fri, Mar 20, 2020 at 01:50:42AM +0000, Tom Smyth wrote:
> > > >Synopsis: <bridge with alot of ports does not forward frames after
> > > >reboot >
> > > >Category: <Network >
> > > >Environment:
> > > System : OpenBSD 6.6
> > > Details : OpenBSD 6.6 (GENERIC.MP) #7: Thu Mar 12 11:55:22 MDT 2020
> > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > >
> > > Architecture: OpenBSD.amd64
> > > Machine : amd64
> > > >Description:
> > > <if you configure a bridge with hostname.bridge and add 90 vlans as
> > > protected ports
> > > the bridge will not forward frames however if you run sh
> > > /etc/netstart bridgex it does
> > > start to forward
> > > ifconfig bridge101 after reboot compared with ifconfig bridge101 after
> > > the restart of the interface
> > > using sh /etc/netstart appear very similar, the vlans appear to be
> > > members of the bridge fine
> > > both the bridge appears to learn mac addresses on ports both after
> > > reboot and after manual restart
> > > of interface)
> > > the only difference that I observed was the interface index
> > > after reboot bridge index was 6
> > > after restarting the interface the bridge index was 98 >
> > > >How-To-Repeat:
> > > <create a hostname.bridge101 file add 90 vlans as protected ports
> > > reboot the machine >
> > > >Fix:
> > > <restarting the bridge interface using sh /etc/netstart bridge101 worked
> > > I have added sh /etc/netstart bridge101 to /etc/rc.local
> >
> > Tom, could you share your hostname.bridge101 file?
> > That might make it easier for others to reproduce the issue.
> >
> > Is there any obvious difference in the counters displayed by
> > netstat -nI bridge101 or netstat -s in the working vs non-working states?
> >
> > And does it also happen on -current?
>
>
>
> --
> Kindest regards,
> Tom Smyth.
--
Kindest regards,
Tom Smyth.
