relayd - Hosts flapping unexpectedly

[Dan Carley]

Hi,

We've been playing with relayd recently - both from 4.5 and the latest
snapshot.

Approximately every hour we are seeing one or two state changes logged. But
I can't see reason for the change of state and there doesn't appear to be a
pattern in the way that the hosts are failed.

According to `relayctl sh su` we have around 30 redirects and 60 hosts. All
of the check modes are TCP. The only other significant options are:

timeout 1000
log updates

An example of the state changes we are seeing:

May 20 13:27:37 sysbuild-obsd45 relayd[30524]: host xxx.xxx.xxx.157, check
tcp (10ms), state up - down, availability 89.17%
May 20 13:27:47 sysbuild-obsd45 relayd[17005]: table lb_ev5_https: 0 added,
1 deleted, 0 changed, 0 killed
May 20 13:27:47 sysbuild-obsd45 relayd[30524]: host xxx.xxx.xxx.157, check
tcp (23ms), state down - up, availability 89.18%
May 20 13:27:57 sysbuild-obsd45 relayd[17005]: table lb_ev5_https: 1 added,
0 deleted, 0 changed, 0 killed

Obviously the host wasn't failed because the check exceeded the timeout
period. Does that mean that the TCP socket returned as closed/RST within
10ms?

A tcpdump of traffic from the relayd host doesn't reveal anything
particularly unusual. Except that no check was attempted at 13:27:37. Would
this have been skipped due to an offending check at 13:27:27?

13:26:57.051947 xxx.xxx.xxx.103.36021  xxx.xxx.xxx.157.443: S
4024993942:4024993942(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale
0,nop,nop,timestamp 3310083888 0 (DF)
13:26:57.052612 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.36021: S
1781097887:1781097887(0) ack 4024993943 win 5792 mss 1460,sackOK,timestamp
1848303010 3310083888,nop,wscale 2 (DF)
13:26:57.052689 xxx.xxx.xxx.103.36021  xxx.xxx.xxx.157.443: . ack 1 win
16384 nop,nop,timestamp 3310083888 1848303010 (DF)
13:26:57.060215 xxx.xxx.xxx.103.36021  xxx.xxx.xxx.157.443: F 1:1(0) ack 1
win 16384 nop,nop,timestamp 3310083888 1848303010 (DF)
13:26:57.060920 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.36021: F 1:1(0) ack 2
win 1448 nop,nop,timestamp 1848303012 3310083888 (DF)
13:26:57.062337 xxx.xxx.xxx.103.36021  xxx.xxx.xxx.157.443: . ack 2 win
16384 nop,nop,timestamp 3310083888 1848303012 (DF)

13:27:07.082681 xxx.xxx.xxx.103.1746  xxx.xxx.xxx.157.443: S
3987093519:3987093519(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale
0,nop,nop,timestamp 3472130146 0 (DF)
13:27:07.083370 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.1746: S
1795324639:1795324639(0) ack 3987093520 win 5792 mss 1460,sackOK,timestamp
1848305518 3472130146,nop,wscale 2 (DF)
13:27:07.083496 xxx.xxx.xxx.103.1746  xxx.xxx.xxx.157.443: . ack 1 win
16384 nop,nop,timestamp 3472130146 1848305518 (DF)
13:27:07.091940 xxx.xxx.xxx.103.1746  xxx.xxx.xxx.157.443: F 1:1(0) ack 1
win 16384 nop,nop,timestamp 3472130146 1848305518 (DF)
13:27:07.092726 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.1746: F 1:1(0) ack 2
win 1448 nop,nop,timestamp 1848305520 3472130146 (DF)
13:27:07.093700 xxx.xxx.xxx.103.1746  xxx.xxx.xxx.157.443: . ack 2 win
16384 nop,nop,timestamp 3472130146 1848305520 (DF)

13:27:17.099730 xxx.xxx.xxx.103.17697  xxx.xxx.xxx.157.443: S
913231965:913231965(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale
0,nop,nop,timestamp 1801123610 0 (DF)
13:27:17.100270 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.17697: S
1801344162:1801344162(0) ack 913231966 win 5792 mss 1460,sackOK,timestamp
1848308022 1801123610,nop,wscale 2 (DF)
13:27:17.100387 xxx.xxx.xxx.103.17697  xxx.xxx.xxx.157.443: . ack 1 win
16384 nop,nop,timestamp 1801123610 1848308022 (DF)
13:27:17.110345 xxx.xxx.xxx.103.17697  xxx.xxx.xxx.157.443: F 1:1(0) ack 1
win 16384 nop,nop,timestamp 1801123610 1848308022 (DF)
13:27:17.56 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.17697: F 1:1(0) ack 2
win 1448 nop,nop,timestamp 1848308025 1801123610 (DF)
13:27:17.111464 xxx.xxx.xxx.103.17697  xxx.xxx.xxx.157.443: . ack 2 win
16384 nop,nop,timestamp 1801123610 1848308025 (DF)

13:27:27.130132 xxx.xxx.xxx.103.44072  xxx.xxx.xxx.157.443: S
2602652962:2602652962(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale
0,nop,nop,timestamp 490198935 0 (DF)
13:27:27.130771 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.44072: S
1805757957:1805757957(0) ack 2602652963 win 5792 mss 1460,sackOK,timestamp
1848310530 490198935,nop,wscale 2 (DF)
13:27:27.130841 xxx.xxx.xxx.103.44072  xxx.xxx.xxx.157.443: . ack 1 win
16384 nop,nop,timestamp 490198935 1848310530 (DF)
13:27:27.140176 xxx.xxx.xxx.103.44072  xxx.xxx.xxx.157.443: F 1:1(0) ack 1
win 16384 nop,nop,timestamp 490198935 1848310530 (DF)
13:27:27.141091 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.44072: F 1:1(0) ack 2
win 1448 nop,nop,timestamp 1848310532 490198935 (DF)
13:27:27.141354 xxx.xxx.xxx.103.44072  xxx.xxx.xxx.157.443: . ack 2 win
16384 nop,nop,timestamp 490198935 1848310532 (DF)

13:27:47.180243 xxx.xxx.xxx.103.15780  xxx.xxx.xxx.157.443: S
352869815:352869815(0) win 16384 mss 1460,nop,nop,sackOK,nop,wscale
0,nop,nop,timestamp 1128451500 0 (DF)
13:27:47.180983 xxx.xxx.xxx.157.443  xxx.xxx.xxx.103.15780: S
1835720746:1835720746(0) ack 352869816 win 

Re: Recommendations on a daily script to check syslog (or other) server security

[Dan Carley]

2009/4/14 LeRoy, Ted tle...@lsisolutions.com

 Hello folks,

 I'm pretty new to OpenBSD and BSD in general, but I have an OpenBSD
 Syslog server up and receiving data.  I'd like to have the system be
 pretty secure, and I'd like to monitor its security via a simple script
 that runs daily.

 Here's what I have in the script at the present time:

 { uptime ; date ; who ; ps -al ; cat /var/log/adduser ; cat
 /var/log/authlog ; cat /var/log/messages ; cat /var/log/secure ; cat
 /var/log/router ; }  daily-log.txt


Not necessarily the advice you are after, but- Sometimes less is more.
Unless you are fortunate to have enough free time to review all of that data
on a daily basis and spot the changes of relevance, then human nature
dictates that you will overlook the important parts, should they eventually
arrive.

Regards,

Re: Odd problem, may be related to relayd

[Dan Carley]

2009/4/5 Brian McCann bjmcc...@gmail.com

 I've seen similar problems...not with relayd, but it still may apply.  I
 had
 a server that was behind a Linksys router on a DSL connection, being
 accessed by a remote user .  The window size (iirc) at the remote user was
 lower then usual, and the DSL provider was blocking the ICMP messages to
 alter the window size.  We had to lower a setting in Windows at the server
 side to fix this.


Technically it won't be relayd that is the cause of your woes because it is
PF will be performing the grunt work of the TCP redirection.

Based on what Brian said, you may find that playing with 'scrub out' and
'max-mss' in your PF rules alleviates the issue.

Although in an ideal world the clients should be fixed.

Re: How do I monitor my PF based firewall?

[Dan Carley]

2009/3/4 Falk Brockerhoff - smartTERRA GmbH n...@smartterra.eu

 Am 04.03.2009 um 11:23 schrieb Lars Noodin:

 Or do you want visualization?
 http://www.openbsd.org/4.4_packages/i386/pfstat-2.3p0.tgz-long.html


 Yes, but I want to use cacti for visualization as I use it for
 anything else :)


If you're using 4.4 then `systat states` does the same job as pfstat and
doesn't require any installation.

For exporting to Cacti, until PF-MIBsm are in OpenSNMPd, you can apply them
to net-snmp. Either from ports or referencing them from a source tarball.
From memory you may have to write the Cacti definitions yourself though.

http://www.packetmischief.ca/openbsd/snmp/

Regards,

Re: How do I monitor my PF based firewall?

[Dan Carley]

On 04/03/2009, Jason Dixon ja...@dixongroup.net wrote:
 On Wed, Mar 04, 2009 at 02:17:30PM +0100, Falk Brockerhoff - smartTERRA GmbH
 wrote:
 Ok, this is a way we can go. Is there any possibility to use the extend
 feature with openbsd builtin snmpd?

 Not currently.

I don't believe there are any plan to do so. You should stick with the
more heavyweight net-snmp if you need extend/exec/pass.

Besides, extend always feels slightly hackish in my opinion. The MIBs
function well.

OpenBGP 4.3/4.4 Gotchas

[Dan Carley]

Hi,

I've run into a couple of gothas in the past week. This isn't so much a bug
report, because everything is fine in -current. But I hope it might serve to
save somebody some time if they stubble across it in the archives.

The first was experienced on a pair of 4.3 machines. Unlike any of our other
transit feeds, we have one provider which appears to re-advertise our own
prefixes back to our alternate routers. The routes are of course considered
invalid because they are not loop free and it hasn't caused us problems
previously. Except this week when applying an inbound filter with
softreconfig in yes and bgpctl reload. We observed all announcements
matching these re-advertised prefixes to be withdrawn from all transit peers
and not reannounced until the filter was removed.

This behaviour was thankfully not replicated with 4.4 in the lab, so we'll
be upgrading promptly. But we were having issues with our 4.4 peers keeping
sessions open to each other. This was resolved with r1.13 of bgpd/timer.c.
I'm curious though whether this will make it into the 4.4 errata as a
reliability fix?

Regards,
Dan