Re: help with pf

[Preston Norvell]

On 2007/12/01 3:04 PM, "Aaron" <[EMAIL PROTECTED]> muttered eloquently:

I believe I see the issue with general traffic flow.  The clue being that
you are being blocked by the generic block drop in log rule (you can get
rule numbers with 'pfctl -vvsr').  You have the destination port on the
source side of the rules.  See below...


> lan_net = "172.16.10.0/24"
> set skip on lo
> #set state-policy if-bound
> scrub in
> nat-anchor "ftp-proxy/*"
> rdr-anchor "ftp-proxy/*"
> rdr log on fxp0 inet proto { tcp udp } from 192.168.3.96/27 to carp0
> port 5900:5905 -> 172.16.10.26
> rdr on fxp3 proto tcp from $lan_net to any port 21 -> 127.0.0.1 port 8021
> nat log on fxp0 from $lan_net to any -> carp0
> pass in on fxp0
> pass out on fxp3
> block in log on fxp3
> pass out on fxp0 from $lan_net to any
> pass in on fxp3 inet proto tcp from $lan_net port { ssh www ntp https
> smtp imap imaps domain } to any
This should be:
pass in on fxp3 inet proto tcp from $lan_net to any port  { ssh www ntp
https smtp imap imaps domain } modulate state
> #pass in on fxp3 inet proto tcp from $lan_net port { ssh www ntp https
> smtp imap imaps domain } to any no state
> pass in on fxp3 inet proto udp from $lan_net port { domain ntp } to any
This should be:
pass in on fxp3 inet proto udp from $lan_net to any port  { domain ntp }
> pass in on fxp3 inet proto icmp from $lan_net to any

 

I'd probably do it a little different however, changing the pass out on fxp0
and pass in on fxp3 to:
pass out quick on fxp0 proto tcp from $lan_net to any modulate state
pass out quick on fxp0 proto { udp, icmp } from $lan_net to any keep state
pass out quick on fxp3 keep state
pass in quick on fxp3 proto tcp from $lan_net to any port { ssh www ntp
https smtp imap imaps domain } keep state
pass in quick on fxp3 proto udp from $lan_net to any port { domain ntp }
keep state

That may have more to do with my own mental logic and configuration style
than any real change in efficacy.

In general I find it most logical to put the general block rule(s) at the
top of the list and then pass/block quick thereafter.  That's largely a
personal choice first and out logic fits my brain best, but as your ruleset
grows it can also impact performance since the entire list of rules does not
necessarily have to be processed on all packets.

;P mn
--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions 
Phone:  (866) SERIALS (737-4257) ext 1094

Trouble with LSI Megaraid 8204/8208XLP in 4.2

[Preston Norvell]

Recently we were building a couple new amd64 machines and purchased a couple
LSI 8204XLP's on the speculation that they would be supported in 4.2 (though
only their bigger brother the 8208XLP was listed explicitly).

Only slightly to our surprise did we discover that the 8204XLP's would not
work.  The device would be found, but instead of the RAID1 logical disk, it
would report the two physical disks (RAID firmware configuration was
checked/rechecked/checkedsomemore).  We tried installing the OS on the low
order disk just to see what would happen.  The OS would install, but on
subsequent boot would fail at the "root device:" stage.

Feeling a certain amount of shame in defying the HCL, we purchased 8208XLPs
as a replacement.  These failed in exactly the same way.  It turns out that
they really are brothers in the sense that the 8204 is an 8208 card with a
blank spot where the second connector would be.

Then we decided to try 4.2 release (we were using -current), and the
drive(s) are not seen at all by the OS installer for 4.2.  We then switched
to a newer -current (from Nov. 1) and ran into the same problem as the
previous attempts with -current.

In some looking though, we see that the card appears to be identified with
the mpi driver instead of the mfi driver as it should, at least according to
mfi(4).

We now have two of each card here that aren't useful to us in the near term,
so we would be happy to send one of each of them along to the driver dev if
it would help future development.

The rest of this message is the dmesg from booting off the Nov 1 -current.

OpenBSD 4.2-current (RAMDISK_CD) #1295: Thu Nov  1 19:18:55 MDT 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 2146996224 (2047MB)
avail mem = 2075045888 (1978MB)
mainbus0 at root
acpi at mainbus0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Dual-Core AMD Opteron(tm) Processor 2212, 2010.58 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLU
SH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line
16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
pci0 at mainbus0 bus 0: configuration mode 1
"NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured
"NVIDIA MCP55 ISA" rev 0xa3 at pci0 dev 1 function 0 not configured
"NVIDIA MCP55 SMBus" rev 0xa3 at pci0 dev 1 function 1 not configured
ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: irq 7, version
1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: irq 10
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0:  SCSI0 5/cdrom
removable
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide1: using irq 11 for native-PCI interrupt
pciide2 at pci0 dev 5 function 1 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide2: using irq 5 for native-PCI interrupt
pciide3 at pci0 dev 5 function 2 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide3: using irq 10 for native-PCI interrupt
ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 6 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
nfe0 at pci0 dev 8 function 0 "NVIDIA MCP55 LAN" rev 0xa3: irq 10, address
00:30:48:7c:97:22
eephy0 at nfe0 phy 2: Marvell 88E1149 Gigabit PHY, rev. 1
nfe1 at pci0 dev 9 function 0 "NVIDIA MCP55 LAN" rev 0xa3: irq 11, address
00:30:48:7c:97:23
eephy1 at nfe1 phy 3: Marvell 88E1149 Gigabit PHY, rev. 1
ppb1 at pci0 dev 10 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "NEC PCIE-PCIX" rev 0x07
pci3 at ppb2 bus 3
ppb3 at pci2 dev 0 function 1 "NEC PCIE-PCIX" rev 0x07
pci4 at ppb3 bus 4
mpi0 at pci4 dev 6 function 0 "Symbios Logic SAS1068" rev 0x02: irq 5
scsibus1 at mpi0: 173 targets
sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct
fixed
sd0: 157066MB, 157067 cyl, 16 head, 127 sec, 512 bytes/sec, 321672960 sec
total
sd1 at scsibus1 targ 1 lun 0:  SCSI3 0/direct
fixed
sd1: 157066MB, 157067 cyl, 16 head, 127 sec, 512 bytes/sec, 321672960 sec
total
ppb4 at pci0 dev 13 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci5 at ppb4 bus 5
ppb5 at pci0 dev 14 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci6 at ppb5 bus 6
ppb6 at pci0 dev 15 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci7 at ppb6 bus 7
pchb0 at pci0 dev 24 function 0 "AMD AMD64 HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 Address Map" rev 0x00
pch

Re: Hoststated and stickiness based on cookie strings

[Preston Norvell]

In our testing today, the round robin appears to spread out traffic just as
it ought to.  Thanks much for that.

In further testing we turned up a couple other things, however.

The first is that we are having problems balancing to multiple app servers
when using the hash or loadbalance algos.  I think the reason for is that
our hash info does exist on initial connection or for atomic requests.

For atomic requests (requests consisting of a preconstructed URL with a
bunch of variables passed to the app, followed by a single response
containing the requested data) , there isn't necessarily distinctive
hashable information.  In this case even the source address may not be
enough given that multiple computers within the client systems may be making
the requests behind a NATing device; in our testing, all connections from a
single host end up on a single appserver (the high ID'd one if using 'hash',
a lower number if using 'loadbalance'). This would be bad in a number of
different scenarios.

For non-atomic requests (by far the largest segment of traffic), we use
cookies or a GET variable to maintain and recognize state depending on
whether they're using a browser or an XML api.  On initial request from the
client computers, they have neither the GET variable nor the cookie.

In the case of the browser, it's delivered to them after their initial
connection (like most session cookies I think).  Concatenated to the end of
this sessionid is a small string we use in subsequent requests to route the
traffic back to the app server the first connection was made on.  If an app
server looks at this little bit of text and sees that it is a string for a
different host, it believes the connection has failed over from a different
host, sends a new cookie to the browser and restarts the sessions with a
new, blank slate.

In the case of the XML api, it's quite similar, except that the session data
is passed to the client in the initial response to their first query.  As
part of the API the take the session info and turn it into a GET variable in
subsequent requests.

That's a long complicated way of restating that according to the protocol
definition we have appear to have no useful data for hashing on initial
connection, and I think this is why we end up in the same bucket for every
request.  It would be nice if it fell back to roundrobin in the case that
nothing really matched for the hash additions, or at least if we could tell
it do so as a non-default (since loadbalance always has a little bit of hash
from the source address).

The other thing we discovered in testing is that in a protocol the query
entity doesn't appear to work with the hash action, at least as of a build
of hoststated from this morning.  We were trying to see if we could
workaround the hashing issue buy feeding it some client-specific data.  The
following fails horribly:

 request query hash "*" from "SS_LibHash"

Cheers,

;P mn


On 2007/11/26 11:36 AM, "Preston Norvell"
<[EMAIL PROTECTED]> muttered eloquently:

> Thanks much.  We're working on getting it compiled and tested.
> 
> Assuming testing goes well, our last major hurdle is the deterministic
> portion of the load balancing, which it sounds like you are thinking about
> already.
> 
> Thanks much again,
> 
> ;P mn
> 
> 
> On 2007/11/22 8:09 AM, "Reyk Floeter" <[EMAIL PROTECTED]> muttered
> eloquently:
> 
>> ok, forget about this diff - i committed the first part (roundrobin)
>> but skipped the loadbalance part because it is wrong to look at the
>> client port in this case (because i want to provide session
>> persistence).
>> 
>> On Thu, Nov 22, 2007 at 12:51:10PM +0100, Reyk Floeter wrote:
>>> - please try the attached diff, it will fix the roundrobin mode by
>>> saving the last index and traversing to the next available host.
>>> 
>>> (you can also have a look at my little test program to verify the alg:
>>> http://team.vantronix.net/~reyk/q.c)
>>> 
>>> - i'm also looking into improving the loadbalance mode. the attached
>>> diff includes the source port in loadbalance mode and the destination
>>> (relay) port in loadbalance and hash mode. make also sure that you
>>> feed in other variables if you want to get better results, for example
>>> 
>>> request hash "Host"
>>> 
>>> to feed the virtual hostname into the hash/loadbalance hash.
>>> 
>>> reyk
>>> 
>>> Index: hoststated.h
>>> ===
>>> RCS file: /cvs/src/usr.sbin/hoststated/hoststated.h,v
>>> retrieving revision 1.81
>>> diff -u -p -r1.81 hoststated.h
>>> --- ho

Re: Hoststated and stickiness based on cookie strings

[Preston Norvell]

Thanks much.  We're working on getting it compiled and tested.

Assuming testing goes well, our last major hurdle is the deterministic
portion of the load balancing, which it sounds like you are thinking about
already.

Thanks much again,

;P mn


On 2007/11/22 8:09 AM, "Reyk Floeter" <[EMAIL PROTECTED]> muttered
eloquently:

> ok, forget about this diff - i committed the first part (roundrobin)
> but skipped the loadbalance part because it is wrong to look at the
> client port in this case (because i want to provide session
> persistence).
> 
> On Thu, Nov 22, 2007 at 12:51:10PM +0100, Reyk Floeter wrote:
>> - please try the attached diff, it will fix the roundrobin mode by
>> saving the last index and traversing to the next available host.
>> 
>> (you can also have a look at my little test program to verify the alg:
>> http://team.vantronix.net/~reyk/q.c)
>> 
>> - i'm also looking into improving the loadbalance mode. the attached
>> diff includes the source port in loadbalance mode and the destination
>> (relay) port in loadbalance and hash mode. make also sure that you
>> feed in other variables if you want to get better results, for example
>> 
>> request hash "Host"
>> 
>> to feed the virtual hostname into the hash/loadbalance hash.
>> 
>> reyk
>> 
>> Index: hoststated.h
>> ===
>> RCS file: /cvs/src/usr.sbin/hoststated/hoststated.h,v
>> retrieving revision 1.81
>> diff -u -p -r1.81 hoststated.h
>> --- hoststated.h 22 Nov 2007 10:09:53 - 1.81
>> +++ hoststated.h 22 Nov 2007 11:45:00 -
>> @@ -327,6 +327,7 @@ struct host {
>> u_longup_cnt;
>> intretry_cnt;
>> struct ctl_tcp_event  cte;
>> + intidx;
>>  };
>>  TAILQ_HEAD(hostlist, host);
>>  
>> Index: relay.c
>> ===
>> RCS file: /cvs/src/usr.sbin/hoststated/relay.c,v
>> retrieving revision 1.65
>> diff -u -p -r1.65 relay.c
>> --- relay.c 22 Nov 2007 10:09:53 - 1.65
>> +++ relay.c 22 Nov 2007 11:45:01 -
>> @@ -463,6 +463,7 @@ relay_init(void)
>> if (rlay->dstnhosts >= RELAY_MAXHOSTS)
>> fatal("relay_init: "
>>"too many hosts in table");
>> +host->idx = rlay->dstnhosts;
>> rlay->dsthost[rlay->dstnhosts++] = host;
>> }
>> log_info("adding %d hosts from table %s%s",
>> @@ -1876,10 +1877,14 @@ relay_hash_addr(struct sockaddr_storage
>> sin4 = (struct sockaddr_in *)ss;
>> p = hash32_buf(&sin4->sin_addr,
>>sizeof(struct in_addr), p);
>> +  p = hash32_buf(&sin4->sin_port,
>> +  sizeof(struct in_addr), p);
>> } else {
>> sin6 = (struct sockaddr_in6 *)ss;
>> p = hash32_buf(&sin6->sin6_addr,
>>sizeof(struct in6_addr), p);
>> +  p = hash32_buf(&sin6->sin6_port,
>> +  sizeof(struct in6_addr), p);
>> }
>>  
>> return (p);
>> @@ -1903,7 +1908,7 @@ relay_from_table(struct session *con)
>> case RELAY_DSTMODE_ROUNDROBIN:
>> if ((int)rlay->dstkey >= rlay->dstnhosts)
>> rlay->dstkey = 0;
>> -  idx = (int)rlay->dstkey++;
>> +  idx = (int)rlay->dstkey;
>> break;
>> case RELAY_DSTMODE_LOADBALANCE:
>> p = relay_hash_addr(&con->in.ss, p);
>> @@ -1933,6 +1938,8 @@ relay_from_table(struct session *con)
>> fatalx("relay_from_table: no active hosts, desynchronized");
>>  
>>   found:
>> + if (rlay->conf.dstmode == RELAY_DSTMODE_ROUNDROBIN)
>> +  rlay->dstkey = host->idx + 1;
>> con->retry = host->conf.retry;
>> con->out.port = table->conf.port;
>> bcopy(&host->conf.ss, &con->out.ss, sizeof(con->out.ss));
>> 

--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions 
Phone:  (866) SERIALS (737-4257) ext 1094

Re: Hoststated and stickiness based on cookie strings

[Preston Norvell]

On 2007/11/18 6:04 PM, "Preston Norvell"
<[EMAIL PROTECTED]> muttered eloquently:

 
> The first is a basic issue with load balancing.  No matter which algorithm
> we choose, initial traffic is extremely heavily waited towards the system in
> the table with the highest id.  In point of experience so far, the only time
> more than one host is reliably used is when using the roundrobin type of
> load-balancing.  If 'loadbalance' or 'hash' is used, 99.9% of traffic ends
> up on a single host; some will end up on other hosts, sometime momentarily
> though, and not what we've been able see as deterministically.  The
> situation with 'loadbalance' we understand since our test system on the
> internet is essentially coming from essentially one address (though even in
> limited testing with a hand full of additional requesting addresses, it
> appears that it works the same).
> 
> With a test of traffic from our test host with roundrobin (50 separate,
> simultaneous single request/response sessions run for several seconds), 797
> of the requests ended up at the high id host and 628 across the remaining 7
> (89 or 90 for each).
> > 

We have discovered the issue with this unbalanced balancing.  The root cause
appears to be some invalid assumptions in the roundrobin code in the
relay_from_table function in relay.c.

If you look at the config (snipped here for space), you will notice that we
have 16 hosts in the appx table.  Hosts 9-16 are offline until further
notice, and it's their existence in the table that is causing the roundrobin
to be more of a half-moon robin.  If we remove them from the table, the
balancing returns to normal.

Here's the theory, born out by experience and some snooping through the
code:

Basically when the requests start coming in, it tries #1 which is up and the
connection is sent there.  Then another connection comes in and it
roundrobins to #2 which is up so the connection is sent there, and so on and
so forth up to the 9th connection.  Then another connection comes in, it
roundrobins to #9 which isn't up so it chews through the table (in backwards
order?), and finds #8 up first so it sends it to #8.  Then the tenth
connection comes in, which it rounrobins to #10, which isn't up so it chews
through the table and finds #8 up first so it sends it to #8.  This happens
until it's gone through the remaining hosts in the table, then it resets to
the first item in the table, sends the next connection to #1, and the next
to #2, etc.  

Pardon me if I get the exact interpretation, but I haven't done C
programming in a very long time.  The balancer logic for roundrobin iterates
through the hosts in the table by incrementing a tracking variable in the
relay's struct.  It then breaks, and hops to the while loop to check if the
host is up.  If it's not up it iterates through the rest of the hosts in the
table until it finds one or runs out of items in the table.  If it runs out
it decides to run through the entire table from the top.  In either of these
cases, I believe the connection is dispatched to the first item it finds,
rather than the next one it should go to according to the theory of
roundrobin.

This exactly matches the mathematical distribution of the sessions in our
logs.  In general the roundrobin seems to suffer with an assumption that a
large block of hosts wouldn't be down at one time.  This is an invalid
assumption (intentional or not) for a production environment where someone
may need to take down a substantial number of hosts at once for maintenance.
In addition, since the same logic is used for all three algorithms
(roundrobin, loadbalance, and hash), it explains why the non-roundrobin
modes were producing consistently incorrect balancing as well.  There is
some stickiness provided by the hash in these additional modes, but their
balancing seems to be similarly borked but in a more complicated fashion.



Thoughts?

Thanks much,

;P mn

--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions <http://www.serialssolutions.com>
Phone:  (866) SERIALS (737-4257) ext 1094

Re: Hoststated and randomly dropped connections

[Preston Norvell]

On 2007/11/20 3:49 PM, "Stuart Henderson" <[EMAIL PROTECTED]> muttered
eloquently:

> On 2007/11/20 14:46, Preston Norvell wrote:
>> kern.maxfiles=32768
> 
> 32k seems excessive ..
> 

It may be.  As I said I pulled it from a balancer config for *BSD for a
commercial balancer.  As we continue testing I will likely back it down to
see what effect it has and as I try to determine what causes the remaining
failed connections (we're up to 318 out of 288199 now).  It will also be
interesting when we bump up the number different relays the load balancer is
handling.


>>> On 2007/11/19 2:20 AM, "Reyk Floeter" <[EMAIL PROTECTED]> muttered
>>> eloquently:
>>> 
>>>> net.inet.ip.ifq.maxlen
> 
> did you look at this?
> 
> if you see >0 in net.inet.ip.ifq.drops, you quite likely need
> to bump it.
> 

Yeah.  It has been sitting at zero since we started this phase of testing,
without any reboots so I took it to mean things were probably ok there.

;P mn

--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions <http://www.serialssolutions.com>
Phone:  (866) SERIALS (737-4257) ext 1094

Re: Hoststated and randomly dropped connections

[Preston Norvell]

After some research on sysctl values for other loadbalancers, I borrowed
some best practices values and set kern.maxfiles and kern.somaxconn to
higher values and it seems to have largely helped the issue.  With 181458
sessions we've seen 257 errors (from any source, may not be the specific
issue we were having before), which is about two orders of magnitude fewer
than before.  I'll be researching these remaining errors, but we're
hammering a single server enough (due to a load balancer algo issue) that it
may simply be a little too much for the app server itself.

These are the values I am using now:
kern.maxfiles=32768
kern.somaxconn=256

I have also increased the the soft limit for files to 512 daemon.

Between these settings, I appear to be in pretty good shape now.

Thanks everyone for the help.

;P mn

On 2007/11/19 2:26 PM, "Preston Norvell"
<[EMAIL PROTECTED]> muttered eloquently:

> Thanks much, 
> 
> I'll start digging into the sysctls.  I'm reasonably certain it isn't
> something with the app servers, because in the tcpdump output I can see the
> conversation between the load balancer and the app server complete
> successfully (all aspects of the request/response even), it's just from the
> load balancer to the client machines that gets tetchy.  I will try the retry
> value though; it certainly wouldn't hurt and sounds like a good idea.
> 
> Thanks again,
> 
> ;P mn
> 
> 
> On 2007/11/19 2:20 AM, "Reyk Floeter" <[EMAIL PROTECTED]> muttered
> eloquently:
> 
>> hi!
>> 
>> are you sure that the apaches are not dropping the connections when
>> you reach a specific limit of max connections? i've seen problems like
>> this with apache2+linux webservers.
>> 
>> - make sure that you tuned some sysctls for hoststated. for example
>> kern.maxfiles, kern.somaxconn, kern.maxclusters,
>> net.inet.ip.ifq.maxlen. you have to be very careful when tuning the
>> sysctls, but you mostly always have to bump them up for L7 load
>> balancing.
>> 
>> - try out the "retry" option in the table configuration. this is a
>> work-around for buggy backends. i experienced that the _backend_
>> servers sometimes drop the inbound connection attempts, so i added
>> this option to immediatly retry it... which works very well.
>> 
>> table foo {
>> real port 80
>> check http '/ZendPlatform/client/getPing.php' code 200
>> 
>> host $www01 retry 2
>> host $www02 retry 2
>> host $www03 retry 2
>> ...
>> 
>> demote carp
>> }
>> 
>> reyk
>> 
>> On Mon, Nov 19, 2007 at 12:14:18AM -0800, Preston Norvell wrote:
>>> We have been trying to migrate from an Apache proxy balancer to hoststated
>>> and have run into a couple issues, one of which I have asked about and the I
>>> write about now.
>>> 
>>> We are using 4.2-stable:
>>> OpenBSD mesh1 4.2 GENERIC.MP#1378 amd64
>>> 
>>> This particular issue is rather odd, such that I'm afraid my description may
>>> be somewhat confusing, but here goes...
>>> 
>>> We are doing layer 7 http load balancing for an application hosted on 8+
>>> machines behind the hoststated box for clients on the Internet.  In our
>>> testing, we seem to have an issue with hoststated somewhat randomly dropping
>>> inbound connections to a resource behind it.  It is not exactly
>>> deterministic, in that we cannot seem to generate a specific packet to make
>>> the connection fail, but it's just about statistically guaranteed to fail.
>>> The failure rate goes up as the traffic increases, though even a sequential
>>> run of 1000 single connections is likely to fail once or twice.
>>> 
>>>> From a tcpdump standpoint, I see the connection established through the
>>>> load
>>> balancer.  The GET request is issued from the client machine, which is
>>> delivered by hoststated to the server, which dutifully considers the request
>>> and returns a valid response.  Oddly though, on the client-facing side of
>>> the load balancer,  immediately after the GET request is received, a FIN is
>>> sent from the load balancer itself.
>>> 
>>> As stated, the likelihood of this occurring goes up with more traffic, even
>>> with low-bandwidth request/response sequences.  The only message of any
>>> import in any log I've looked in is the following from /var/log/daemon:
>>> 
>>> Nov 18 17:17:02 mesh1 hoststated[1945]: relay appx, session 2948 (50
>>> active), a.b.c.d -> 10.100.0.208:8080, s

Re: Hoststated and randomly dropped connections

[Preston Norvell]

Thanks much, 

I'll start digging into the sysctls.  I'm reasonably certain it isn't
something with the app servers, because in the tcpdump output I can see the
conversation between the load balancer and the app server complete
successfully (all aspects of the request/response even), it's just from the
load balancer to the client machines that gets tetchy.  I will try the retry
value though; it certainly wouldn't hurt and sounds like a good idea.

Thanks again,

;P mn


On 2007/11/19 2:20 AM, "Reyk Floeter" <[EMAIL PROTECTED]> muttered
eloquently:

> hi!
> 
> are you sure that the apaches are not dropping the connections when
> you reach a specific limit of max connections? i've seen problems like
> this with apache2+linux webservers.
> 
> - make sure that you tuned some sysctls for hoststated. for example
> kern.maxfiles, kern.somaxconn, kern.maxclusters,
> net.inet.ip.ifq.maxlen. you have to be very careful when tuning the
> sysctls, but you mostly always have to bump them up for L7 load
> balancing.
> 
> - try out the "retry" option in the table configuration. this is a
> work-around for buggy backends. i experienced that the _backend_
> servers sometimes drop the inbound connection attempts, so i added
> this option to immediatly retry it... which works very well.
> 
> table foo {
> real port 80
> check http '/ZendPlatform/client/getPing.php' code 200
> 
> host $www01 retry 2
> host $www02 retry 2
> host $www03 retry 2
> ...
> 
> demote carp
> }
> 
> reyk
> 
> On Mon, Nov 19, 2007 at 12:14:18AM -0800, Preston Norvell wrote:
>> We have been trying to migrate from an Apache proxy balancer to hoststated
>> and have run into a couple issues, one of which I have asked about and the I
>> write about now.
>> 
>> We are using 4.2-stable:
>> OpenBSD mesh1 4.2 GENERIC.MP#1378 amd64
>> 
>> This particular issue is rather odd, such that I'm afraid my description may
>> be somewhat confusing, but here goes...
>> 
>> We are doing layer 7 http load balancing for an application hosted on 8+
>> machines behind the hoststated box for clients on the Internet.  In our
>> testing, we seem to have an issue with hoststated somewhat randomly dropping
>> inbound connections to a resource behind it.  It is not exactly
>> deterministic, in that we cannot seem to generate a specific packet to make
>> the connection fail, but it's just about statistically guaranteed to fail.
>> The failure rate goes up as the traffic increases, though even a sequential
>> run of 1000 single connections is likely to fail once or twice.
>> 
>>> From a tcpdump standpoint, I see the connection established through the load
>> balancer.  The GET request is issued from the client machine, which is
>> delivered by hoststated to the server, which dutifully considers the request
>> and returns a valid response.  Oddly though, on the client-facing side of
>> the load balancer,  immediately after the GET request is received, a FIN is
>> sent from the load balancer itself.
>> 
>> As stated, the likelihood of this occurring goes up with more traffic, even
>> with low-bandwidth request/response sequences.  The only message of any
>> import in any log I've looked in is the following from /var/log/daemon:
>> 
>> Nov 18 17:17:02 mesh1 hoststated[1945]: relay appx, session 2948 (50
>> active), a.b.c.d -> 10.100.0.208:8080, session failed
>> 
>> There are no blocks in pf, and no errors as far as the app server is
>> concerned.  The connections work fine through a similarly configured OpenBSD
>> firewall without hoststated in the loop.
>> 
>> I'm not sure where to start looking next to narrow down the issue farther,
>> does anyone have any suggestions?
>> 
>> Thanks much,
>> 
>> ;P mn
>> 
>> --
>> Preston M Norvell <[EMAIL PROTECTED]>
>> Systems/Network Administrator
>> Serials Solutions <http://www.serialssolutions.com>
>> Phone:  (866) SERIALS (737-4257) ext 1094
>> 

--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions <http://www.serialssolutions.com>
Phone:  (866) SERIALS (737-4257) ext 1094

Hoststated and randomly dropped connections

[Preston Norvell]

We have been trying to migrate from an Apache proxy balancer to hoststated
and have run into a couple issues, one of which I have asked about and the I
write about now.

We are using 4.2-stable:
OpenBSD mesh1 4.2 GENERIC.MP#1378 amd64

This particular issue is rather odd, such that I'm afraid my description may
be somewhat confusing, but here goes...

We are doing layer 7 http load balancing for an application hosted on 8+
machines behind the hoststated box for clients on the Internet.  In our
testing, we seem to have an issue with hoststated somewhat randomly dropping
inbound connections to a resource behind it.  It is not exactly
deterministic, in that we cannot seem to generate a specific packet to make
the connection fail, but it's just about statistically guaranteed to fail.
The failure rate goes up as the traffic increases, though even a sequential
run of 1000 single connections is likely to fail once or twice.

>From a tcpdump standpoint, I see the connection established through the load
balancer.  The GET request is issued from the client machine, which is
delivered by hoststated to the server, which dutifully considers the request
and returns a valid response.  Oddly though, on the client-facing side of
the load balancer,  immediately after the GET request is received, a FIN is
sent from the load balancer itself.

As stated, the likelihood of this occurring goes up with more traffic, even
with low-bandwidth request/response sequences.  The only message of any
import in any log I've looked in is the following from /var/log/daemon:

Nov 18 17:17:02 mesh1 hoststated[1945]: relay appx, session 2948 (50
active), a.b.c.d -> 10.100.0.208:8080, session failed

There are no blocks in pf, and no errors as far as the app server is
concerned.  The connections work fine through a similarly configured OpenBSD
firewall without hoststated in the loop.

I'm not sure where to start looking next to narrow down the issue farther,
does anyone have any suggestions?

Thanks much,

;P mn

--
Preston M Norvell <[EMAIL PROTECTED]>
Systems/Network Administrator
Serials Solutions 
Phone:  (866) SERIALS (737-4257) ext 1094

Hoststated and stickiness based on cookie strings

[Preston Norvell]

We have been trying to migrate from an Apache proxy balancer to hoststated
and have run into a couple issues, one of which a document and another I
will send along later.

We are using 4.2-stable:
OpenBSD mesh1 4.2 GENERIC.MP#1378 amd64

Our first issue is in getting load balancing to occur in a deterministic
fashion.  There are actually two parts to this, but first a basic
description of what we are trying to do...

We are doing layer 7 balancing with http for hosts connecting from the
Inernet.  Traffic is to be spread across a sizeable number of machines (8
during this testing, but 16 or more in production).  Because the application
to be balanced is very sensitive to sessions (requests are made, results
take a while to queue and are typically referenced in subsequent requests),
cookies or GET variables (depending on how the application is accessed) are
used to manage state.

The first is a basic issue with load balancing.  No matter which algorithm
we choose, initial traffic is extremely heavily waited towards the system in
the table with the highest id.  In point of experience so far, the only time
more than one host is reliably used is when using the roundrobin type of
load-balancing.  If 'loadbalance' or 'hash' is used, 99.9% of traffic ends
up on a single host; some will end up on other hosts, sometime momentarily
though, and not what we've been able see as deterministically.  The
situation with 'loadbalance' we understand since our test system on the
internet is essentially coming from essentially one address (though even in
limited testing with a hand full of additional requesting addresses, it
appears that it works the same).

With a test of traffic from our test host with roundrobin (50 separate,
simultaneous single request/response sessions run for several seconds), 797
of the requests ended up at the high id host and 628 across the remaining 7
(89 or 90 for each).

For our application, the level of unbalanced weighting towards a host would
be bad, very bad.  A little further down is our hoststated.conf contents.
Our hope is that we are missing something basic in configuration that we are
missing.  Is this the expected behavior or have we misconfigured something?

The second issue we is related to this in that we have a value in the
sessionid in the cookie (or the GET variables depending on how folks are
accessing the system).  This variable helps us point the request at a
specific machine in the load balance pool since, as stated, it is very
important subsequent requests end up going to the same host.  For instance,
the variable might be something like sessionID=1234bcdfadedf.APP1.  With our
Apache load balancer we can grab this value from the request, extract the
APP1 off the end, then route to a specific host associated with APP1.  It
doesn't appear from the ways we have interpreted the man page, nor from any
way we have attempted to configure it that this kind of deterministic
routing is possible in hoststated.  Is possible with hoststated?

Any help is greatly appreciated.

The rest of this post is our config (some of the various permutations of
load balancing are left commented out):


#== MACROS
 #== IPs
CS1="10.100.0.201"
CS2="10.100.0.202"
CS3="10.100.0.203"
CS4="10.100.0.204"
CS5="10.100.0.205"
CS6="10.100.0.206"
CS7="10.100.0.207"
CS8="10.100.0.208"
CS9="10.100.0.209"
CS10="10.100.0.210"
CS11="10.100.0.211"
CS12="10.100.0.212"
CS13="10.100.0.213"
CS14="10.100.0.214"
CS15="10.100.0.215"
CS16="10.100.0.216"

#== GLOBAL OPTIONS
interval 10
timeout 200
prefork 5
log updates

#== TABLES
# mapped to tables in pf.conf
table appx {
real port 8080

#== check hosts in table via hash of systemStatus
#check http "/systemStatus" \
#digest "fc3626a53938f22eed804aaec987cfe0b762b9f8"
check icmp
 
#== the actual hosts to push into the table
host $CS1
host $CS2
host $CS3
host $CS4
host $CS5
host $CS6
host $CS7
host $CS8
host $CS9
host $CS10
host $CS11
host $CS12
host $CS13
host $CS14
host $CS15
host $CS16
}

#== PROTOCOLS
# Layer 7 games

protocol appx {
#== define what protocol we're manipulating
protocol http

#== RFC2616
header append "$REMOTE_ADDR" to "X-Forwarded-For"
header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"

#== grab session in URL for sru and log
# order is important

#== colon delimters
request url hash "::*::" log

#== url encoded colon delimiters
request url hash "%3A%3A*%3A%3A" log

#== grab session from Cookie header in app and log
request cookie hash "csSessionId" log

#== TCP performance options
tcp { nodelay, sack, socket buffer 65536, backlog 128 }
}

#== RELAYS
# this is where the things actually happen
relay appx {
#== use protocol defined above
protocol appx

#== bind to ip and listen
listen on