Hi, I'm hoping for some tips in architecting a fairly complex network
app that I would like to port to POE. It would be helpful to get some
advice on how to best use POE to avoid as many dead ends as possible
The basic situation is that we have a VERY limited high latency network
connection (2,400 baud, 1 second latency, $1.50/min) and we want to
squeeze the maximum possible down that connection.
Right now I have a fairly advanced proxy application is written in
vanilla perl with no special event architecture. However, for various
reasons it needs a substantial update and I'm considering porting to POE
to try and simplify the plumbing
Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a single
TCP connection across the link, optimising packet size where possible,
compressing all data using one of several (known in advance) compression
algorithms which can change packet by packet (knowledge of the protocol
allows minimal buffer flushing) and also some light rewriting of the
protocol on the fly (for efficiency)
The rough algorithm/components are as follows:
- Client side has an ordinary email program which talks to the proxy
application
- Proxy application on the client side is relatively smart and parses
the protocol (POP/SMTP) in some depth and rewrites the protocol a
little, eg don't waste a packet sending jsut the username, batch
username and password together in a single net packet.
- Server side can be less advanced because largely it's talking over a
high speed network and replies are largely just streamed anyway.
- Server side does a small amount of protocol rewriting, but largely
it's done at a higher level, eg dealing with connection drops/restarts
- Server side clearly deals with many, many more connections than the
client side (order of a few hundred to a few thousand outstanding client
connections at any one time). Compression buffers consume a few MB
each, therefore need to be careful of mem usage and ensure a forking
solution which will return memory to the OS eventually
- Net protocol uses a single tcp connection and encapsulates using a
simple protocol with a header stating original port, and compression
protocol (this allows us to increase compression efficiency and manage
tcp slow start more effectively)
- Compression buffer is flushed infrequently and the protocol layer
knows when we need to flush (complicates where to place the compression
code)
- We use a TCP CORK type emulation in the output buffer to decrease the
number of packets where possible, eg received a "username ok", we happen
to know that the "password ok/wrong" will be a millisecond behind, so
try and batch them. However, again the protocol layer knows when there
is nothing else coming and can force a flush
- We can change the compression algorithm per packet. Try to separate
data and text for best compression. Also if the connection is actually
discovered to be over broadband then turn off the slow compression and
use something more efficient
- I need to be able to get to the TCP connection handle to set a bunch
of advanced options
- Perhaps most importantly on the client side we need to carefully apply
backpressure on the application so that we don't end up with too much
data in the OS net buffers. With the network sending at 200 bytes per
second, it's very easy to end up with relatively enormous buffered
amounts of data and the client application quickly gets bored. The two
parts of the client proxy need to communicate and stop accepting data
from the client as soon as we have more than a small amount buffered on
the outbound side.
The question really is how best to break this up into POE components,
wheels, filters, etc Are there any gotchas to be aware of in terms of
scalability, mem, ability to do intra communication, etc
Roughly each side looks like (say) POP server + POP Client back to back,
but then there are some outer layers to wrap that inside another net
protocol for transmission, and an outer layer on that to handle the
compression. Annoyingly the compression and net layer needs to
communicate back to the inner layers to apply some backpressure and also
to be told when there is an explicit flush (prioritisation of packets is
also a future direction). Also because the network is so slow there
needs to be quite a lot of partial responses sent/received in order that
each end doesn't get bored waiting for the whole response
How might I best break this up in POE? What will my skeleton app look like?
This is a couple thousand lines of perl app today and not so easy to
maintain. Hoping to improve on that
Thanks for any thoughts
Ed W
