It's unfair to ask for a comprehensive ("avoid as many dead ends as
possible") analysis of a complex system that is only described in
basic terms. The best anyone can do under these circumstances is
provide generic advice. Specific advice will invariably miss some
hitherto unexplained detail that invalidates the analyst's assumptions
and sets them back. Repeat for every mis- or insufficiently
understood detail, and the frustration (and time cost) mount. One set
of dead ends has been replaced with another.
The best way to break an application into POE entities is not to.
Decompose the application into entities and relationships that make
sense for the application. Determine the requirements first, then
apply POE to fulfill them.
As described, your application seems to have a lot of special
requirements. POE::Kernel has primitive methods that may be suitable
if POE::Component and POE::Wheel classes are too high-level or too
generic. Wheels and components use POE::Kernel as well, so new
abstractions will work alongside existing ones.
If you have the time, I recommend looking at _Design Methods for
Reactive Systems_ (http://www.amazon.com/gp/product/1558607552?tag=poeperlorg-20
). I'm about 200 pages into it, and I like it a lot so far.
--
Rocco Caputo - rcap...@pobox.com
On May 23, 2009, at 07:48, Ed Wildgoose wrote:
Hi, I'm hoping for some tips in architecting a fairly complex
network app that I would like to port to POE. It would be helpful
to get some advice on how to best use POE to avoid as many dead ends
as possible
The basic situation is that we have a VERY limited high latency
network connection (2,400 baud, 1 second latency, $1.50/min) and we
want to squeeze the maximum possible down that connection.
Right now I have a fairly advanced proxy application is written in
vanilla perl with no special event architecture. However, for
various reasons it needs a substantial update and I'm considering
porting to POE to try and simplify the plumbing
Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a
single TCP connection across the link, optimising packet size where
possible, compressing all data using one of several (known in
advance) compression algorithms which can change packet by packet
(knowledge of the protocol allows minimal buffer flushing) and also
some light rewriting of the protocol on the fly (for efficiency)
The rough algorithm/components are as follows:
- Client side has an ordinary email program which talks to the proxy
application
- Proxy application on the client side is relatively smart and
parses the protocol (POP/SMTP) in some depth and rewrites the
protocol a little, eg don't waste a packet sending jsut the
username, batch username and password together in a single net packet.
- Server side can be less advanced because largely it's talking over
a high speed network and replies are largely just streamed anyway.
- Server side does a small amount of protocol rewriting, but largely
it's done at a higher level, eg dealing with connection drops/restarts
- Server side clearly deals with many, many more connections than
the client side (order of a few hundred to a few thousand
outstanding client connections at any one time). Compression
buffers consume a few MB each, therefore need to be careful of mem
usage and ensure a forking solution which will return memory to the
OS eventually
- Net protocol uses a single tcp connection and encapsulates using a
simple protocol with a header stating original port, and compression
protocol (this allows us to increase compression efficiency and
manage tcp slow start more effectively)
- Compression buffer is flushed infrequently and the protocol layer
knows when we need to flush (complicates where to place the
compression code)
- We use a TCP CORK type emulation in the output buffer to decrease
the number of packets where possible, eg received a "username ok",
we happen to know that the "password ok/wrong" will be a millisecond
behind, so try and batch them. However, again the protocol layer
knows when there is nothing else coming and can force a flush
- We can change the compression algorithm per packet. Try to
separate data and text for best compression. Also if the connection
is actually discovered to be over broadband then turn off the slow
compression and use something more efficient
- I need to be able to get to the TCP connection handle to set a
bunch of advanced options
- Perhaps most importantly on the client side we need to carefully
apply backpressure on the application so that we don't end up with
too much data in the OS net buffers. With the network sending at
200 bytes per second, it's very easy to end up with relatively
enormous buffered amounts of data and the client application quickly
gets bored. The two parts of the client proxy need to communicate
and stop accepting data from the client as soon as we have more than
a small amount buffered on the outbound side.
The question really is how best to break this up into POE
components, wheels, filters, etc Are there any gotchas to be aware
of in terms of scalability, mem, ability to do intra communication,
etc
Roughly each side looks like (say) POP server + POP Client back to
back, but then there are some outer layers to wrap that inside
another net protocol for transmission, and an outer layer on that to
handle the compression. Annoyingly the compression and net layer
needs to communicate back to the inner layers to apply some
backpressure and also to be told when there is an explicit flush
(prioritisation of packets is also a future direction). Also
because the network is so slow there needs to be quite a lot of
partial responses sent/received in order that each end doesn't get
bored waiting for the whole response
How might I best break this up in POE? What will my skeleton app
look like?
This is a couple thousand lines of perl app today and not so easy to
maintain. Hoping to improve on that
Thanks for any thoughts
Ed W
