Re: Architecture advice requested
On May 29, 2009, at 04:48, Ed W wrote: Rocco Caputo wrote: The best way to break an application into POE entities is not to. Decompose the application into entities and relationships that make sense for the application. Determine the requirements first, then apply POE to fulfill them. OK, to be specific, the heart of it is a proxy server, which seems to be fulfilled by using protocol X client wheel talking to protocol X server wheel? The bits I guess I am most interested in are: - How I might architect the POP server client to be one step removed from the socket and have the following pipeline attached to the socket: data_in - decompress - parse input data into separate protocols and feed to correct wheel. Is this another wheel, or done as a filter? (or something else?) I might implement this in application code until I knew the requirements and constraints on the data_in - decompress - parse flow. As those emerged (or were specified), I might refactor the generic transformative bits into their own modules. The most significant trade-off seems to be whether reusability or throughput is more important. If reusability: I would refactor the transformative code into POE::Filter, POE::Driver and/or POE::Wheel namespaces. If throughput, I would evaluate whether custom object APIs fit the application better. - Same for the output. ie taking the output of multiple wheels, compressing the output and then blocking it into a simple protocol to multiplex over a single TCP connection (just data packet with a header on it) Funneling would be done by a component (aka a module that uses POE). There seems to be a lot of decision logic and the possibility that the funnel will respond to individual clients. Something higher level must be responsible for the side effects since POE::Filter and POE::Wheel::ReadWrite are strictly transformative. - The main reason for looking at POE is that while it's mostly a pipeline, occasionally one distinct bit of the app needs to talk to another distinct bit, eg some remote end hangs up suddenly, the client end may not be expecting this as part of it's normal protocol and we need to wake it up and force it to terminate the other side of the proxy. Any special features of POE which would be especially helpful here? It sounds like you want the client and server sides of the proxy to be loosely coupled. For instance, the remote end may hang up suddenly at the end of a message, but the local end may still be receiving. You might want to wait for the local end to finish before shutting down the connection. POE can help accomplish this by making it easier to decouple the client and server sides of the proxy. The proxy behavior is then goverend by the message protocol you define between the client and server, rather than by the particular protocols on either end. Recently I've seen POE used to proxy between stateless HTTP and a stateful game. The idea seems similar. It allows an HTTP client to come and go, but it keeps the game connected. It adapts the two by tracking the game's state in the proxy, then updating the user when they refresh the page. POE doesn't have a component for mapping client and server sides of proxies. If you write one, I hope you can release it. -- Rocco Caputo - rcap...@pobox.com
Re: Architecture advice requested
I'm sorry that I missed the point completely. I'll avoid commenting in the future. good luck with the project. On Mon, May 25, 2009 at 2:44 AM, Ed W li...@wildgooses.com wrote: chris fedde wrote: My second reaction is to wonder if you can do better in perl than a demand dial PPP link would provide. ? You need both. At $1.50/min you want this dialup link only up when you need it Right now since most users are on windows we just show them the normal windows dialup feature and they dial when they need to do stuf My third reaction is to wonder if it is not better to use something like UUCP and batch load across the link. Hardly. Doing the algorithm described you are pretty close to a solid block of data moving in both directions simultaneously, including compression and management of tcp packet size. So the efficiency approaches that of UUCP + intensive compression, but both ends talk a standard protocol Interactive protocols like POP, IMAP, and HTTP are going to be very hard to use over such a slow high latency link. ?? That's why this project exists in the first place? Just to be clear: this exists already, I'm doing a structural enhancement. Current code is really only a few hundred lines at it's core, runs to a couple of thousand once you handle all the protocol parsing, gui and other special cases Download a trial from http://www.mailasail.com - the client is packed as an EXE, but I doubt that will stop an enterprising soul who wants to see how it works... If your goal is just email then I'd recommend setting up a local MTA and configure it to use ETRN to collect mail queued at an MTA on an internet connected host. OK, so that's only about 5x slower than my proposed solution. No compression and large latency due to each SMTP command being sent one by one. Don't forget that tcp slow start will also hurt you. I think my suggestion is much better... You'll also want very good spam filtering on the connected side to avoid paying for too much spam. Sure - done. Also we do a huge amount more like killing redundant email headers, eliminating extra HTML sections and optionally blackberry like attachment mangling into text. What I am really looking for are suggestions on how to structure a small proxy application which will really stretch the POE architecture because the outer layers are quite integrated with the inner layers The inner side of the app is presumably just going to be several wheels proxying to each other, eg pop server is the in side, talking to a pop client wheel which then goes back out to the network and the real server. However, given that I want to compress this output and multiplex it, is it best to call that an output filter? How might I signal to the output filter that I want the compression buffered at this point in the stream? How might I indicate that this stream should get priority over this other stream?) What about the receiving end which gets a compressed stream packing several protocols. There will need to be a bit of logic to handle corruption, invalid input, etc. Then it needs to unpack this into several protocol streams and feed them to several protocol parsers. Does that sound like an input filter or is it a wheel because there might be some state information (several packets to join, etc) Any thoughts? Any demo apps which show parts of this functionality? Thanks Ed W
Re: Architecture advice requested
It's unfair to ask for a comprehensive (avoid as many dead ends as possible) analysis of a complex system that is only described in basic terms. The best anyone can do under these circumstances is provide generic advice. Specific advice will invariably miss some hitherto unexplained detail that invalidates the analyst's assumptions and sets them back. Repeat for every mis- or insufficiently understood detail, and the frustration (and time cost) mount. One set of dead ends has been replaced with another. The best way to break an application into POE entities is not to. Decompose the application into entities and relationships that make sense for the application. Determine the requirements first, then apply POE to fulfill them. As described, your application seems to have a lot of special requirements. POE::Kernel has primitive methods that may be suitable if POE::Component and POE::Wheel classes are too high-level or too generic. Wheels and components use POE::Kernel as well, so new abstractions will work alongside existing ones. If you have the time, I recommend looking at _Design Methods for Reactive Systems_ (http://www.amazon.com/gp/product/1558607552?tag=poeperlorg-20 ). I'm about 200 pages into it, and I like it a lot so far. -- Rocco Caputo - rcap...@pobox.com On May 23, 2009, at 07:48, Ed Wildgoose wrote: Hi, I'm hoping for some tips in architecting a fairly complex network app that I would like to port to POE. It would be helpful to get some advice on how to best use POE to avoid as many dead ends as possible The basic situation is that we have a VERY limited high latency network connection (2,400 baud, 1 second latency, $1.50/min) and we want to squeeze the maximum possible down that connection. Right now I have a fairly advanced proxy application is written in vanilla perl with no special event architecture. However, for various reasons it needs a substantial update and I'm considering porting to POE to try and simplify the plumbing Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a single TCP connection across the link, optimising packet size where possible, compressing all data using one of several (known in advance) compression algorithms which can change packet by packet (knowledge of the protocol allows minimal buffer flushing) and also some light rewriting of the protocol on the fly (for efficiency) The rough algorithm/components are as follows: - Client side has an ordinary email program which talks to the proxy application - Proxy application on the client side is relatively smart and parses the protocol (POP/SMTP) in some depth and rewrites the protocol a little, eg don't waste a packet sending jsut the username, batch username and password together in a single net packet. - Server side can be less advanced because largely it's talking over a high speed network and replies are largely just streamed anyway. - Server side does a small amount of protocol rewriting, but largely it's done at a higher level, eg dealing with connection drops/restarts - Server side clearly deals with many, many more connections than the client side (order of a few hundred to a few thousand outstanding client connections at any one time). Compression buffers consume a few MB each, therefore need to be careful of mem usage and ensure a forking solution which will return memory to the OS eventually - Net protocol uses a single tcp connection and encapsulates using a simple protocol with a header stating original port, and compression protocol (this allows us to increase compression efficiency and manage tcp slow start more effectively) - Compression buffer is flushed infrequently and the protocol layer knows when we need to flush (complicates where to place the compression code) - We use a TCP CORK type emulation in the output buffer to decrease the number of packets where possible, eg received a username ok, we happen to know that the password ok/wrong will be a millisecond behind, so try and batch them. However, again the protocol layer knows when there is nothing else coming and can force a flush - We can change the compression algorithm per packet. Try to separate data and text for best compression. Also if the connection is actually discovered to be over broadband then turn off the slow compression and use something more efficient - I need to be able to get to the TCP connection handle to set a bunch of advanced options - Perhaps most importantly on the client side we need to carefully apply backpressure on the application so that we don't end up with too much data in the OS net buffers. With the network sending at 200 bytes per second, it's very easy to end up with relatively enormous buffered amounts of data and the client application quickly gets bored. The two parts of the client proxy need to communicate and stop accepting data from the
Re: Architecture advice requested
chris fedde wrote: My second reaction is to wonder if you can do better in perl than a demand dial PPP link would provide. ? You need both. At $1.50/min you want this dialup link only up when you need it Right now since most users are on windows we just show them the normal windows dialup feature and they dial when they need to do stuf My third reaction is to wonder if it is not better to use something like UUCP and batch load across the link. Hardly. Doing the algorithm described you are pretty close to a solid block of data moving in both directions simultaneously, including compression and management of tcp packet size. So the efficiency approaches that of UUCP + intensive compression, but both ends talk a standard protocol Interactive protocols like POP, IMAP, and HTTP are going to be very hard to use over such a slow high latency link. ?? That's why this project exists in the first place? Just to be clear: this exists already, I'm doing a structural enhancement. Current code is really only a few hundred lines at it's core, runs to a couple of thousand once you handle all the protocol parsing, gui and other special cases Download a trial from http://www.mailasail.com - the client is packed as an EXE, but I doubt that will stop an enterprising soul who wants to see how it works... If your goal is just email then I'd recommend setting up a local MTA and configure it to use ETRN to collect mail queued at an MTA on an internet connected host. OK, so that's only about 5x slower than my proposed solution. No compression and large latency due to each SMTP command being sent one by one. Don't forget that tcp slow start will also hurt you. I think my suggestion is much better... You'll also want very good spam filtering on the connected side to avoid paying for too much spam. Sure - done. Also we do a huge amount more like killing redundant email headers, eliminating extra HTML sections and optionally blackberry like attachment mangling into text. What I am really looking for are suggestions on how to structure a small proxy application which will really stretch the POE architecture because the outer layers are quite integrated with the inner layers The inner side of the app is presumably just going to be several wheels proxying to each other, eg pop server is the in side, talking to a pop client wheel which then goes back out to the network and the real server. However, given that I want to compress this output and multiplex it, is it best to call that an output filter? How might I signal to the output filter that I want the compression buffered at this point in the stream? How might I indicate that this stream should get priority over this other stream?) What about the receiving end which gets a compressed stream packing several protocols. There will need to be a bit of logic to handle corruption, invalid input, etc. Then it needs to unpack this into several protocol streams and feed them to several protocol parsers. Does that sound like an input filter or is it a wheel because there might be some state information (several packets to join, etc) Any thoughts? Any demo apps which show parts of this functionality? Thanks Ed W
Architecture advice requested
Hi, I'm hoping for some tips in architecting a fairly complex network app that I would like to port to POE. It would be helpful to get some advice on how to best use POE to avoid as many dead ends as possible The basic situation is that we have a VERY limited high latency network connection (2,400 baud, 1 second latency, $1.50/min) and we want to squeeze the maximum possible down that connection. Right now I have a fairly advanced proxy application is written in vanilla perl with no special event architecture. However, for various reasons it needs a substantial update and I'm considering porting to POE to try and simplify the plumbing Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a single TCP connection across the link, optimising packet size where possible, compressing all data using one of several (known in advance) compression algorithms which can change packet by packet (knowledge of the protocol allows minimal buffer flushing) and also some light rewriting of the protocol on the fly (for efficiency) The rough algorithm/components are as follows: - Client side has an ordinary email program which talks to the proxy application - Proxy application on the client side is relatively smart and parses the protocol (POP/SMTP) in some depth and rewrites the protocol a little, eg don't waste a packet sending jsut the username, batch username and password together in a single net packet. - Server side can be less advanced because largely it's talking over a high speed network and replies are largely just streamed anyway. - Server side does a small amount of protocol rewriting, but largely it's done at a higher level, eg dealing with connection drops/restarts - Server side clearly deals with many, many more connections than the client side (order of a few hundred to a few thousand outstanding client connections at any one time). Compression buffers consume a few MB each, therefore need to be careful of mem usage and ensure a forking solution which will return memory to the OS eventually - Net protocol uses a single tcp connection and encapsulates using a simple protocol with a header stating original port, and compression protocol (this allows us to increase compression efficiency and manage tcp slow start more effectively) - Compression buffer is flushed infrequently and the protocol layer knows when we need to flush (complicates where to place the compression code) - We use a TCP CORK type emulation in the output buffer to decrease the number of packets where possible, eg received a username ok, we happen to know that the password ok/wrong will be a millisecond behind, so try and batch them. However, again the protocol layer knows when there is nothing else coming and can force a flush - We can change the compression algorithm per packet. Try to separate data and text for best compression. Also if the connection is actually discovered to be over broadband then turn off the slow compression and use something more efficient - I need to be able to get to the TCP connection handle to set a bunch of advanced options - Perhaps most importantly on the client side we need to carefully apply backpressure on the application so that we don't end up with too much data in the OS net buffers. With the network sending at 200 bytes per second, it's very easy to end up with relatively enormous buffered amounts of data and the client application quickly gets bored. The two parts of the client proxy need to communicate and stop accepting data from the client as soon as we have more than a small amount buffered on the outbound side. The question really is how best to break this up into POE components, wheels, filters, etc Are there any gotchas to be aware of in terms of scalability, mem, ability to do intra communication, etc Roughly each side looks like (say) POP server + POP Client back to back, but then there are some outer layers to wrap that inside another net protocol for transmission, and an outer layer on that to handle the compression. Annoyingly the compression and net layer needs to communicate back to the inner layers to apply some backpressure and also to be told when there is an explicit flush (prioritisation of packets is also a future direction). Also because the network is so slow there needs to be quite a lot of partial responses sent/received in order that each end doesn't get bored waiting for the whole response How might I best break this up in POE? What will my skeleton app look like? This is a couple thousand lines of perl app today and not so easy to maintain. Hoping to improve on that Thanks for any thoughts Ed W
