Re: Architecture advice requested
It's unfair to ask for a comprehensive (avoid as many dead ends as possible) analysis of a complex system that is only described in basic terms. The best anyone can do under these circumstances is provide generic advice. Specific advice will invariably miss some hitherto unexplained detail that invalidates the analyst's assumptions and sets them back. Repeat for every mis- or insufficiently understood detail, and the frustration (and time cost) mount. One set of dead ends has been replaced with another. The best way to break an application into POE entities is not to. Decompose the application into entities and relationships that make sense for the application. Determine the requirements first, then apply POE to fulfill them. As described, your application seems to have a lot of special requirements. POE::Kernel has primitive methods that may be suitable if POE::Component and POE::Wheel classes are too high-level or too generic. Wheels and components use POE::Kernel as well, so new abstractions will work alongside existing ones. If you have the time, I recommend looking at _Design Methods for Reactive Systems_ (http://www.amazon.com/gp/product/1558607552?tag=poeperlorg-20 ). I'm about 200 pages into it, and I like it a lot so far. -- Rocco Caputo - rcap...@pobox.com On May 23, 2009, at 07:48, Ed Wildgoose wrote: Hi, I'm hoping for some tips in architecting a fairly complex network app that I would like to port to POE. It would be helpful to get some advice on how to best use POE to avoid as many dead ends as possible The basic situation is that we have a VERY limited high latency network connection (2,400 baud, 1 second latency, $1.50/min) and we want to squeeze the maximum possible down that connection. Right now I have a fairly advanced proxy application is written in vanilla perl with no special event architecture. However, for various reasons it needs a substantial update and I'm considering porting to POE to try and simplify the plumbing Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a single TCP connection across the link, optimising packet size where possible, compressing all data using one of several (known in advance) compression algorithms which can change packet by packet (knowledge of the protocol allows minimal buffer flushing) and also some light rewriting of the protocol on the fly (for efficiency) The rough algorithm/components are as follows: - Client side has an ordinary email program which talks to the proxy application - Proxy application on the client side is relatively smart and parses the protocol (POP/SMTP) in some depth and rewrites the protocol a little, eg don't waste a packet sending jsut the username, batch username and password together in a single net packet. - Server side can be less advanced because largely it's talking over a high speed network and replies are largely just streamed anyway. - Server side does a small amount of protocol rewriting, but largely it's done at a higher level, eg dealing with connection drops/restarts - Server side clearly deals with many, many more connections than the client side (order of a few hundred to a few thousand outstanding client connections at any one time). Compression buffers consume a few MB each, therefore need to be careful of mem usage and ensure a forking solution which will return memory to the OS eventually - Net protocol uses a single tcp connection and encapsulates using a simple protocol with a header stating original port, and compression protocol (this allows us to increase compression efficiency and manage tcp slow start more effectively) - Compression buffer is flushed infrequently and the protocol layer knows when we need to flush (complicates where to place the compression code) - We use a TCP CORK type emulation in the output buffer to decrease the number of packets where possible, eg received a username ok, we happen to know that the password ok/wrong will be a millisecond behind, so try and batch them. However, again the protocol layer knows when there is nothing else coming and can force a flush - We can change the compression algorithm per packet. Try to separate data and text for best compression. Also if the connection is actually discovered to be over broadband then turn off the slow compression and use something more efficient - I need to be able to get to the TCP connection handle to set a bunch of advanced options - Perhaps most importantly on the client side we need to carefully apply backpressure on the application so that we don't end up with too much data in the OS net buffers. With the network sending at 200 bytes per second, it's very easy to end up with relatively enormous buffered amounts of data and the client application quickly gets bored. The two parts of the client proxy need to communicate and stop accepting data from the
Component::Server::TCP: how to get access to the server's HEAP from a client handler?
Hi,
I'm using POE for 3 months now with great pleasure. Thanks to all the
developers!
I'm now playing with TCP client and server.
I've created a POE::Component::Server::TCP which has global state stored in
its HEAP, initialized in the Started event.
How can I get access to that HEAP from the Client* events (ClientConnected
in particular) as those are called from a different session ?
Maybe there is a parent-child relationship between the sessions I could use
to retrieve the server session and call it's -get_heap() ?.
Below is some sample code that I'm sure you'll help me to complete.
Olivier
dolmen on cpan.org
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests = 8;
use POE qw(Component::Server::TCP Component::Client::TCP);
my $host = 'localhost';
my $port = 12344;
POE::Component::Server::TCP-new(
Alias = server,
Address = $host,
Port = $port,
Started = sub {
pass [Server] Started #.$_[SESSION]-ID;
$_[HEAP]{test} = 42;
},
ClientConnected = sub {
pass [Server] ClientConnected #.$_[SESSION]-ID;
# How to get access to {test} from the server session's HEAP ?
#is(???, 42, Server HEAP access);
},
ClientInput = sub {
pass [Server] ClientInput #.$_[SESSION]-ID;
diag($_[ARG0]);
},
ClientDisconnected = sub {
pass [Server] ClientDisconnected #.$_[SESSION]-ID;
$poe_kernel-post('server', 'shutdown');
},
);
POE::Component::Client::TCP-new(
Alias = client,
RemoteAddress = $host,
RemotePort= $port,
Started = sub {
pass [Client] Started;
},
Connected = sub {
pass [Client] Connected;
$_[HEAP]{server}-put(Hello);
$poe_kernel-yield('shutdown');
},
ServerInput = sub { },
Disconnected = sub {
pass [Client] Disconnected;
},
);
$poe_kernel-run;
pass 'Stopped';
Re: Component::Server::TCP: how to get access to the server's HEAP from a client handler?
Le 26 mai 2009 16:46, Mark Morgan makk...@gmail.com a écrit :
From the P:C:S:TCP docs, it looks like you should be able to do this
by passing a reference to it via ClientArgs constructor argument.
You may need to end up weakening it in client, to allow the server
sessions and anything else the client heap is referring to to be
reaped.
Mark.
I have found a solution using ClientArgs.
However it is currently broken, at least it doesn't work as the
documentation says.
See my code below. I've shown with JSON what I get in ClientConnected args :
- socket is missing
- ARG0 is the given ClientArgs. It would me more useful to get Args
flattened in ARG0..$#_
The documentation for this code has still many TODOs and no one is
complaining on RT so probably not many use it.
http://search.cpan.org/~rcaputo/POE-1.005/lib/POE/Component/Server/TCP.pm#Default_Child_Connection_Sessionshttp://search.cpan.org/%7Ercaputo/POE-1.005/lib/POE/Component/Server/TCP.pm#Default_Child_Connection_Sessions
Olivier
http://search.cpan.org/~dolmen/ http://search.cpan.org/%7Edolmen/
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests = 8;
use JSON 'to_json';
use POE qw(Component::Server::TCP Component::Client::TCP);
my $host = 'localhost';
my $port = 12344;
POE::Component::Server::TCP-new(
Alias = server,
Address = $host,
Port = $port,
ClientArgs= [ {} ],
Started = sub {
pass [Server] Started #.$_[SESSION]-ID;
my $server_data = $_[HEAP]-{server_data} = $_[ARG0]; # The hash given
in Args
$server_data-{test} = 42;
},
ClientConnected = sub {
# According to the documentation ARG0 is the socket, ARG1 is ClientArgs
# But the doc doesn't match the code :(
# We get ClientArgs in ARG0, but it would be more consistent to get it
flattened
diag(to_json([ @_[ARG0..$#_] ]));
my $server_data = $_[HEAP]-{server_data} = $_[ARG0]-[0]; # The hash
given in Args
pass [Server] ClientConnected #.$_[SESSION]-ID;
is($server_data-{test}, 42, Server HEAP access);
},
ClientInput = sub {
pass [Server] ClientInput #.$_[SESSION]-ID;
diag($_[ARG0]);
my $server_data = $_[HEAP]-{server_data};
is($server_data-{test}, 42, Server HEAP access);
},
ClientDisconnected = sub {
pass [Server] ClientDisconnected #.$_[SESSION]-ID;
$poe_kernel-post('server', 'shutdown');
},
);
POE::Component::Client::TCP-new(
Alias = client,
RemoteAddress = $host,
RemotePort= $port,
Started = sub {
pass [Client] Started;
},
Connected = sub {
pass [Client] Connected;
$_[HEAP]{server}-put(Hello);
$poe_kernel-yield('shutdown');
},
ServerInput = sub { },
Disconnected = sub {
pass [Client] Disconnected;
},
);
$poe_kernel-run;
pass 'Stopped';
Re: Component::Server::TCP: how to get access to the server's HEAP from a client handler?
From the P:C:S:TCP docs, it looks like you should be able to do this
by passing a reference to it via ClientArgs constructor argument.
You may need to end up weakening it in client, to allow the server
sessions and anything else the client heap is referring to to be
reaped.
Mark.
2009/5/26 Olivier Mengué olivier.men...@gmail.com:
Hi,
I'm using POE for 3 months now with great pleasure. Thanks to all the
developers!
I'm now playing with TCP client and server.
I've created a POE::Component::Server::TCP which has global state stored in
its HEAP, initialized in the Started event.
How can I get access to that HEAP from the Client* events (ClientConnected
in particular) as those are called from a different session ?
Maybe there is a parent-child relationship between the sessions I could use
to retrieve the server session and call it's -get_heap() ?.
Below is some sample code that I'm sure you'll help me to complete.
Olivier
dolmen on cpan.org
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests = 8;
use POE qw(Component::Server::TCP Component::Client::TCP);
my $host = 'localhost';
my $port = 12344;
POE::Component::Server::TCP-new(
Alias = server,
Address = $host,
Port = $port,
Started = sub {
pass [Server] Started #.$_[SESSION]-ID;
$_[HEAP]{test} = 42;
},
ClientConnected = sub {
pass [Server] ClientConnected #.$_[SESSION]-ID;
# How to get access to {test} from the server session's HEAP ?
#is(???, 42, Server HEAP access);
},
ClientInput = sub {
pass [Server] ClientInput #.$_[SESSION]-ID;
diag($_[ARG0]);
},
ClientDisconnected = sub {
pass [Server] ClientDisconnected #.$_[SESSION]-ID;
$poe_kernel-post('server', 'shutdown');
},
);
POE::Component::Client::TCP-new(
Alias = client,
RemoteAddress = $host,
RemotePort = $port,
Started = sub {
pass [Client] Started;
},
Connected = sub {
pass [Client] Connected;
$_[HEAP]{server}-put(Hello);
$poe_kernel-yield('shutdown');
},
ServerInput = sub { },
Disconnected = sub {
pass [Client] Disconnected;
},
);
$poe_kernel-run;
pass 'Stopped';
