I see two problems.
max_open => 1000, but it appears you've left max_per_host to the default of 4.
You're enqueuing 800 requests all at once. POE's message dispatch queue is
FIFO, so all the requests must occur before the first response arrives. With a
low max_per_host, it's quite likely that many of the requests are timing out.
Increase max_per_host to 1000.
Consider a more sophisticated throttling algorithm that fire off 800+ requests
all at once.
--
Rocco Caputo <rcap...@pobox.com>
On Oct 4, 2011, at 01:47, Anil Thyagarajan wrote:
> Hi,
>
> We have a data-cloud setup for caching (memory+persistent). Clients talk to
> cloud via REST API for key/value storage and retrieval
>
> I have a requirement to load the data-cloud with different RPS(request per
> sec) under different infrastructure and system profiles, to finally build a
> matrix on performance(response-time).
>
> The approach I have taken to load the system with n-rps in x-duration(secs):
>
> - For each second yield to an event that again creates a new event for every
> (time + 1/n-rps) interval within each second interval.
>
> #pool -> max_open=1000
> POE::Component::Client::HTTP->spawn(Alias => 'ua', ConnectionManager =>
> $pool);
>
>
> sub _start {
> my ($k,$h) = @_[KERNEL,HEAP];
>
> $h->{'payload'} = substr('CLOUD-TEST' x 26, 0, 256);
> $h->{'payload_size'} = 256;
>
> $k->yield('schedule_work');
> }
>
> sub schedule_work {
> my ($k,$s,$h) = @_[KERNEL,SESSION,HEAP];
>
> my $time = Time::HiRes::time();
>
> for(my $i=1;$i<=$h->{x-duration'};$i++) {
> $k->alarm_set('gen_work', $time + $i);
> }
> }
>
> sub gen_work {
> my ($k,$s,$h) = @_[KERNEL,SESSION,HEAP];
>
> my $ival = 1/$h->{'rps'};
>
> my $time = Time::HiRes::time();
>
> my $_ival = 0;
> for(my $i=0;$i<$h->{'rps'};$i++) {
> $k->alarm_set('some_work', $time + $_ival, $_ival, $time, 'PUT');
> $_ival += $ival;
> }
> }
>
> sub some_work {
> my ($k,$s,$h) = @_[KERNEL,SESSION,HEAP];
>
> my ($frac, $t_ref, $method) = @_[ARG0,ARG1,ARG2];
>
> if ($method eq 'PUT') {
>
> $h->{'id_ival'}->{$h->{'id'}} = $frac;
> my $data = gen_header_url('PUT', $h->{'id'}++, $h->{'payload_size'});
> my $req = new HTTP::Request 'PUT' => $data->{'url'};
>
> $req->header(%{$data->{'header'}});
> $req->content($h->{'payload'});
>
> $k->post("ua" => "request", "got_response", $req);
> }
> }
>
>
> The above gives perfect response time for loads of 200 RPS and 180 sec
> duration. But when the RPS is increased to 1000 with the same duration, the
> script takes a long time and 90% of the values do not have the response data
> collected. So, this is a limitation as I need to load the system with ~
> 20000RPS for 5 mins duration with a payload of at-least 100KB (key-value).
>
> Can the group suggest if this is the correct approach or I need to fine tune
> my code. I haven't see specfic examples for such need. Any suggestions will
> be helpful,
>
> Thanks,
> Anil
