Re: Naming advice for retry manager module
Thanks for the tip... we'll consider it. I would never have considered "backoff" as a keyword for this. On Tue, Apr 21, 2009 at 6:08 PM, Jonathan Yu wrote: > Hi: > > What about http://search.cpan.org/~dlo/Proc-BackOff-0.02/lib/Proc/BackOff.pm > > Proc::BackOff. It seems to implement a function similar to TCP packet > retry backoff... > > The idea is that for every failure you wait X time before the next > request; the next time, you wait 2X. etc. But there is also an > exponential backoff one. > > Hope this helps. I haven't read the module description thoroughly but > it deals with doing retries in a way that doesn't totally hammer a > system and bring it to its knees. > > Cheers, > > Jonathan > > On Tue, Apr 21, 2009 at 4:11 PM, Bill Ward wrote: >> I am planning to write a new module that would manage retries. Let's say >> you want to talk to some network service that might have errors or be >> offline, and if you get certain kinds of errors (e.g. the host is being >> rebooted, so it's not responding, but will shortly) you want to try again >> after some set interval. But you don't want to retry forever - eventually >> it should be a hard error. >> >> We already have this kind of logic embedded in one place but I want to write >> a generic object that would basically hold the retry parameters >> (RETRY_COUNT, RETRY_DIE, RETRY_SLEEP, RETRY_SLEEP_MULTIPLIER, >> RETRY_SLEEP_LIMIT) and respond to queries like: >> >> - Something failed - should I retry or die? (if number of retries so far is >> less than RETRY_COUNT) >> - How long should I sleep for / wake me up when the sleep time has passed >> - etc. >> >> I haven't seen anything on CPAN that does this - a quick search for "retry" >> on CPAN yields tons of results but they all appear to be very >> domain-specific or just a mention in the documentation of some particular >> module. >> >> Something like Object::Retry maybe? Then things can inherit from it? >> > -- Check out my LEGO blog at http://www.brickpile.com/ View my photos at http://flickr.com/photos/billward/ Follow me at http://twitter.com/williamward
Re: On 'unpack()' - How much did I eat?
Quoth r...@ringlet.net (Peter Pentchev): > On Tue, Apr 21, 2009 at 10:24:10AM +0100, Paul LeoNerd Evans wrote: > > I find lately I've been doing a lot of binary protocol work, taking > > messages that live in TCP streams or files or similar, and doing lots of > > pack()/unpack() on them. > [snip] > > > > Is there some neater way to do this? Can I either: > > > > a: Get unpack() to consume bytes from the LVALUE > > > > b: Ask unpack() how much it ate in a generic fashion? > > Brief answer: > - it's possible by patching the Perl source, see the last paragraph > for an explanation about a possible patch; > - it could be done as an external module that must be kept in sync > with Perl's pp_pack.c. What am I missing here? It appears to me unpacking "." gives the current byte position in the string, which is what is needed. ~% perl -E'my $bin = pack "NN", 10, 12; say for unpack ".N.N.", $bin' 0 10 4 12 8 Ben
Re: On 'unpack()' - How much did I eat?
If a Perl patch can be made available for this purpose, then why not an XS/C module? On Tue, Apr 21, 2009 at 8:17 PM, Peter Pentchev wrote: > On Tue, Apr 21, 2009 at 10:24:10AM +0100, Paul LeoNerd Evans wrote: >> I find lately I've been doing a lot of binary protocol work, taking >> messages that live in TCP streams or files or similar, and doing lots of >> pack()/unpack() on them. > [snip] >> >> Is there some neater way to do this? Can I either: >> >> a: Get unpack() to consume bytes from the LVALUE >> >> b: Ask unpack() how much it ate in a generic fashion? > > Brief answer: > - it's possible by patching the Perl source, see the last paragraph > for an explanation about a possible patch; > - it could be done as an external module that must be kept in sync > with Perl's pp_pack.c. > > From a quick look at pp_pack.c in both Perl 5.8.9 and Perl 5.10.0, > both options are possible in theory, but both of them require > modifying the Perl source in some way :( > > The least intrusive way IMHO would be to un-staticize the unpack_rec() > routine, so either another part of core Perl or a module that > "promises" to stay in sync with pp_pack's unpackstr() routine actually > *can* make use of unpack_rec's last argument. > > Of course, it is still possible for an external module to duplicate > the whole of pp_pack.c and take great pains to stay in sync with core, > but I'm not sure anyone would actually *want* to do that :> Although, > actually, I hereby volunteer to do it - to try to make a separate Perl > module out of a copy of 5.10.0's pp_pack.c, and then follow Perl's > development and update it as needed - that is, if there's anyone who > actually thinks this would be a good idea and if there's no-one who > thinks it would be a very bad idea. I can see why it could be a bad > idea, but here's a call for opinions / votes / whatever :) If people > want it, I can try. > > An easier (from a programmer's point of view) and nightmarish > (from a module writer's point of view) solution would be a patch to > Perl that adds another function (say, lunpack()) to pp_pack.c, that > does pretty much the same as the current unpack(), but also return > (or store somewhere) the number of bytes consumed. It would only > be useful if it is actually accepted into the core Perl and makes it > into a release that you can "require". I think I could write a patch > like that tomorrow after I've actually had some sleep :) > > G'luck, > Peter > > -- > Peter Pentchev r...@ringlet.net r...@space.bg r...@freebsd.org > PGP key: http://people.FreeBSD.org/~roam/roam.key.asc > Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 > I am the meaning of this sentence. >
Re: Naming advice for retry manager module
Hi: What about http://search.cpan.org/~dlo/Proc-BackOff-0.02/lib/Proc/BackOff.pm Proc::BackOff. It seems to implement a function similar to TCP packet retry backoff... The idea is that for every failure you wait X time before the next request; the next time, you wait 2X. etc. But there is also an exponential backoff one. Hope this helps. I haven't read the module description thoroughly but it deals with doing retries in a way that doesn't totally hammer a system and bring it to its knees. Cheers, Jonathan On Tue, Apr 21, 2009 at 4:11 PM, Bill Ward wrote: > I am planning to write a new module that would manage retries. Let's say > you want to talk to some network service that might have errors or be > offline, and if you get certain kinds of errors (e.g. the host is being > rebooted, so it's not responding, but will shortly) you want to try again > after some set interval. But you don't want to retry forever - eventually > it should be a hard error. > > We already have this kind of logic embedded in one place but I want to write > a generic object that would basically hold the retry parameters > (RETRY_COUNT, RETRY_DIE, RETRY_SLEEP, RETRY_SLEEP_MULTIPLIER, > RETRY_SLEEP_LIMIT) and respond to queries like: > > - Something failed - should I retry or die? (if number of retries so far is > less than RETRY_COUNT) > - How long should I sleep for / wake me up when the sleep time has passed > - etc. > > I haven't seen anything on CPAN that does this - a quick search for "retry" > on CPAN yields tons of results but they all appear to be very > domain-specific or just a mention in the documentation of some particular > module. > > Something like Object::Retry maybe? Then things can inherit from it? >
Re: On 'unpack()' - How much did I eat?
On Tue, Apr 21, 2009 at 10:24:10AM +0100, Paul LeoNerd Evans wrote: > I find lately I've been doing a lot of binary protocol work, taking > messages that live in TCP streams or files or similar, and doing lots of > pack()/unpack() on them. [snip] > > Is there some neater way to do this? Can I either: > > a: Get unpack() to consume bytes from the LVALUE > > b: Ask unpack() how much it ate in a generic fashion? Brief answer: - it's possible by patching the Perl source, see the last paragraph for an explanation about a possible patch; - it could be done as an external module that must be kept in sync with Perl's pp_pack.c. From a quick look at pp_pack.c in both Perl 5.8.9 and Perl 5.10.0, both options are possible in theory, but both of them require modifying the Perl source in some way :( The least intrusive way IMHO would be to un-staticize the unpack_rec() routine, so either another part of core Perl or a module that "promises" to stay in sync with pp_pack's unpackstr() routine actually *can* make use of unpack_rec's last argument. Of course, it is still possible for an external module to duplicate the whole of pp_pack.c and take great pains to stay in sync with core, but I'm not sure anyone would actually *want* to do that :> Although, actually, I hereby volunteer to do it - to try to make a separate Perl module out of a copy of 5.10.0's pp_pack.c, and then follow Perl's development and update it as needed - that is, if there's anyone who actually thinks this would be a good idea and if there's no-one who thinks it would be a very bad idea. I can see why it could be a bad idea, but here's a call for opinions / votes / whatever :) If people want it, I can try. An easier (from a programmer's point of view) and nightmarish (from a module writer's point of view) solution would be a patch to Perl that adds another function (say, lunpack()) to pp_pack.c, that does pretty much the same as the current unpack(), but also return (or store somewhere) the number of bytes consumed. It would only be useful if it is actually accepted into the core Perl and makes it into a release that you can "require". I think I could write a patch like that tomorrow after I've actually had some sleep :) G'luck, Peter -- Peter Pentchev r...@ringlet.netr...@space.bgr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 I am the meaning of this sentence. pgpvUENQhLWoq.pgp Description: PGP signature
Re: Naming advice for retry manager module
On Tue, Apr 21, 2009 at 3:12 PM, David Nicol wrote:
> On Tue, Apr 21, 2009 at 3:11 PM, Bill Ward wrote:
>> Something like Object::Retry maybe? Then things can inherit from it?
>
> The proposed module sounds more like a has-a than an is-a. Or maybe
> just a new method that would get included in the caller's namespace.
> Set the defaults at import time, then call it with a single coderef:
>
>
> use retry MIN_SLEEP => 3, MAX_SLEEP => 10, TIMEOUT => 120
> my $asdf_server_connection;
> retry sub { $asdf_server_connection = connect ( (new socket), ...)
> or die } # whatever
>
>
> the retry::retry routine would be something like
>
> sub retry::retry($) {
> my $start_time = time;
> my $result;
> for(;;){
> eval { $result = &$_[0]; 1 } and return $result;
> my $E = $@;
> $LOGGER and $LOGGER->( "$E invoked from @{[caller]}");
> time - $start_time > $TIMEOUT and die "$E\n";
> sleep ($MIN_SLEEP + rand ( $MAX_SLEEP - $MIN_SLEEP ))
> }
> }
>
> Everything in all-caps would be retry:: package variables, or scalars
> tied to a per-caller(1)[0] hash to mitigate AAAD surprises. Or
> incorporated into the particular Retry::Object, as you are wisely
> suggesting.
>
> I've written stuff like that often enough that it would be cool to be
> able to have a standard name and API for it, too.
Well, there could be multiple things that have retry functionality, so
I would want to use object attributes rather than globals ... but
you're right, it's more has-a than is-a.
Re: Naming advice for retry manager module
On Tue, Apr 21, 2009 at 3:11 PM, Bill Ward wrote:
> Something like Object::Retry maybe? Then things can inherit from it?
The proposed module sounds more like a has-a than an is-a. Or maybe
just a new method that would get included in the caller's namespace.
Set the defaults at import time, then call it with a single coderef:
use retry MIN_SLEEP => 3, MAX_SLEEP => 10, TIMEOUT => 120
my $asdf_server_connection;
retry sub { $asdf_server_connection = connect ( (new socket), ...)
or die } # whatever
the retry::retry routine would be something like
sub retry::retry($) {
my $start_time = time;
my $result;
for(;;){
eval { $result = &$_[0]; 1 } and return $result;
my $E = $@;
$LOGGER and $LOGGER->( "$E invoked from @{[caller]}");
time - $start_time > $TIMEOUT and die "$E\n";
sleep ($MIN_SLEEP + rand ( $MAX_SLEEP - $MIN_SLEEP ))
}
}
Everything in all-caps would be retry:: package variables, or scalars
tied to a per-caller(1)[0] hash to mitigate AAAD surprises. Or
incorporated into the particular Retry::Object, as you are wisely
suggesting.
I've written stuff like that often enough that it would be cool to be
able to have a standard name and API for it, too.
--
"Your appreciation is my glorious gratuity." -- Herman Melville on IP rights
Naming advice for retry manager module
I am planning to write a new module that would manage retries. Let's say you want to talk to some network service that might have errors or be offline, and if you get certain kinds of errors (e.g. the host is being rebooted, so it's not responding, but will shortly) you want to try again after some set interval. But you don't want to retry forever - eventually it should be a hard error. We already have this kind of logic embedded in one place but I want to write a generic object that would basically hold the retry parameters (RETRY_COUNT, RETRY_DIE, RETRY_SLEEP, RETRY_SLEEP_MULTIPLIER, RETRY_SLEEP_LIMIT) and respond to queries like: - Something failed - should I retry or die? (if number of retries so far is less than RETRY_COUNT) - How long should I sleep for / wake me up when the sleep time has passed - etc. I haven't seen anything on CPAN that does this - a quick search for "retry" on CPAN yields tons of results but they all appear to be very domain-specific or just a mention in the documentation of some particular module. Something like Object::Retry maybe? Then things can inherit from it?
Re: On 'unpack()' - How much did I eat?
Hi Paul. Paul LeoNerd Evans wrote: I find lately I've been doing a lot of binary protocol work, taking messages that live in TCP streams or files or similar, and doing lots of pack()/unpack() on them. this strikes me as more than a little inelegant. That 4 floating about there is a "magic" constant. Things like this get worse ( not every day I get to brag about my module, so please be patient... :-) ) How about my module, Data::ParseBinary? it gives you a lot of options. If you are reading from a string, then you can use a string reader stream, and it will remember the current location, so the next parse will start from it. If you have a "live" string, that you want to add binary data to the end and delete from the beginning, you can use StringRef stream reader, adding add new data to the end. For deleting data from the beginning, you can ask where it is now ($stream->tell), substr from the beginning, and then $stream->seek(0). Or, if you are reading from the network, maybe you should just write a custom network stream, and parse directly off it. And finally, if you are reading from file, there is a File stream reader, that does it for you. Warning: if you are looking for performance, this module is the wrong one. Shmuel.
Google Announces Nine Students in GSoC2009 with The Perl Foundation
Howdy, I have the extreme pleasure to announce that the Google Summer of Code 2009 has officially started and The Perl Foundation will be mentoring 9 students this year in a variety of projects. A breakdown of each student project and mentor with links to the project abstract can be found at [1]. If you would like to keep up with recent updates, then subscribe to this RSS feed [2]. If you would like to get a little more involved, come join us in #soc-help on irc.perl.org or join the tpf-gsoc-students list [3]. [1] http://leto.net/dukeleto.pl/2009/04/google-announces-nine-students-in-gsoc2009-with-the-perl-fou.html [2] http://leto.net/dukeleto.pl/atom.xml [3] http://groups.google.com/group/tpf-gsoc-students Thanks to everyone involved, including students with projects that were not accepted. We had a limited number of spots and some very good applications could not be accepted. With a bit more spit and polish some would be a great fit for a TPF grant. Thank you to *everyone* who applied, and if you did not get accepted this year, you can still implement your project and become part of the community, without getting paid. I promise, we don't bite. Stay tuned for further updates. Cheers, -- Jonathan Leto jonat...@leto.net http://leto.net
Re: On 'unpack()' - How much did I eat?
On Tue, Apr 21, 2009 at 10:24:10AM +0100, Paul LeoNerd Evans wrote:
> sub pull_int
> {
> my $self = shift;
> my $i = unpack( "N", $self->{buffer} );
> substr( $i, 0, 4 ) = "";
> return $i;
> }
Uh.. obviously, that's meant to be
substr( $self->{buffer}, 0, 4 ) = "";
/me unpacks more coffee into self...
--
Paul "LeoNerd" Evans
leon...@leonerd.org.uk |CPAN ID: PEVANS
srand($,=" ");print sort{rand>0.5}grep{0.8>rand}qw(another Just hacker of Perl)
signature.asc
Description: Digital signature
On 'unpack()' - How much did I eat?
I find lately I've been doing a lot of binary protocol work, taking
messages that live in TCP streams or files or similar, and doing lots of
pack()/unpack() on them.
I find what works best is to wrap up a message into an object, so I can
do things like:
my $field = $message->pull_int();
where
sub pull_int
{
my $self = shift;
my $i = unpack( "N", $self->{buffer} );
substr( $i, 0, 4 ) = "";
return $i;
}
this strikes me as more than a little inelegant. That 4 floating about
there is a "magic" constant. Things like this get worse
sub pull_strz
{
my $self = shift;
my $s = unpack( "Z*", $self->{buffer} );
substr( $i, 0, len($s) + 1 ) = "";
return $s;
}
Is there some neater way to do this? Can I either:
a: Get unpack() to consume bytes from the LVALUE
b: Ask unpack() how much it ate in a generic fashion?
One suggestion I've seen would be to
sub unpack_and_eat
{
my ( $format, $str ) = @_;
my @values = unpack( $format . " A*", $str );
$_[1] = pop @values;
return @values;
}
but that seems a bit messy, all that O(n^2) copying about the place if
we're going to, say, pull lots of small ints out of a really big buffer.
There surely has to be a better way...
--
Paul "LeoNerd" Evans
leon...@leonerd.org.uk |CPAN ID: PEVANS
srand($,=" ");print sort{rand>0.5}grep{0.8>rand}qw(another Just hacker of Perl)
signature.asc
Description: Digital signature
