Re: Storable Stream - Module namespace help
On Sun, Aug 8, 2010 at 4:58 AM, Marco Neves [ModAuthors] perl-module-auth...@knowhunter.cjb.net wrote: Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I'm confused -- a stream can't be stored in a file, because it is continually generated. Also, what is different about this freezer/thawer that makes it superior to, for instance, YAML or compressed Data::Dumper dumps? That is, for what general class of use cases would one prefer the Neves data marshalling system to the available others? Also, if you're reusing other things in it -- you say storables are you doing something like prefixing length and colon to the output of Storable and jamming a bunch of those together? -- the underlying technology and then stream might make sense, especially if it can take an open handle instead of a file name in its constructor. Such a thing could be nice for creating an abstracted pipe between distributed processes for instance.
Re: Storable Stream - Module namespace help
On Monday 09 August 2010 20:59:50 David Nicol wrote: On Sun, Aug 8, 2010 at 4:58 AM, Marco Neves [ModAuthors] perl-module-auth...@knowhunter.cjb.net wrote: Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I'm confused -- a stream can't be stored in a file, because it is continually generated. Also, what is different about this freezer/thawer that makes it superior to, for instance, YAML or compressed Data::Dumper dumps? That is, for what general class of use cases would one prefer the Neves data marshalling system to the available others? Also, if you're reusing other things in it -- you say storables are you doing something like prefixing length and colon to the output of Storable and jamming a bunch of those together? -- the underlying technology and then stream might make sense, especially if it can take an open handle instead of a file name in its constructor. Such a thing could be nice for creating an abstracted pipe between distributed processes for instance. Hi, Yes, the module does store a lot of storables on a single file, and yes, it accepts an open handler as constructor. I written it to transfer large amounts of small data structures between servers. I'm thinking to use the name Archive::StorableStream for the module. The git repository is in: http://git.magick-source.net/perl5/storablestream Before you tell it, I know, none of the docs are still done. mpn -- Magick Source http://www.magick-source.net
Re: Storable Stream - Module namespace help
On Mon, Aug 9, 2010 at 12:41 PM, Marco Neves perl-module-auth...@knowhunter.cjb.net wrote: On Monday 09 August 2010 20:59:50 David Nicol wrote: On Sun, Aug 8, 2010 at 4:58 AM, Marco Neves [ModAuthors] perl-module-auth...@knowhunter.cjb.net wrote: Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I'm confused -- a stream can't be stored in a file, because it is continually generated. Also, what is different about this freezer/thawer that makes it superior to, for instance, YAML or compressed Data::Dumper dumps? That is, for what general class of use cases would one prefer the Neves data marshalling system to the available others? Also, if you're reusing other things in it -- you say storables are you doing something like prefixing length and colon to the output of Storable and jamming a bunch of those together? -- the underlying technology and then stream might make sense, especially if it can take an open handle instead of a file name in its constructor. Such a thing could be nice for creating an abstracted pipe between distributed processes for instance. Hi, Yes, the module does store a lot of storables on a single file, and yes, it accepts an open handler as constructor. I written it to transfer large amounts of small data structures between servers. I'm thinking to use the name Archive::StorableStream for the module. The git repository is in: http://git.magick-source.net/perl5/storablestream Before you tell it, I know, none of the docs are still done. FWIW, have you considered how to deal with what happens if your chunks don't match the size prefix? If I read your protocol correctly, you are sending the length of the following blob as ASCII digits terminated by a \0. $blob = Storable::nfreeze( ... ) $packed = length( $blob ) . \0$blob and to read: $length = 0 + $packed; Storable::thaw( substr $packed, length $length ) I dunno your objection to YAML but it has the nice property that it provides context for helping to recover from mis-transmitted documents in the stream by virtue of having a nested-ness marker (the indentation) and both starting and ending document tokens (--- and ...). Further, I would highly discourage you from naming anything using Storable with Archive because absolutely no one should ever consider actually *storing* a Storable blob. If you keep one around then you'll be tempted to read it in a client version that didn't match what you created it with. That way lies madness. Josh
Re: Storable Stream - Module namespace help
Does it actually use the Storable format/algorithm or are you using storable in a more generic sense of the word? If the latter, I'd avoid using it in the name of your module to avoid confusion. On Sun, Aug 8, 2010 at 2:58 AM, Marco Neves [ModAuthors] perl-module-auth...@knowhunter.cjb.net wrote: Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I created it in a private Namespace using in the company I worked for but, with they permission, and because I think this may be useful for someone else, I want to publish it on cpan. I'm finishing documentation and tests, before uploading it to CPAN, but I don't know exactly which namespace to use. For me, the obvious namespace would be Storable::Stream (and eventually Storable::Stream::GZip), but as Storable is part of core, I think we are not expected to publish modules in that namespace. So, which name should I use for this small module? Thanks, mpneves -- Magick Source http://www.magick-source.net -- Magick Source http://www.magick-source.net -- Check out my LEGO blog at http://www.brickpile.com/ View my photos at http://flickr.com/photos/billward/ Follow me at http://twitter.com/williamward
Re: Storable Stream - Module namespace help
* Joshua ben Jore twi...@gmail.com [2010-08-10 01:55]: Further, I would highly discourage you from naming anything using Storable with Archive because absolutely no one should ever consider actually *storing* a Storable blob. If you keep one around then you'll be tempted to read it in a client version that didn't match what you created it with. That way lies madness. Good point. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Storable Stream - Module namespace help
* Joshua ben Jore twi...@gmail.com [2010-08-09 06:15]: I'd feel much better suggesting you consider streaming with YAML or JSON or something hand-rolled but anything but Storable. I would strike YAML from that list as well. Whereas JSON is great. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Storable Stream - Module namespace help
Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I created it in a private Namespace using in the company I worked for but, with they permission, and because I think this may be useful for someone else, I want to publish it on cpan. I'm finishing documentation and tests, before uploading it to CPAN, but I don't know exactly which namespace to use. For me, the obvious namespace would be Storable::Stream (and eventually Storable::Stream::GZip), but as Storable is part of core, I think we are not expected to publish modules in that namespace. So, which name should I use for this small module? Thanks, mpneves -- Magick Source http://www.magick-source.net -- Magick Source http://www.magick-source.net
Re: Storable Stream - Module namespace help
* Marco Neves [ModAuthors] perl-module-auth...@knowhunter.cjb.net [2010-08-08
12:00]:
So, which name should I use for this small module?
First thing that popped into my head was Archive::Storable.
Now I can’t decide whether it’s perfect or silly.
--
*AUTOLOAD=*_;sub _{s/::([^:]*)$/print$1,(,$\/, )[defined
wantarray]/e;chop;$_}
Just-another-Perl-hack;
#Aristotle Pagaltzis // http://plasmasturm.org/
Storable Stream - Module namespace help
Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I created it in a private Namespace using in the company I worked for but, with they permission, and because I think this may be useful for someone else, I want to publish it on cpan. I'm finishing documentation and tests, before uploading it to CPAN, but I don't know exactly which namespace to use. For me, the obvious namespace would be Storable::Stream (and eventually Storable::Stream::GZip), but as Storable is part of core, I think we are not expected to publish modules in that namespace. So, which name should I use for this small module? Thanks, mpneves -- Magick Source http://www.magick-source.net
Re: Storable Stream - Module namespace help
On Sun, Aug 8, 2010 at 2:40 AM, theMage [Marco Neves] perl-c...@knowhunter.cjb.net wrote: Hello, In need for a way to transfer a large amount of small datastructures I created a module that stores and retrieves a stream of storables in a file. I created it in a private Namespace using in the company I worked for but, with they permission, and because I think this may be useful for someone else, I want to publish it on cpan. I'm finishing documentation and tests, before uploading it to CPAN, but I don't know exactly which namespace to use. For me, the obvious namespace would be Storable::Stream (and eventually Storable::Stream::GZip), but as Storable is part of core, I think we are not expected to publish modules in that namespace. So, which name should I use for this small module? The namespace ought to be just fine. The package Storable is owned by core but Storable::Stream isn't. FWIW though, I am provoked by SO MUCH PAIN transferring Storable objects between processes and servers that I feel like I must warn you against ever actually using Storable in IPC. I've had so many problems when one of either client or server for Storable-trading IPC wanted to change. Have you tried upgrading or downgrading either side of the transaction to a new, potentially incompatible version of Storable yet? I'd feel much better suggesting you consider streaming with YAML or JSON or something hand-rolled but anything but Storable. Consider that maybe your next client will want to be implemented in Ruby but now won't be able to read the data. BTW, where's your repo? github? I'd like to take a gander. Josh
