Re: Cerealed v0.2.0 - a(nother) D serialisation library
Done.
Thanks for the feature request and using Cerealed! I hope it's
useful to you.
Atila
On Monday, 8 September 2014 at 19:09:30 UTC, Vicente wrote:
Hello Atila,
I've seen that cerealed has made much progress, it's now at
v0.5.0
This post is for a feature request, I would like to use it with
a struct like this:
import cerealed;
struct nested_associative_array { nested_associative_array[int]
x; }
struct some_struct {
string[] x;
int[][] y;
nested_associative_array[] z;
};
void main(char[][] args) {
some_struct original, restored;
auto enc = Cerealiser();
enc ~= original;
auto dec = Decerealiser(enc.bytes);
restored = dec.value!(some_struct);
assert(original==restored);
}
The serialization process compiles fine, but the restoring part
fails to compile.
I've tried to figure out how to do it, but I am still a
beginner on D.
How difficult do you think it is?
The alternative, orange, works fine, but the struct I'm trying
to export to disk is a few hundreds of MB in size and orange
takes hours to complete and may also run out of memory. Your
implementation, instead, is really fast and efficient!
Re: Cerealed v0.2.0 - a(nother) D serialisation library
I'll take a look. It's easier to post this on github though.
Atila
On Monday, 8 September 2014 at 19:09:30 UTC, Vicente wrote:
Hello Atila,
I've seen that cerealed has made much progress, it's now at
v0.5.0
This post is for a feature request, I would like to use it with
a struct like this:
import cerealed;
struct nested_associative_array { nested_associative_array[int]
x; }
struct some_struct {
string[] x;
int[][] y;
nested_associative_array[] z;
};
void main(char[][] args) {
some_struct original, restored;
auto enc = Cerealiser();
enc ~= original;
auto dec = Decerealiser(enc.bytes);
restored = dec.value!(some_struct);
assert(original==restored);
}
The serialization process compiles fine, but the restoring part
fails to compile.
I've tried to figure out how to do it, but I am still a
beginner on D.
How difficult do you think it is?
The alternative, orange, works fine, but the struct I'm trying
to export to disk is a few hundreds of MB in size and orange
takes hours to complete and may also run out of memory. Your
implementation, instead, is really fast and efficient!
Re: Cerealed v0.2.0 - a(nother) D serialisation library
assert(foo2.a[1] == 10); // will this pass? Ah, I see. No, it won't. Which answers the other question about serialising references to objects. I'm told this is a common problem, but this being my first bash at a serialisation library... I was certain that Orange would be more mature and handle more corner cases than this library (or else it wouldn't be considered for inclusion in the std lib), but I wanted to share it since it works for what I'm doing, does things a bit differently and does what I need. Also, it was fun coding it. If anything, if you think some of these ideas are worth it, feel free to use them. I'm just glad to not have to write the same old bit twiddling code for networking headers again! Even if you know the identifier in the header file you need a static type when deserializing: decerealiser.value!(Foo) Even if you can instantiate the right subclass you need to have the static type information to deserialize it. It is possible to do, as I have done in Orange. The requirement is to register the subclasses that need to be serialized through a base class reference. See https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d#L241..L262 and https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d#L787..L788 True, I thought of doing something like it to simplify my MQTT broker implementation but ended up deciding against it.
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 2013-11-20 18:26, Atila Neves wrote: Ah, I see. No, it won't. Which answers the other question about serialising references to objects. I'm told this is a common problem, but this being my first bash at a serialisation library... It's a lot easier to add support for references than slices. The reason is that you always know when there is a reference type. With arrays you don't know if it's a slice of some array or an actual array. The type system treats both slices and arrays the same. -- /Jacob Carlborg
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 2013-11-19 09:19, Rory McGuire wrote: Thanks. I believe that is the example that made me think that its main purpose was for archiving. So is an Archive actually a serialization driver? Can one make a protobuf driver or a hessian driver? Yes, that should be possible. It depends on how the data looks like. The upcoming std.serialization version will relax quite a lot of the requirements of the archiver. That means more archivers can be implemented. Are the serializers able to stream data, with something like a range interface? No, unfortunately not. I'm working on including Orange into Phobos as std.serialization. That version will have a range API. -- /Jacob Carlborg
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On Monday, 18 November 2013 at 20:43:51 UTC, Jacob Carlborg wrote: On 2013-11-18 17:49, Atila Neves wrote: http://code.dlang.org/packages/cerealed https://github.com/atilaneves/cerealed How does it handle: * Slices - are they restored? It serialises the slice and restores it as it was, not the data behind it. I should add some tests for that. * Pointers - are they restored? No, good point. I keep forgetting there are legitimate uses of pointers in D. I was looking at the SList implementation the other day and was surprised to see new used to create a struct on the heap, something that would seem totally normal to me in C++. It should be easy enough to extend to pointers, I'll add that in the next version. * Objects - is the same object only serialized once or duplicated? I'm not sure I understand the question. * Serializing through base class reference - possible? No. I can't see how that would be possible without implementing an interface (I may well be wrong), and that was the kind of thing I really didn't want to do. Not only that, but for the kind of code I write, I either want to send something to the network, in which case I know which class to instantiate and don't need a base class reference, or I want to unmarshall bytes that came in. In the latter case there's usually an identifier in the header to tell the factory method which class to instantiante. So that doesn't usually come up. Atila
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On Tue, Nov 19, 2013 at 3:24 PM, Atila Neves atila.ne...@gmail.com wrote: * Objects - is the same object only serialized once or duplicated? I'm not sure I understand the question. ... Atila Perhaps Jacob refers to a lot of serialization formats being capable of storing references to an object, this makes it possible to store circular lists for example.
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 2013-11-19 14:30, Rory McGuire wrote: Perhaps Jacob refers to a lot of serialization formats being capable of storing references to an object, this makes it possible to store circular lists for example. Yes, exactly. -- /Jacob Carlborg
Cerealed v0.2.0 - a(nother) D serialisation library
http://code.dlang.org/packages/cerealed
https://github.com/atilaneves/cerealed
But what about Orange / std.serialization? Well,
1. I started this before I knew about Orange. This is actually
based on a library I wrote in C++
(https://bitbucket.org/atilaneves/cereal) for the game I'm trying
to finish, and it dawned on me it would probably be easier in D.
It was.
2. Orange serialises to XML, I need/want binary. I could've
written a binary archive, I guess, but I'd rather do this.
3. I would've written this for fun anyway.
The deal here is binary serialisation with minimal to no
boilerplate. As mentioned in the README, I used this to implement
a MQTT broker with minimal fuss. Serialising classes, structs,
and normal D types requires writing nothing at all. Custom
serialisation can be handled in two different ways and both of
them require writing only one function for both struct/class -
bytes and vice-versa. Since a code sample is worth a 1000 words:
import cerealed; //uses package.d from 2.064
struct MyStruct {
ubyte mybyte1;
@NoCereal uint nocereal1; //won't be serialised
//the next 3 members will all take up one byte
@Bits!4 ubyte nibble; //gets packed into 4 bits
@Bits!1 ubyte bit; //gets packed into 1 bit
@Bits!3 ubyte bits3; //gets packed into 3 bits
ubyte mybyte2;
}
auto enc = new Cerealiser();
enc ~= MyStruct(3, 123, 14, 1, 2, 42);
assert(enc.bytes == [ 3, 0xea /*1110 1 010*/, 42]);
auto dec = new Decerealizer([ 3, 0xea, 42]); //US spelling
works too
//the 2nd value is 0 and not 123 since that value
//doesn't get serialised/deserialised
assert(dec.value!MyStruct == MyStruct(3, 0, 14, 1, 2, 42));
Custom serialisation is explained in the README. I had to use it
for my MQTT project and it worked quite well.
I still need to add a bunch of negative tests and accompanying
code but this is good enough to fool around with for the moment.
Atila
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 2013-11-18 17:49, Atila Neves wrote: http://code.dlang.org/packages/cerealed https://github.com/atilaneves/cerealed How does it handle: * Slices - are they restored? * Pointers - are they restored? * Objects - is the same object only serialized once or duplicated? * Serializing through base class reference - possible? -- /Jacob Carlborg
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 18 Nov 2013 22:45, Jacob Carlborg d...@me.com wrote: How does it handle: * Slices - are they restored? * Pointers - are they restored? * Objects - is the same object only serialized once or duplicated? * Serializing through base class reference - possible? -- /Jacob Carlborg Hi Jacob, Is there an example somewhere of how to use Orange to serialise data that will not be going to a file? When I looked at it appeared as though Orange's archives are designed for file serialising only. -Rory
Re: Cerealed v0.2.0 - a(nother) D serialisation library
On 2013-11-19 06:49, Rory McGuire wrote: Hi Jacob, Is there an example somewhere of how to use Orange to serialise data that will not be going to a file? When I looked at it appeared as though Orange's archives are designed for file serialising only. Orange doesn't have any data transport layer or similar. But you can do whatever you want with the data you get from the archiver. Check the usage example in the readme[1]. Use archive.data to get the typed data after serialization and do whatever you want with it. https://github.com/jacob-carlborg/orange#simple-usage-example -- /Jacob Carlborg
