Re: [Chicken-users] Protocol Buffers for CHICKEN
I'm not sure the suspension egg needs a dependency on a serializer at all. It would still serve its purpose if continuation-suspend and continuation-resume passed around the native continuation objects and let the client do the serialization. Good point. I will change it accordingly. And configurability is almost always a good thing :-) Indeed. Thanks again for protobuf! cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
On 2013-05-30 22:32, Felix wrote: [...] Very good - thanks a lot! I'll give this a try. Should we simply replace the serialization or do you think it would make sense to let the user choose a serialization mechanism for suspensions? [...] Hello Felix, I'm not sure the suspension egg needs a dependency on a serializer at all. It would still serve its purpose if continuation-suspend and continuation-resume passed around the native continuation objects and let the client do the serialization. And configurability is almost always a good thing :-) Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
yes, I'm aware of suspension and the magic going on inside that egg is precisely what I referred to by fiddling around with green threads. In fact, protobuf can easily act as a drop-in replacement for s11n and I just verified that it does work just fine with suspensions! So I can say with more confidence now that continuation serialization through protobuf is functional. Apply the attached patch to the the suspension egg, install it and see for yourself :-) Very good - thanks a lot! I'll give this a try. Should we simply replace the serialization or do you think it would make sense to let the user choose a serialization mechanism for suspensions? cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
Felix scripsit: Very good - thanks a lot! I'll give this a try. Should we simply replace the serialization or do you think it would make sense to let the user choose a serialization mechanism for suspensions? Given the advantages of PBs, I'd just move to them for all applications of serialization, and abandon the native Chicken serialization. -- It was dreary and wearisome. Cold clammy winter still held sway in this forsaken country. The only green was the scum of livid weed on the dark greasy surfaces of the sullen waters. Dead grasses and rotting reeds loomed up in the mists like ragged shadows of long-forgotten summers. --The Passage of the Marshes http://www.ccil.org/~cowan ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
From: Thomas Chust ch...@web.de Subject: [Chicken-users] Protocol Buffers for CHICKEN Date: Wed, 29 May 2013 00:15:20 +0200 Hello, during the CHICKEN spring thing in Cologne I started to work on a new egg [1] implementing the protocol buffer [2] serialization format, which is now in a usable and tested state. If you don't need or want to use a specific schema for your data, you can use the protobuf egg as a generic serialization solution that produces platform-independent binary representations of (almost) any CHICKEN values: Whoa! Incredible! Finally a decent serialization library. Is closure/continuation-serialization reliable? I did some simple stupid tests and at least serializing a continuation in csi produced an error due to a not-serializable port (which is understandable). cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
closure serialization definitely works, I have tried round tripping compiled and interpreted procedures with and without surrounding context through serialize and deserialize and they are still functional after reading them back in. Excellent. Continuations should work, too, but it is non-trivial to construct a continuation that doesn't close over all sorts of crazy stuff like the input ports from which the interpreter reads library files etc. Fiddling around with green threads to delimit the continuations and using the facilities to inject external dependencies into the serialization context of protobuf should do the job, though. Well, ports for example will never be fully serializable (with the exception of standard ports). Have you seen suspensions? https://wiki.call-cc.org/eggref/4/suspension They currently use s11n but protobuf would be the better mechanism. This might be the path for having a real termite for chicken ... cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
Hi, during the CHICKEN spring thing in Cologne I started to work on a new egg [1] implementing the protocol buffer [2] serialization format, which is now in a usable and tested state. Wow! This looks super cool! I'm so sad to have missed the Spring Thing. I wonder if you might entertain some rookie questions with regard to the way serialisation and deserialisation of procedures works. + Does it emit the object code of compiled procedures? + What of the execute bit in the OS's page mapping? + What happens if that procedure closes over a global variable or parameter? What value will it take? + Does it work between different binaries? As I understand it, the s11n doesn't work in the general case for this. + What of binaries compiled with different Chicken or GCC versions? + How does the binding and loading of eggs work? Thanks for spending the time to create this. Regards, @ndy -- andy...@ashurst.eu.org http://www.ashurst.eu.org/ 0x7EBA75FF ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
[Chicken-users] Protocol Buffers for CHICKEN
Hello,
during the CHICKEN spring thing in Cologne I started to work on a new
egg [1] implementing the protocol buffer [2] serialization format, which
is now in a usable and tested state.
If you don't need or want to use a specific schema for your data, you
can use the protobuf egg as a generic serialization solution that
produces platform-independent binary representations of (almost) any
CHICKEN values:
$ cat source.scm
(require-library protobuf)
(import protobuf-generic)
(serialize (lambda (x) (print (* 2 x
$ csi -s source.scm lambda.pbf
$ cat sink.scm
(require-library protobuf)
(import protobuf-generic)
((deserialize) 42)
$ csi -s sink.scm lambda.pbf
84
The serialized data can be read by other protocol buffer enabled
applications, it may not have the most convenient structure, though.
So if you have a need to communicate with other software that uses
protocol buffer definitions, you can use the protoc compiler plugin that
comes with this egg to generate a CHICKEN binding automatically from
existing schema definitions:
$ cat person.proto
package person;
message Person {
required int32 id = 1;
required string name = 2;
optional string email = 3;
}
$ protoc --proto_path=. --chicken_out=. person.proto
$ cat test.scm
(require-library protobuf)
(include person.scm)
(import protobuf person)
(serialize (make-person id: 42 name: Jane Doe))
$ csi -s test.scm | \
protoc --proto_path=. person.proto --decode=person.Person
id: 42
name: Jane Doe
Deserialization is just as simple with a call to (deserialize person).
The protobuf messages are represented as SRFI-99 records in CHICKEN that
can be manipulated as usual. Enumeration values are represented as symbols.
If you're interested, check the egg documentation for advanced features
and give the library a try :-)
Ciao,
Thomas
--
[1] https://wiki.call-cc.org/eggref/4/protobuf
[2] http://protobuf.googlecode.com/
--
When C++ is your hammer, every problem looks like your thumb.
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
This is very welcome!
I wonder if this would be useful for storing data in a posix shared
memory block...
-Dan
On 5/28/2013 3:15 PM, Thomas Chust wrote:
Hello,
during the CHICKEN spring thing in Cologne I started to work on a new
egg [1] implementing the protocol buffer [2] serialization format, which
is now in a usable and tested state.
If you don't need or want to use a specific schema for your data, you
can use the protobuf egg as a generic serialization solution that
produces platform-independent binary representations of (almost) any
CHICKEN values:
$ cat source.scm
(require-library protobuf)
(import protobuf-generic)
(serialize (lambda (x) (print (* 2 x
$ csi -s source.scm lambda.pbf
$ cat sink.scm
(require-library protobuf)
(import protobuf-generic)
((deserialize) 42)
$ csi -s sink.scm lambda.pbf
84
The serialized data can be read by other protocol buffer enabled
applications, it may not have the most convenient structure, though.
So if you have a need to communicate with other software that uses
protocol buffer definitions, you can use the protoc compiler plugin that
comes with this egg to generate a CHICKEN binding automatically from
existing schema definitions:
$ cat person.proto
package person;
message Person {
required int32 id = 1;
required string name = 2;
optional string email = 3;
}
$ protoc --proto_path=. --chicken_out=. person.proto
$ cat test.scm
(require-library protobuf)
(include person.scm)
(import protobuf person)
(serialize (make-person id: 42 name: Jane Doe))
$ csi -s test.scm | \
protoc --proto_path=. person.proto --decode=person.Person
id: 42
name: Jane Doe
Deserialization is just as simple with a call to (deserialize person).
The protobuf messages are represented as SRFI-99 records in CHICKEN that
can be manipulated as usual. Enumeration values are represented as symbols.
If you're interested, check the egg documentation for advanced features
and give the library a try :-)
Ciao,
Thomas
--
[1] https://wiki.call-cc.org/eggref/4/protobuf
[2] http://protobuf.googlecode.com/
___
Chicken-users mailing list
Chicken-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
On 2013-05-29 00:32, Dan Leslie wrote: [...] I wonder if this would be useful for storing data in a posix shared memory block... [...] Hello Dan, that is certainly possible, you would just combine serialize and call-with-output-string to obtain data you can copy into a shared buffer and call-with-input-string plus deserialize to extract the stored value on the receiving side. However, shared memory between processes or threads on the same machine has the advantage that one can place data in native formats in there without having to care about endianness issues etc. So perhaps object-evict and friends would be more efficient in this case. The protobuf serialization could be quite useful for distributed computing applications. Combining serialization of thunks with network transport and some cryptographic authentication scheme yields you could quickly construct a secure remote compute job submission server or a distributed map reduce network, for example. Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Protocol Buffers for CHICKEN
Huh, now that is useful! https://wiki.call-cc.org/man/4/Unit%20lolevel#object-evict Still, if I ever have call to use pshm for ipc between chicken and not-chicken workers then this egg would probably be useful. Thanks again, -Dan On 5/28/2013 5:24 PM, Thomas Chust wrote: On 2013-05-29 00:32, Dan Leslie wrote: [...] I wonder if this would be useful for storing data in a posix shared memory block... [...] Hello Dan, that is certainly possible, you would just combine serialize and call-with-output-string to obtain data you can copy into a shared buffer and call-with-input-string plus deserialize to extract the stored value on the receiving side. However, shared memory between processes or threads on the same machine has the advantage that one can place data in native formats in there without having to care about endianness issues etc. So perhaps object-evict and friends would be more efficient in this case. The protobuf serialization could be quite useful for distributed computing applications. Combining serialization of thunks with network transport and some cryptographic authentication scheme yields you could quickly construct a secure remote compute job submission server or a distributed map reduce network, for example. Ciao, Thomas ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
