Re: Cap'n Proto for D v0.1.2
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang Hi Thomas, Great that you are working on this. I strongly advise you to advertise your work on this page: https://capnproto.org/otherlang.html Cheers, Johan
Re: Cap'n Proto for D v0.1.2
On Thursday, 20 April 2017 at 18:25:03 UTC, Jonathan M Davis wrote: On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via Digitalmars-d- announce wrote: LOL. Oh, I remember a coworker bringing up this library. It's the one that the site claimed was infinitely faster than protocol buffers (IIRC, because of some work that protocol buffers does that Captain Proto skips entirely). Exactly! Well, "Captain Proto" knows what he does, as he was the man that wrote google protobuf. The same way of working of Cap'n Proto is what Google introduced later in FlatBuffer... but it's missing totally all the good stuff taken from e-lang, that in my opinion is a big plus. Don't take me wrong, I know that a lot is to be implemented, today, but I love man with a clean path towards brilliant ideas [1] [1] https://capnproto.org/rpc.html /Paolo
Re: Cap'n Proto for D v0.1.2
On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via Digitalmars-d- announce wrote: > "Cap’n Proto is an insanely fast data interchange format and > capability-based RPC system. Think JSON, except binary. Or think > Protocol Buffers, except faster." > > This is the initial public release of my optimized port of the > Java implementation of Cap'n Proto. > > State: > * Passes Cap'n Proto testsuite. > * Optimized. Just a little slower than the official C++ > implementation (see benchmarks on github). > * Missing RPC part of Cap'n Proto. > > http://code.dlang.org/packages/capnproto-dlang > https://github.com/ThomasBrixLarsen/capnproto-dlang LOL. Oh, I remember a coworker bringing up this library. It's the one that the site claimed was infinitely faster than protocol buffers (IIRC, because of some work that protocol buffers does that Captain Proto skips entirely). - Jonathan M Davis
Re: Cap'n Proto for D v0.1.2
On Wednesday, 19 April 2017 at 19:30:03 UTC, Thomas Brix Larsen wrote: On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote: [...] Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'. The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap. But reading using random access *is* a feature of Cap'n Proto. So when reading a memory mapped Cap'n Proto file, getters will be faster if you use it in a non-sequential way. I have made some minor changes to make adaptors responsible for allocating buffers. This made it possible to add a new adaptor MemoryMapped and a sample[1] to show it off. I have also verified that no buffer allocations happen while reading using MemoryMapped. Note that writing using memory mapping is still not possible. [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/samples/mmap.d
Re: Cap'n Proto for D v0.1.2
On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote: [...] Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'. The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap. But reading using random access *is* a feature of Cap'n Proto. So when reading a memory mapped Cap'n Proto file, getters will be faster if you use it in a non-sequential way. The format of Cap'n Proto doesn't currently support *writing* to a memory mapped file.
Re: Cap'n Proto for D v0.1.2
On Wednesday, 19 April 2017 at 16:52:14 UTC, Thomas Brix Larsen wrote: Take a look at FileDescriptor[1]. It is a class I've added to support read/write using File from std.stdio. You can create a similar streamer using std.mmfile. I believe that this would be enough for memory mapped reading. [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d Ok, thanks. I took a look at several capnproto implementations just now, and didn't see any tests for a mmap 'feature'. The roadmap doc below indicates it doesn't exist, and perhaps there are some details yet to be resolved to make it 'friendly' for a mmap. https://capnproto.org/roadmap.html mmap-friendly mutable storage format: Define a standard storage format that is friendly to mmap-based use while allowing modification. (With the current serialization format, mmap is only useful for read-only structures.) Possibly based on the ORM interface, updates only possible at the granularity of a whole ORM entry. In java the MappedByteBuffer can be used with a RandomAccessFile channel, and then the accesses of the loaded map can be random, without requiring sequential accesses of the whole mapped file. So java nio already has some higher level classes in place that would make it easier to develop a first implementation of the mmap features.
Re: Cap'n Proto for D v0.1.2
On Wednesday, 19 April 2017 at 16:34:02 UTC, Jay Norwood wrote: [...] This info from stackoverflow also seems to imply that MappedByteBuffer would be required for some of the capnproto features. So, could you explain a little more about what are the capabilities of the current d library implementation, with just the ByteBuffer implemented from the java nio code? Thanks, Jay [...] The port of ByteBuffer just wraps a slice when reading in D. Take a look at FileDescriptor[1]. It is a class I've added to support read/write using File from std.stdio. You can create a similar streamer using std.mmfile. I believe that this would be enough for memory mapped reading. [1]: https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d
Re: Cap'n Proto for D v0.1.2
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." The features below, from the capnproto.org description, interest me. However a MappedByteBuffer would be used for the mmap feature in java. https://capnproto.org/ mmap: Read a large Cap’n Proto file by memory-mapping it. The OS won’t even read in the parts that you don’t access. Inter-language communication: Calling C++ code from, say, Java or Python tends to be painful or slow. With Cap’n Proto, the two languages can easily operate on the same in-memory data structure. Inter-process communication: Multiple processes running on the same machine can share a Cap’n Proto message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread. This info from stackoverflow also seems to imply that MappedByteBuffer would be required for some of the capnproto features. So, could you explain a little more about what are the capabilities of the current d library implementation, with just the ByteBuffer implemented from the java nio code? Thanks, Jay http://stackoverflow.com/questions/29361058/read-proto-partly-instead-of-full-parsing-in-java 'If you are willing to consider using a different protocol framework, Cap'n Proto is extremely similar in design to Protobufs but features in the ability to read only the part of the message you care about. Cap'n Proto incurs no overhead for the fields you don't examine, other than obviously the bandwidth and memory to receive the raw message bytes. If you are reading from a file, and you use memory mapping (MappedByteBuffer in Java), then only the parts of the message you actually use will be read from disk.'
Re: Cap'n Proto for D v0.1.2
On 2017-04-18 23:08, Dmitry Olshansky wrote: Risking a flamewar but what's wrong with Java? I don't like any language that force me to write in a particular style or paradigm. Because all problems cannot be solved (or not in a good way) in the same way. That said, my D code is quite heavily influenced by Java. -- /Jacob Carlborg
Re: Cap'n Proto for D v0.1.2
On 4/18/17 9:14 PM, Swoorup Joshi wrote: On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. State: * Passes Cap'n Proto testsuite. * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github). * Missing RPC part of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang Java?? Yikes Risking a flamewar but what's wrong with Java? --- Dmitry Olshansky
Re: Cap'n Proto for D v0.1.2
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. State: * Passes Cap'n Proto testsuite. * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github). * Missing RPC part of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang Great Job! I'm following Cap'n Proto, and it's very very interesting... I would love also the RPC part, maybe based on Vibe... have you any plan to implement that? I'm really curious to try it! --- Paolo
Re: Cap'n Proto for D v0.1.2
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: "Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. State: * Passes Cap'n Proto testsuite. * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github). * Missing RPC part of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang Java?? Yikes
Cap'n Proto for D v0.1.2
"Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. State: * Passes Cap'n Proto testsuite. * Optimized. Just a little slower than the official C++ implementation (see benchmarks on github). * Missing RPC part of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang
