Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

On Tuesday, July 25, 2017 at 11:37:48 PM UTC+2, Lehi Toskin wrote:
> One thing I'm curious about is what things can you and can you not pack? In 
> the README it shows bytes being packed, which seems a little obvious, but 
> what about (transparent) structs? Hashes? Lists? I'm very interested in this 
> package... for science!

I am not packing bytes in the readme, I am packing an integer in hexadecimal
notation so one can see easier how the integer corresponds to the packed bytes:
#x1234 becomes #xCD #x12 #x34, the first byte is the type tag for unsigned
16-bit integers and the other two are the bytes of the integer.

Packing means turning a Racket object into a byte sequence that can be sent to
another process. The recipient might not know anything about Racket, but it
still needs to be able to figure out how  to put the byte sequence back 
together,
that's why there is a type tag.

In theory you can pack anything as long as the recipient knows how it was
packed. Vectors and lists get packed into MessagePack arrays, hash tables into
MessagePack maps. If you want to pack a custom type like a point struct
'(struct point (x y))' you have a number of options. You could use the
MessagePack ext type which allows you to assign a custom type tag, then you
have to tell the recipient that for example the magic number 13 means point. If
the struct is transparent and you can 'read' and 'write' it in Racket you could
also pack the point as a string, but then you somehow have to make sure the
recipient knows that the string is meant to be an object and not just some
arbitrary text.

I think it would be worthwhile to later add the ability to define your own
packing functions so that you could then just call '(pack my-point out)'.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

Here is what I get from profiling the different dispatch methods in 'unpack'.
First I pack a positive fixnum which is among the first few conditions.

Cond:
cpu time: 810 real time: 3367 gc time: 0
cpu time: 815 real time: 3428 gc time: 0
cpu time: 815 real time: 3197 gc time: 0
cpu time: 806 real time: 3045 gc time: 0
cpu time: 806 real time: 3125 gc time: 0

cpu time: 811.5 real time: 3232.4

Case:
cpu time: 817 real time: 3122 gc time: 0
cpu time: 835 real time: 3544 gc time: 0
cpu time: 830 real time: 3337 gc time: 0
cpu time: 824 real time: 3216 gc time: 0
cpu time: 840 real time: 3667 gc time: 0

cpu time: 829.2 real time: 3377.2

Vector:
cpu time: 844 real time: 3839 gc time: 0
cpu time: 826 real time: 3253 gc time: 0
cpu time: 835 real time: 3194 gc time: 0
cpu time: 840 real time: 3165 gc time: 0
cpu time: 838 real time: 3354 gc time: 0

cpu time: 836.6 real time: 3361.0

Next I packed a negative fixnum, which is the last condition case:

Cond:
cpu time: 953 real time: 3556 gc time: 0
cpu time: 940 real time: 3035 gc time: 0
cpu time: 986 real time: 4250 gc time: 2
cpu time: 946 real time: 3425 gc time: 0
cpu time: 945 real time: 3233 gc time: 0

cpu time: 954.0 real time: 3499.8

Case:
cpu time: 890 real time: 2820 gc time: 0
cpu time: 896 real time: 2799 gc time: 0
cpu time: 904 real time: 2783 gc time: 2
cpu time: 884 real time: 2751 gc time: 0
cpu time: 887 real time: 2789 gc time: 0

cpu time: 892.2 real time: 2788.4

Vector:
cpu time: 866 real time: 2704 gc time: 0
cpu time: 884 real time: 2981 gc time: 0
cpu time: 880 real time: 2661 gc time: 3
cpu time: 864 real time: 2647 gc time: 0
cpu time: 864 real time: 2597 gc time: 0

cpu time: 871.6 real time: 2718.0

The average is below the line for each. I tested it by unpacking an array of
2^16 items, all the values were hard-coded so there is no chance of poor macros
messing things up. The dispatch vector is a vector of functions like this:

(define dispatch
  (vector
...
(lambda (in) (unpack-float 32 in))  ; #xCA
...
  ))

The unpacking function looks like this:

(define (unpack in)
  (define tag (read-byte in))
  (time ((vector-ref dispatch tag) in)))

So in conclusion it looks like the vector is the fastest when it comes to cases
later down the list while the 'cond' is the fastest at first, but slowest at
the end, slower than the vector at the beginning. The case averages somewhere
between the two.

Should I pick 'case' so I get average performance consistently, or should I
pick the vector so I get the best performance in the worst case? I don't think
one can predict which type of data users will most likely want to unpack on
average.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

Thank you all for your help. I have moved to project to a flat file structure
now and reorganised everything hopefully to Racket standards (thank you,
Philip), and I have changed the complicated contracts to use 'integer-in'. I
think the API can stay as it is now, there is one struct and two functions, I
don't know anything that could be done that breaks backwards compatibility for
that. The package is up on the Racket package catalog:

https://pkgd.racket-lang.org/pkgn/package/msgpack

Do you guys think it is better to have a mandatory in/out port argument, or
should the port default to the current input/output port? I think the latter
makes it too easy to forget the port and I don't think users would want to use
stdin/stdout for this very often.

Another question: what about the name of the repository? Should it be renamed
to "msgpack" to match the name of the library or does the name not matter? I
can change the URL of the repository on GitLab, that would break all current
links, but at the moment that's not a big deal.

I'll clean the interior parts a bit more later (the 'unpack' dispatch and
cleaning the style), but that's not a priority as long as the external parts
are all set in place and working.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Jon Zeppieri]

On Mon, Jul 24, 2017 at 4:40 PM, Jay McCarthy  wrote:
> On Mon, Jul 24, 2017 at 3:18 PM, Alejandro Sanchez
>  wrote:
>>> - I'm curious of the performance. In particular, I would expect that a
>>> computed jump in unpack could do you good. Did you try that?
>> I haven’t investigated performance yet. As I said, I am new to Racket, this 
>> is my first time doing anything useful in it, my only previous Scheme 
>> knowledge was from doing the exercises in SICP and dabbling in Guile a bit. 
>> What is a computed jump?
>
> Rather than having a big `cond`, you could look up the function that
> does the work in a vector and then call it. IMHO, msgpack was designed
> with that in mind, because tags that aren't immediate values are all
> nicely ordered. So you'd check the size, subtract a constant, and grab
> the appropriate procedure from a constant vector.
>

Or you can use `case`. Racket's `case` with densely-distributed fixnum
constants (like in your code) will:

1. Use a vector lookup to go from the case label (i.e. the tag value)
to the index of the RHS clause.
2. Use an open-coded binary search on the index to get to the code itself.

(The technique is described here:
http://scheme2006.cs.uchicago.edu/07-clinger.pdf)

You might want to define a macro on top of `case`, though, to handle
ranges of tags, since `case` doesn't have syntax for that.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Philip McGrath]

You probably want `integer-in` for the contract on `ext`.

Have you read about the special treatment of test submodules?
https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._main-and-test%29

That is the idiomatic Racket way to write tests that run automatically when
you want them and don't get in the way when you don't. If you have
particularly extensive tests, it can also make sense to have a file where
the program consists of nothing but the test submodule.

-Philip

On Mon, Jul 24, 2017 at 5:50 PM, Alejandro Sanchez 
wrote:

>
> > On 24 Jul 2017, at 22:40, Jay McCarthy  wrote:
> >
> > On Mon, Jul 24, 2017 at 3:18 PM, Alejandro Sanchez
> >  wrote:
> >>> - I'm curious of the performance. In particular, I would expect that a
> >>> computed jump in unpack could do you good. Did you try that?
> >> I haven’t investigated performance yet. As I said, I am new to Racket,
> this is my first time doing anything useful in it, my only previous Scheme
> knowledge was from doing the exercises in SICP and dabbling in Guile a bit.
> What is a computed jump?
> >
> > Rather than having a big `cond`, you could look up the function that
> > does the work in a vector and then call it. IMHO, msgpack was designed
> > with that in mind, because tags that aren't immediate values are all
> > nicely ordered. So you'd check the size, subtract a constant, and grab
> > the appropriate procedure from a constant vector.
> OK, that’s the sort of thing I would have done in C where the tag would be
> an index into an array of function pointers. Can you please point me to
> where in the manual it explains how to profile a single function?
>
> >>> - Your package collection is 'multi, which is fine, but normally you
> >>> just do that when you're defining something like data/heap or
> >>> net/turkeyrpc, where you are extending some existing collection. In
> >>> particular, you define msgpack and then you also define the test/pack
> >>> collection (where you might expect it to be tests/msgpack/pack). I
> >>> recommend having your collection be "msgpack" and putting your tests
> >>> inside a tests sub-directory.
> >> Just to make sure I understood correctly: ‘msgpack’ is the umbrella
> module that users import, ‘msgpack/test/pack’ (and ‘unpack’) are the test
> modules that will be run for testing only. How about the directory
> structure? I like to keep all source files in a source directory (my
> original reason for doing ‘multi), can I still do something like this?
> >>
> >>|-README
> >>|-LICENSE
> >>|-info.rkt
> >>|-source
> >>  |-msgpack.rkt
> >>  |-pack.rkt
> >>  |-unpack.rkt
> >>|-test
> >>  |-pack.rkt
> >>  |-pack
> >>|- ...
> >>  |-unpack.rkt
> >>  |-unpack
> >>|- …
> >>
> >> It doesn’t have to be exactly this structure, but the idea is that all
> project-realted files are in the root, all the source files in the source
> directory and all the test files in the test directory.
> >
> > You can do that, but you'd have to have an additional `main.rkt` file
> > at the top-level that would require the things in `source` then
> > re-export them. It is not really Racket style to do what you're
> > talking about, however. If you did do that, then you could call the
> > `source` directory, `private` and then it would have a Racket-y name,
> > but your project isn't really large enough to warrant it and those
> > files aren't actually provide.
> >
> > Furthermore, your test/pack.rkt and test/unpack.rkt modules aren't
> > necessary, because you should be testing with `raco test -c msgpack`,
> > which will just go find everything. There's no need to build such
> > things yourself. (Although, FWIW, I also wouldn't have separate those
> > tests into such small files with just one or two, because they are,
> > again, so small.)
> I guess I’m weird that way, but I think of a project like a box. When you
> buy a thing and open the box you want all the contents to be neatly
> separated: here is the manual, here is the warranty card, here are the
> parts, all wrapped nicely in a bag. You wouldn’t want the contents to be
> loose and spill all over the floor. That’s why I like to separate the
> project
> into directories by functionality (documentation, source, tests, manuals,
> …). Oh well, if that is the Racket style I’ll do it your way.
>
>
> >>> - On a style level, I think you should remove your lets and turn your
> >>> if/begin blocks into conds, for example:
> >> Good point.
> >>
> >>>
> >>> On Mon, Jul 24, 2017 at 9:17 AM, Alejandro Sanchez
> >>>  wrote:
>  Hello dear Racketeers,
> 
>  I have been writing an implementation of the MessagePack protocol for
> Racket
>  and I think the library is ready to be showcased now:
> 
>  https://gitlab.com/HiPhish/MsgPack.rkt
> 
> 
>  ### What is MessagePack? ###
> 
>  MessagePack is a binary data 

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

> On 24 Jul 2017, at 22:40, Jay McCarthy  wrote:
> 
> On Mon, Jul 24, 2017 at 3:18 PM, Alejandro Sanchez
>  wrote:
>>> - I'm curious of the performance. In particular, I would expect that a
>>> computed jump in unpack could do you good. Did you try that?
>> I haven’t investigated performance yet. As I said, I am new to Racket, this 
>> is my first time doing anything useful in it, my only previous Scheme 
>> knowledge was from doing the exercises in SICP and dabbling in Guile a bit. 
>> What is a computed jump?
> 
> Rather than having a big `cond`, you could look up the function that
> does the work in a vector and then call it. IMHO, msgpack was designed
> with that in mind, because tags that aren't immediate values are all
> nicely ordered. So you'd check the size, subtract a constant, and grab
> the appropriate procedure from a constant vector.
OK, that’s the sort of thing I would have done in C where the tag would be an 
index into an array of function pointers. Can you please point me to where in 
the manual it explains how to profile a single function?

>>> - Your package collection is 'multi, which is fine, but normally you
>>> just do that when you're defining something like data/heap or
>>> net/turkeyrpc, where you are extending some existing collection. In
>>> particular, you define msgpack and then you also define the test/pack
>>> collection (where you might expect it to be tests/msgpack/pack). I
>>> recommend having your collection be "msgpack" and putting your tests
>>> inside a tests sub-directory.
>> Just to make sure I understood correctly: ‘msgpack’ is the umbrella module 
>> that users import, ‘msgpack/test/pack’ (and ‘unpack’) are the test modules 
>> that will be run for testing only. How about the directory structure? I like 
>> to keep all source files in a source directory (my original reason for doing 
>> ‘multi), can I still do something like this?
>> 
>>|-README
>>|-LICENSE
>>|-info.rkt
>>|-source
>>  |-msgpack.rkt
>>  |-pack.rkt
>>  |-unpack.rkt
>>|-test
>>  |-pack.rkt
>>  |-pack
>>|- ...
>>  |-unpack.rkt
>>  |-unpack
>>|- …
>> 
>> It doesn’t have to be exactly this structure, but the idea is that all 
>> project-realted files are in the root, all the source files in the source 
>> directory and all the test files in the test directory.
> 
> You can do that, but you'd have to have an additional `main.rkt` file
> at the top-level that would require the things in `source` then
> re-export them. It is not really Racket style to do what you're
> talking about, however. If you did do that, then you could call the
> `source` directory, `private` and then it would have a Racket-y name,
> but your project isn't really large enough to warrant it and those
> files aren't actually provide.
> 
> Furthermore, your test/pack.rkt and test/unpack.rkt modules aren't
> necessary, because you should be testing with `raco test -c msgpack`,
> which will just go find everything. There's no need to build such
> things yourself. (Although, FWIW, I also wouldn't have separate those
> tests into such small files with just one or two, because they are,
> again, so small.)
I guess I’m weird that way, but I think of a project like a box. When you
buy a thing and open the box you want all the contents to be neatly
separated: here is the manual, here is the warranty card, here are the
parts, all wrapped nicely in a bag. You wouldn’t want the contents to be
loose and spill all over the floor. That’s why I like to separate the project
into directories by functionality (documentation, source, tests, manuals,
…). Oh well, if that is the Racket style I’ll do it your way.


>>> - On a style level, I think you should remove your lets and turn your
>>> if/begin blocks into conds, for example:
>> Good point.
>> 
>>> 
>>> On Mon, Jul 24, 2017 at 9:17 AM, Alejandro Sanchez
>>>  wrote:
 Hello dear Racketeers,
 
 I have been writing an implementation of the MessagePack protocol for 
 Racket
 and I think the library is ready to be showcased now:
 
 https://gitlab.com/HiPhish/MsgPack.rkt
 
 
 ### What is MessagePack? ###
 
 MessagePack is a binary data serialisation format. The website describes it
 "like JSON but fast and small". Unlike JSON the goal is not a format that's
 human-readable, but one that can be very quickly serialised, transported 
 and
 serialised.
 
 http://msgpack.org/
 
 
 ### About the Racket implementation ###
 
 My goal was to keep everything as simple as possible: there are only two
 functions: pack and unpack. If there is more than one way of packing an
 object
 the smallest format is selected automatically. Here is a taste:
 
  (require msgpack/pack msgpack/unpack)
  ;;; A wild hodgepodge to pack
  (define vec #(1 2 "hello" '(3 4) '() #t))

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

I will look into that later, one thing at a time :)

> On 24 Jul 2017, at 21:45, Vincent St-Amour  
> wrote:
> 
> Hi Alejandro,
> 
> This looks cool!
> 
> I don't see it listed at pkgs.racket-lang.org. It would be easier for
> users to discover it if you posted it there.
> 
> Vincent
> 
> 
> 
> On Mon, 24 Jul 2017 08:17:30 -0500,
> Alejandro Sanchez wrote:
>> 
>> Hello dear Racketeers,
>> 
>> I have been writing an implementation of the MessagePack protocol for Racket
>> and I think the library is ready to be showcased now:
>> 
>> https://gitlab.com/HiPhish/MsgPack.rkt
>> 
>> ### What is MessagePack? ###
>> 
>> MessagePack is a binary data serialisation format. The website describes it
>> "like JSON but fast and small". Unlike JSON the goal is not a format that's
>> human-readable, but one that can be very quickly serialised, transported and
>> serialised.
>> 
>> http://msgpack.org/
>> 
>> ### About the Racket implementation ###
>> 
>> My goal was to keep everything as simple as possible: there are only two
>> functions: pack and unpack. If there is more than one way of packing an 
>> object
>> the smallest format is selected automatically. Here is a taste:
>> 
>> (require msgpack/pack msgpack/unpack)
>> ;;; A wild hodgepodge to pack
>> (define vec #(1 2 "hello" '(3 4) '() #t))
>> ;;; A byte string of packed data
>> (define packed
>> (call-with-output-bytes (λ (out) (pack vec out
>> ;;; Unpack the data again
>> (define upacked
>> (call-with-input-bytes packed (λ (in) (unpack in
>> 
>> As you can see, data is packed to and unpacked from a binary port. I think 
>> this
>> is better than packing/unpacking to binary string because MessagePack is
>> primarily used for inter-process communication, so there is not much point in
>> keeping the packed data inside a process.
>> 
>> I'd appreciate it a lot if a seasoned Racketeer could take a look at my code,
>> in particular if the library is set up properly (the info.rkt files), this is
>> my first time doing something in Racket. I am also open to suggestions about
>> the API, I haven't committed to version 1.0 yet. In particular, I am not
>> familiar with the modularity conventions of Racket libraries, i.e. if it is 
>> OK
>> to have 'msgpack/pack' and 'msgpack/unpack' or if everything should be 
>> covered
>> by one large 'provide' from 'msgpack'? There is one new type 'ext' declared,
>> should that be part of 'msgpack' or should I move it to 'msgpack/types'
>> instead?
>> 
>> On a related note, I find it really annoying that 'integer->integer-bytes' 
>> and
>> 'integer-bytes->integer' do not support 8-bit integers. Is there a reason for
>> that? I had to write all sorts of ugly extra code for the 8-bit cases. I 
>> opened
>> an issue on GitHub about it (#1754).
>> 
>> ### What's next? ###
>> 
>> Once the API settles I would like to move the library to typed Racket. I 
>> would
>> also like to submit it to the Racket packages catalog. The reason I wrote 
>> this
>> library is because I want to eventually write a Racket API client for Neovim:
>> 
>> https://github.com/neovim/neovim
>> https://github.com/neovim/neovim/wiki/Related-projects#api-clients
>> 
>> Neovim is a fork of Vim which aims to stay backwards compatible with Vim, but
>> at the same time bring the code base to modern standards, add long-wanted
>> features and make the editor easier to extend. They have already done a lot 
>> of
>> work, such asynchronous job control, a built-in terminal emulator, Lua
>> scripting and in particular a remote API.
>> 
>> The remote API allows one to write plugins in any language, provided there 
>> is a
>> client for that language. In contrast, Vim has to be compiled with support 
>> for
>> additional scripting languages and the integration burden was on the Vim
>> developers. This meant that popular languages like Python would be pretty 
>> well
>> supported, but more obscure languages were practically useless because no one
>> would re-compile their Vim just for one plugin. The remote API approach means
>> that Racket integration can be de-coupled from the editor development, and we
>> can write plugins that can make use of Racket libraries. One could for 
>> example
>> implement some of the DrRacket features using DrRacket as a library instead 
>> of
>> re-inventing the wheel. It would also be possible to integrate Neovim inside
>> DrRacket or write a Neovim GUI in Racket (GUIs are just very complex plugins 
>> in
>> Neovim).
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Racket Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to racket-users+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Jay McCarthy]

On Mon, Jul 24, 2017 at 3:18 PM, Alejandro Sanchez
 wrote:
>> - I'm curious of the performance. In particular, I would expect that a
>> computed jump in unpack could do you good. Did you try that?
> I haven’t investigated performance yet. As I said, I am new to Racket, this 
> is my first time doing anything useful in it, my only previous Scheme 
> knowledge was from doing the exercises in SICP and dabbling in Guile a bit. 
> What is a computed jump?

Rather than having a big `cond`, you could look up the function that
does the work in a vector and then call it. IMHO, msgpack was designed
with that in mind, because tags that aren't immediate values are all
nicely ordered. So you'd check the size, subtract a constant, and grab
the appropriate procedure from a constant vector.

>> - Your package collection is 'multi, which is fine, but normally you
>> just do that when you're defining something like data/heap or
>> net/turkeyrpc, where you are extending some existing collection. In
>> particular, you define msgpack and then you also define the test/pack
>> collection (where you might expect it to be tests/msgpack/pack). I
>> recommend having your collection be "msgpack" and putting your tests
>> inside a tests sub-directory.
> Just to make sure I understood correctly: ‘msgpack’ is the umbrella module 
> that users import, ‘msgpack/test/pack’ (and ‘unpack’) are the test modules 
> that will be run for testing only. How about the directory structure? I like 
> to keep all source files in a source directory (my original reason for doing 
> ‘multi), can I still do something like this?
>
> |-README
> |-LICENSE
> |-info.rkt
> |-source
>   |-msgpack.rkt
>   |-pack.rkt
>   |-unpack.rkt
> |-test
>   |-pack.rkt
>   |-pack
> |- ...
>   |-unpack.rkt
>   |-unpack
> |- …
>
> It doesn’t have to be exactly this structure, but the idea is that all 
> project-realted files are in the root, all the source files in the source 
> directory and all the test files in the test directory.

You can do that, but you'd have to have an additional `main.rkt` file
at the top-level that would require the things in `source` then
re-export them. It is not really Racket style to do what you're
talking about, however. If you did do that, then you could call the
`source` directory, `private` and then it would have a Racket-y name,
but your project isn't really large enough to warrant it and those
files aren't actually provide.

Furthermore, your test/pack.rkt and test/unpack.rkt modules aren't
necessary, because you should be testing with `raco test -c msgpack`,
which will just go find everything. There's no need to build such
things yourself. (Although, FWIW, I also wouldn't have separate those
tests into such small files with just one or two, because they are,
again, so small.)

>> - On a style level, I think you should remove your lets and turn your
>> if/begin blocks into conds, for example:
> Good point.
>
>>
>> On Mon, Jul 24, 2017 at 9:17 AM, Alejandro Sanchez
>>  wrote:
>>> Hello dear Racketeers,
>>>
>>> I have been writing an implementation of the MessagePack protocol for Racket
>>> and I think the library is ready to be showcased now:
>>>
>>> https://gitlab.com/HiPhish/MsgPack.rkt
>>>
>>>
>>> ### What is MessagePack? ###
>>>
>>> MessagePack is a binary data serialisation format. The website describes it
>>> "like JSON but fast and small". Unlike JSON the goal is not a format that's
>>> human-readable, but one that can be very quickly serialised, transported and
>>> serialised.
>>>
>>> http://msgpack.org/
>>>
>>>
>>> ### About the Racket implementation ###
>>>
>>> My goal was to keep everything as simple as possible: there are only two
>>> functions: pack and unpack. If there is more than one way of packing an
>>> object
>>> the smallest format is selected automatically. Here is a taste:
>>>
>>>   (require msgpack/pack msgpack/unpack)
>>>   ;;; A wild hodgepodge to pack
>>>   (define vec #(1 2 "hello" '(3 4) '() #t))
>>>   ;;; A byte string of packed data
>>>   (define packed
>>> (call-with-output-bytes (λ (out) (pack vec out
>>>   ;;; Unpack the data again
>>>   (define upacked
>>> (call-with-input-bytes packed (λ (in) (unpack in
>>>
>>>
>>> As you can see, data is packed to and unpacked from a binary port. I think
>>> this
>>> is better than packing/unpacking to binary string because MessagePack is
>>> primarily used for inter-process communication, so there is not much point
>>> in
>>> keeping the packed data inside a process.
>>>
>>> I'd appreciate it a lot if a seasoned Racketeer could take a look at my
>>> code,
>>> in particular if the library is set up properly (the info.rkt files), this
>>> is
>>> my first time doing something in Racket. I am also open to suggestions about
>>> the API, I haven't committed to version 1.0 yet. In particular, I am not
>>> familiar with the modularity conventions of 

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Jack Firth]

> Just to make sure I understood correctly: ‘msgpack’ is the umbrella module 
> that users import, ‘msgpack/test/pack’ (and ‘unpack’) are the test modules 
> that will be run for testing only. How about the directory structure? I like 
> to keep all source files in a source directory (my original reason for doing 
> ‘multi), can I still do something like this?
> 
> |-README
> |-LICENSE
> |-info.rkt
> |-source
>   |-msgpack.rkt
>   |-pack.rkt
>   |-unpack.rkt
> |-test
>   |-pack.rkt
>   |-pack
> |- ...
>   |-unpack.rkt
>   |-unpack
> |- …
> 
> It doesn’t have to be exactly this structure, but the idea is that all 
> project-realted files are in the root, all the source files in the source 
> directory and all the test files in the test directory.

That you cannot do, and if you wish to do that keeping your code as one package 
might be a little unidiomatic. If you want to keep your test code completely 
separate from your implementation code, and you want both to be separate from 
the top level root of the project, you could have two separate packages each 
with a single collection like so:

|-README
|-LICENSE
|-msgpack-lib
  |-info.rkt ;; collection is "msgpack"
  |-main.rkt
  |-pack.rkt
  |-unpack.rkt
|-msgpack-test
  |-info.rkt ;; collection is "msgpack"
  |-pack-test.rkt
  |-unpack-test.rkt

Having said that, have you considered test submodules? They allow you to write 
your tests in the same file as the code they're testing, while keeping test 
dependencies separate from your library's normal runtime dependencies. With 
test submodules, your code would probably look like this:

(define (pack ...) ...)

(module+ test
  test pack ...)

(define (unpack ...) ...)

(module+ test
  test unpack ...)

And your directory structure would look like this:

|-README
|-LICENSE
|-info.rkt ;; collection is "msgpack"
|-main.rkt
|-pack.rkt ;; has test submodule
|-unpack.rkt ;; has test submodule

You could also put the code into a subdirectory package like above, if you 
really want to keep the project files and the source files separate.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Vincent St-Amour]

Hi Alejandro,

This looks cool!

I don't see it listed at pkgs.racket-lang.org. It would be easier for
users to discover it if you posted it there.

Vincent



On Mon, 24 Jul 2017 08:17:30 -0500,
Alejandro Sanchez wrote:
> 
> Hello dear Racketeers,
> 
> I have been writing an implementation of the MessagePack protocol for Racket
> and I think the library is ready to be showcased now:
> 
> https://gitlab.com/HiPhish/MsgPack.rkt
> 
> ### What is MessagePack? ###
> 
> MessagePack is a binary data serialisation format. The website describes it
> "like JSON but fast and small". Unlike JSON the goal is not a format that's
> human-readable, but one that can be very quickly serialised, transported and
> serialised.
> 
> http://msgpack.org/
> 
> ### About the Racket implementation ###
> 
> My goal was to keep everything as simple as possible: there are only two
> functions: pack and unpack. If there is more than one way of packing an object
> the smallest format is selected automatically. Here is a taste:
> 
> (require msgpack/pack msgpack/unpack)
> ;;; A wild hodgepodge to pack
> (define vec #(1 2 "hello" '(3 4) '() #t))
> ;;; A byte string of packed data
> (define packed
> (call-with-output-bytes (λ (out) (pack vec out
> ;;; Unpack the data again
> (define upacked
> (call-with-input-bytes packed (λ (in) (unpack in
> 
> As you can see, data is packed to and unpacked from a binary port. I think 
> this
> is better than packing/unpacking to binary string because MessagePack is
> primarily used for inter-process communication, so there is not much point in
> keeping the packed data inside a process.
> 
> I'd appreciate it a lot if a seasoned Racketeer could take a look at my code,
> in particular if the library is set up properly (the info.rkt files), this is
> my first time doing something in Racket. I am also open to suggestions about
> the API, I haven't committed to version 1.0 yet. In particular, I am not
> familiar with the modularity conventions of Racket libraries, i.e. if it is OK
> to have 'msgpack/pack' and 'msgpack/unpack' or if everything should be covered
> by one large 'provide' from 'msgpack'? There is one new type 'ext' declared,
> should that be part of 'msgpack' or should I move it to 'msgpack/types'
> instead?
> 
> On a related note, I find it really annoying that 'integer->integer-bytes' and
> 'integer-bytes->integer' do not support 8-bit integers. Is there a reason for
> that? I had to write all sorts of ugly extra code for the 8-bit cases. I 
> opened
> an issue on GitHub about it (#1754).
> 
> ### What's next? ###
> 
> Once the API settles I would like to move the library to typed Racket. I would
> also like to submit it to the Racket packages catalog. The reason I wrote this
> library is because I want to eventually write a Racket API client for Neovim:
> 
> https://github.com/neovim/neovim
> https://github.com/neovim/neovim/wiki/Related-projects#api-clients
> 
> Neovim is a fork of Vim which aims to stay backwards compatible with Vim, but
> at the same time bring the code base to modern standards, add long-wanted
> features and make the editor easier to extend. They have already done a lot of
> work, such asynchronous job control, a built-in terminal emulator, Lua
> scripting and in particular a remote API.
> 
> The remote API allows one to write plugins in any language, provided there is 
> a
> client for that language. In contrast, Vim has to be compiled with support for
> additional scripting languages and the integration burden was on the Vim
> developers. This meant that popular languages like Python would be pretty well
> supported, but more obscure languages were practically useless because no one
> would re-compile their Vim just for one plugin. The remote API approach means
> that Racket integration can be de-coupled from the editor development, and we
> can write plugins that can make use of Racket libraries. One could for example
> implement some of the DrRacket features using DrRacket as a library instead of
> re-inventing the wheel. It would also be possible to integrate Neovim inside
> DrRacket or write a Neovim GUI in Racket (GUIs are just very complex plugins 
> in
> Neovim).
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to racket-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

> - I think you should have one module, `msgpack` that exports everything
OK. I my defense, I was originally planning to have a number of ‘pack-*’ 
methods like ‘pack-uin8’, ‘pack-uint16’ and so on, but there was nothing to be 
gained but a bloated interface.

> - You should rewrite your modules to use the `#lang racket/base`
> language so they don't force users to import so much stuff
Good idea, I’ll do that.

> - On integer->integer-bytes, I think that we should support 8-bit
> integers, but in the meantime, I think you should write your own
> version (integer->integer-bytes*) rather than having the handling in
> many places in the code.
Yes, I’ll do that eventually, I was just exhausted from getting it working and 
presentable first.

> - I'm curious of the performance. In particular, I would expect that a
> computed jump in unpack could do you good. Did you try that?
I haven’t investigated performance yet. As I said, I am new to Racket, this is 
my first time doing anything useful in it, my only previous Scheme knowledge 
was from doing the exercises in SICP and dabbling in Guile a bit. What is a 
computed jump?

> - Your package collection is 'multi, which is fine, but normally you
> just do that when you're defining something like data/heap or
> net/turkeyrpc, where you are extending some existing collection. In
> particular, you define msgpack and then you also define the test/pack
> collection (where you might expect it to be tests/msgpack/pack). I
> recommend having your collection be "msgpack" and putting your tests
> inside a tests sub-directory.
Just to make sure I understood correctly: ‘msgpack’ is the umbrella module that 
users import, ‘msgpack/test/pack’ (and ‘unpack’) are the test modules that will 
be run for testing only. How about the directory structure? I like to keep all 
source files in a source directory (my original reason for doing ‘multi), can I 
still do something like this?

|-README
|-LICENSE
|-info.rkt
|-source
  |-msgpack.rkt
  |-pack.rkt
  |-unpack.rkt
|-test
  |-pack.rkt
  |-pack
|- ...
  |-unpack.rkt
  |-unpack
|- …

It doesn’t have to be exactly this structure, but the idea is that all 
project-realted files are in the root, all the source files in the source 
directory and all the test files in the test directory.

> - On a style level, I think you should remove your lets and turn your
> if/begin blocks into conds, for example:
Good point.

> 
> On Mon, Jul 24, 2017 at 9:17 AM, Alejandro Sanchez
>  wrote:
>> Hello dear Racketeers,
>> 
>> I have been writing an implementation of the MessagePack protocol for Racket
>> and I think the library is ready to be showcased now:
>> 
>> https://gitlab.com/HiPhish/MsgPack.rkt
>> 
>> 
>> ### What is MessagePack? ###
>> 
>> MessagePack is a binary data serialisation format. The website describes it
>> "like JSON but fast and small". Unlike JSON the goal is not a format that's
>> human-readable, but one that can be very quickly serialised, transported and
>> serialised.
>> 
>> http://msgpack.org/
>> 
>> 
>> ### About the Racket implementation ###
>> 
>> My goal was to keep everything as simple as possible: there are only two
>> functions: pack and unpack. If there is more than one way of packing an
>> object
>> the smallest format is selected automatically. Here is a taste:
>> 
>>   (require msgpack/pack msgpack/unpack)
>>   ;;; A wild hodgepodge to pack
>>   (define vec #(1 2 "hello" '(3 4) '() #t))
>>   ;;; A byte string of packed data
>>   (define packed
>> (call-with-output-bytes (λ (out) (pack vec out
>>   ;;; Unpack the data again
>>   (define upacked
>> (call-with-input-bytes packed (λ (in) (unpack in
>> 
>> 
>> As you can see, data is packed to and unpacked from a binary port. I think
>> this
>> is better than packing/unpacking to binary string because MessagePack is
>> primarily used for inter-process communication, so there is not much point
>> in
>> keeping the packed data inside a process.
>> 
>> I'd appreciate it a lot if a seasoned Racketeer could take a look at my
>> code,
>> in particular if the library is set up properly (the info.rkt files), this
>> is
>> my first time doing something in Racket. I am also open to suggestions about
>> the API, I haven't committed to version 1.0 yet. In particular, I am not
>> familiar with the modularity conventions of Racket libraries, i.e. if it is
>> OK
>> to have 'msgpack/pack' and 'msgpack/unpack' or if everything should be
>> covered
>> by one large 'provide' from 'msgpack'? There is one new type 'ext' declared,
>> should that be part of 'msgpack' or should I move it to 'msgpack/types'
>> instead?
>> 
>> On a related note, I find it really annoying that 'integer->integer-bytes'
>> and
>> 'integer-bytes->integer' do not support 8-bit integers. Is there a reason
>> for
>> that? I had to write all sorts of ugly extra code for the 8-bit cases. I
>> opened
>> an issue on GitHub 

Re: [racket-users] [ANN] MessagePack implementation for Racke

[Jay McCarthy]

This is awesome, thank you!

Some advice:
- I think you should have one module, `msgpack` that exports everything
- You should rewrite your modules to use the `#lang racket/base`
language so they don't force users to import so much stuff
- On integer->integer-bytes, I think that we should support 8-bit
integers, but in the meantime, I think you should write your own
version (integer->integer-bytes*) rather than having the handling in
many places in the code.
- I'm curious of the performance. In particular, I would expect that a
computed jump in unpack could do you good. Did you try that?
- Your package collection is 'multi, which is fine, but normally you
just do that when you're defining something like data/heap or
net/turkeyrpc, where you are extending some existing collection. In
particular, you define msgpack and then you also define the test/pack
collection (where you might expect it to be tests/msgpack/pack). I
recommend having your collection be "msgpack" and putting your tests
inside a tests sub-directory.
- On a style level, I think you should remove your lets and turn your
if/begin blocks into conds, for example:

You have

```
(define (pack-map m out)
  (let ([len (hash-count m)])
(if (<= len #b)
  (write-byte (bitwise-ior len #b1000) out)
  (begin
(cond
  [(uint16? len) (write-byte #xDE out)]
  [(uint32? len) (write-byte #xDF out)]
  [else (error "An map may contain at most 2^32 - 1 items")])
(write-bytes (integer->bytes len #f) out)))
(for ([(key value) (in-hash m)])
  (pack key   out)
  (pack value out
```

But I would write:

```
(define (pack-map m out)
  (define len (hash-count m))
  (cond
[(<= len #b)
 (write-byte (bitwise-ior len #b1000) out)]
[else
 (cond
   [(uint16? len) (write-byte #xDE out)]
   [(uint32? len) (write-byte #xDF out)]
   [else (error "An map may contain at most 2^32 - 1 items")])
 (write-bytes (integer->bytes len #f) out)])
  (for ([(key value) (in-hash m)])
(pack key   out)
(pack value out)))
```

On Mon, Jul 24, 2017 at 9:17 AM, Alejandro Sanchez
 wrote:
> Hello dear Racketeers,
>
> I have been writing an implementation of the MessagePack protocol for Racket
> and I think the library is ready to be showcased now:
>
> https://gitlab.com/HiPhish/MsgPack.rkt
>
>
> ### What is MessagePack? ###
>
> MessagePack is a binary data serialisation format. The website describes it
> "like JSON but fast and small". Unlike JSON the goal is not a format that's
> human-readable, but one that can be very quickly serialised, transported and
> serialised.
>
> http://msgpack.org/
>
>
> ### About the Racket implementation ###
>
> My goal was to keep everything as simple as possible: there are only two
> functions: pack and unpack. If there is more than one way of packing an
> object
> the smallest format is selected automatically. Here is a taste:
>
>(require msgpack/pack msgpack/unpack)
>;;; A wild hodgepodge to pack
>(define vec #(1 2 "hello" '(3 4) '() #t))
>;;; A byte string of packed data
>(define packed
>  (call-with-output-bytes (λ (out) (pack vec out
>;;; Unpack the data again
>(define upacked
>  (call-with-input-bytes packed (λ (in) (unpack in
>
>
> As you can see, data is packed to and unpacked from a binary port. I think
> this
> is better than packing/unpacking to binary string because MessagePack is
> primarily used for inter-process communication, so there is not much point
> in
> keeping the packed data inside a process.
>
> I'd appreciate it a lot if a seasoned Racketeer could take a look at my
> code,
> in particular if the library is set up properly (the info.rkt files), this
> is
> my first time doing something in Racket. I am also open to suggestions about
> the API, I haven't committed to version 1.0 yet. In particular, I am not
> familiar with the modularity conventions of Racket libraries, i.e. if it is
> OK
> to have 'msgpack/pack' and 'msgpack/unpack' or if everything should be
> covered
> by one large 'provide' from 'msgpack'? There is one new type 'ext' declared,
> should that be part of 'msgpack' or should I move it to 'msgpack/types'
> instead?
>
> On a related note, I find it really annoying that 'integer->integer-bytes'
> and
> 'integer-bytes->integer' do not support 8-bit integers. Is there a reason
> for
> that? I had to write all sorts of ugly extra code for the 8-bit cases. I
> opened
> an issue on GitHub about it (#1754).
>
>
> ### What's next? ###
>
> Once the API settles I would like to move the library to typed Racket. I
> would
> also like to submit it to the Racket packages catalog. The reason I wrote
> this
> library is because I want to eventually write a Racket API client for
> Neovim:
>
> https://github.com/neovim/neovim
> https://github.com/neovim/neovim/wiki/Related-projects#api-clients
>
> Neovim is a fork of Vim which aims to stay backwards 

[racket-users] [ANN] MessagePack implementation for Racke

[Alejandro Sanchez]

Hello dear Racketeers,

I have been writing an implementation of the MessagePack protocol for Racket
and I think the library is ready to be showcased now:

https://gitlab.com/HiPhish/MsgPack.rkt 


### What is MessagePack? ###

MessagePack is a binary data serialisation format. The website describes it
"like JSON but fast and small". Unlike JSON the goal is not a format that's
human-readable, but one that can be very quickly serialised, transported and
serialised.

http://msgpack.org/ 


### About the Racket implementation ###

My goal was to keep everything as simple as possible: there are only two
functions: pack and unpack. If there is more than one way of packing an object
the smallest format is selected automatically. Here is a taste:

   (require msgpack/pack msgpack/unpack)
   ;;; A wild hodgepodge to pack
   (define vec #(1 2 "hello" '(3 4) '() #t))
   ;;; A byte string of packed data
   (define packed
 (call-with-output-bytes (λ (out) (pack vec out
   ;;; Unpack the data again
   (define upacked
 (call-with-input-bytes packed (λ (in) (unpack in


As you can see, data is packed to and unpacked from a binary port. I think this
is better than packing/unpacking to binary string because MessagePack is
primarily used for inter-process communication, so there is not much point in
keeping the packed data inside a process.

I'd appreciate it a lot if a seasoned Racketeer could take a look at my code,
in particular if the library is set up properly (the info.rkt files), this is
my first time doing something in Racket. I am also open to suggestions about
the API, I haven't committed to version 1.0 yet. In particular, I am not
familiar with the modularity conventions of Racket libraries, i.e. if it is OK
to have 'msgpack/pack' and 'msgpack/unpack' or if everything should be covered
by one large 'provide' from 'msgpack'? There is one new type 'ext' declared,
should that be part of 'msgpack' or should I move it to 'msgpack/types'
instead?

On a related note, I find it really annoying that 'integer->integer-bytes' and
'integer-bytes->integer' do not support 8-bit integers. Is there a reason for
that? I had to write all sorts of ugly extra code for the 8-bit cases. I opened
an issue on GitHub about it (#1754).


### What's next? ###

Once the API settles I would like to move the library to typed Racket. I would
also like to submit it to the Racket packages catalog. The reason I wrote this
library is because I want to eventually write a Racket API client for Neovim:

https://github.com/neovim/neovim 
https://github.com/neovim/neovim/wiki/Related-projects#api-clients 


Neovim is a fork of Vim which aims to stay backwards compatible with Vim, but
at the same time bring the code base to modern standards, add long-wanted
features and make the editor easier to extend. They have already done a lot of
work, such asynchronous job control, a built-in terminal emulator, Lua
scripting and in particular a remote API.

The remote API allows one to write plugins in any language, provided there is a
client for that language. In contrast, Vim has to be compiled with support for
additional scripting languages and the integration burden was on the Vim
developers. This meant that popular languages like Python would be pretty well
supported, but more obscure languages were practically useless because no one
would re-compile their Vim just for one plugin. The remote API approach means
that Racket integration can be de-coupled from the editor development, and we
can write plugins that can make use of Racket libraries. One could for example
implement some of the DrRacket features using DrRacket as a library instead of
re-inventing the wheel. It would also be possible to integrate Neovim inside
DrRacket or write a Neovim GUI in Racket (GUIs are just very complex plugins in
Neovim).

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.