I've looked closely at utf-8 encoders/decoders in JS vs TextEncoder, and I'm not sure fast string encoding/decoding alone is going to bring us the kind of speed improvements we need to make binary formats a clear win over JSON. (Some of this might be because the ArrayBuffer implementations are slower than they could be.)
Relevant JSPerfs (bops is the library used by messagepack-js): http://jsperf.com/utf8-encoding-methods/2 http://jsperf.com/utf8-decoding-methods/2 Source code for the encoding/decoding methods in the jsperfs: https://github.com/favila/utfate That said this is still exciting. On Monday, August 4, 2014 9:30:29 AM UTC-5, David Nolen wrote: > Peter, > > > > Cool let us know if you pursue this and have any numbers to share. It > > may be the tipping point for going for binary formats like > > Fressian/MessagePack over JSON representations. > > > > David > > > > On Mon, Aug 4, 2014 at 10:26 AM, Peter Schuck <[email protected]> wrote: > > > That would definitely help, a large percentage of the conversion time is > > > manually converting strings to utf-8 bytes and back again. Thanks for > > > pointing that out. > > > > > > Peter Schuck > > > > > > > > > On Sun, Aug 3, 2014 at 9:14 AM, David Nolen <[email protected]> wrote: > > >> > > >> RE: writing/reading strings I wonder if this won't help > > >> https://plus.google.com/+AddyOsmani/posts/4GgX9CD6c1X? > > >> > > >> David > > >> > > >> On Sun, Aug 3, 2014 at 9:51 AM, Peter Schuck <[email protected]> wrote: > > >> > Fressian's caching strategy is very flexible and powerful but doesn't > > >> > have a > > >> > lot of documetnation. I view Fressian's caching as similar to memory > > >> > references for Clojure values. You can cache any value no matter how > > >> > large > > >> > and once the value is cached referencing it again is only a few bytes. > > >> > Fressian caches symbols, keywords, and tagged value tags by default. > > >> > > > >> > What I mean by "significant structural similarity" is a data structure > > >> > or > > >> > steam of data structures that have frequent repetitions of compound > > >> > values. > > >> > An AST would probably be a canonical example of this. > > >> > > > >> > Fressian is about five to ten times as slow as Transit and not really > > >> > suited > > >> > for the vast majority of applications where relatively small data > > >> > structures > > >> > need to be written and read as quickly as possible. It took significant > > >> > effort to get Fressian's speed to where it is now and I don't see a way > > >> > forward to get it to match Transit's speed. The main bottlenecks are > > >> > reading and writing numbers and strings, hashing value, and determining > > >> > an > > >> > value's type. Until those bottlenecks get resolved in future versions > > >> > of > > >> > JavaScript piggybacking of of JSON, like Transit, is the way to go for > > >> > speed. > > >> > > > >> > I hope that answers most questions, feel free to ask for more > > >> > clarification. > > >> > I learned a lot about how to write fast ClojureScript / JavaScript and > > >> > plan > > >> > on writing a blog post about it in the near future. > > >> > > > >> > Peter Schuck > > >> > > > >> > > > >> > On Sat, Aug 2, 2014 at 4:23 PM, Alex Miller <[email protected]> wrote: > > >> >> > > >> >> I took "significant structural similarity" to primarily mean at least > > >> >> maps > > >> >> with similar keys, which Transit caching will cover. > > >> >> > > >> >> > > >> >> On Sat, Aug 2, 2014 at 1:49 PM, David Nolen <[email protected]> > > >> >> wrote: > > >> >>> > > >> >>> In Transit, maps keys, symbols, keywords, and tagged value tags are > > >> >>> subject to caching. > > >> >>> > > >> >>> Fressian's caching strategy is far more flexible from what I > > >> >>> understand. That said transit-cljs is 20-30X faster than > > >> >>> cljs.reader/read-string on the benchmarks I've tried across various > > >> >>> browser and command line JS environments. > > >> >>> > > >> >>> David > > >> >>> > > >> >>> On Sat, Aug 2, 2014 at 2:26 PM, Sean Grove <[email protected]> wrote: > > >> >>> > I thought transit's caching only applied to map keys? Pretty unclear > > >> >>> > on > > >> >>> > what > > >> >>> > Fressian's can do > > >> >>> > > > >> >>> > > > >> >>> > On Sat, Aug 2, 2014 at 8:23 AM, Alex Miller <[email protected]> > > >> >>> > wrote: > > >> >>> >> > > >> >>> >> Cool stuff Peter. It would be interesting to compare performance > > >> >>> >> with > > >> >>> >> transit-cljs https://github.com/cognitect/transit-cljs. Transit has > > >> >>> >> the same > > >> >>> >> caching and extensibility benefits of Fressian but leverages the > > >> >>> >> very > > >> >>> >> fast > > >> >>> >> JavaScript parser capabilities built into the browser, so is likely > > >> >>> >> faster. > > >> >>> >> > > >> >>> >> Alex > > >> >>> >> > > >> >>> >> On Friday, August 1, 2014 3:33:58 PM UTC-5, [email protected] > > >> >>> >> wrote: > > >> >>> >> > I'm happy to announce a port of Fressian to ClojureScript. > > >> >>> >> > > > >> >>> >> > The public api mirrors data.fressian api with a few exceptions. > > >> >>> >> > Records > > >> >>> >> > don't have a generic writer but you can easily append the > > >> >>> >> > handlers > > >> >>> >> > with the > > >> >>> >> > write-record function. There is no inheritance lookup for types > > >> >>> >> > so > > >> >>> >> > every > > >> >>> >> > different type will need it's own handler. The tagged helper > > >> >>> >> > functions > > >> >>> >> > (tagged-object?, tag, tagged-value) are not included. > > >> >>> >> > > > >> >>> >> > I see the use case for Fressian in ClojureScript streaming large > > >> >>> >> > amounts > > >> >>> >> > of data that has significant structural similarity. Fressian > > >> >>> >> > caching > > >> >>> >> > capabilities allow large values to be represented as a single > > >> >>> >> > integer in the > > >> >>> >> > bytestream. > > >> >>> >> > > > >> >>> >> > This is my first large ClojureScript library so any suggestions > > >> >>> >> > for > > >> >>> >> > improvements are welcome. > > >> >>> >> > > > >> >>> >> > Thanks to Bendyworks for letting me develop this during my work > > >> >>> >> > hours. > > >> >>> >> > > > >> >>> >> > Clojars Link: https://clojars.org/longshi > > >> >>> >> > Repo: https://github.com/spinningtopsofdoom/longshi > > >> >>> >> > > > >> >>> >> > Peter Schuck > > >> >>> >> > > >> >>> >> -- > > >> >>> >> Note that posts from new members are moderated - please be patient > > >> >>> >> with > > >> >>> >> your first post. > > >> >>> >> --- > > >> >>> >> You received this message because you are subscribed to the Google > > >> >>> >> Groups > > >> >>> >> "ClojureScript" group. > > >> >>> >> To unsubscribe from this group and stop receiving emails from it, > > >> >>> >> send > > >> >>> >> an > > >> >>> >> email to [email protected]. > > >> >>> >> To post to this group, send email to > > >> >>> >> [email protected]. > > >> >>> >> Visit this group at http://groups.google.com/group/clojurescript. > > >> >>> > > > >> >>> > > > >> >>> > -- > > >> >>> > Note that posts from new members are moderated - please be patient > > >> >>> > with > > >> >>> > your > > >> >>> > first post. > > >> >>> > --- > > >> >>> > You received this message because you are subscribed to the Google > > >> >>> > Groups > > >> >>> > "ClojureScript" group. > > >> >>> > To unsubscribe from this group and stop receiving emails from it, > > >> >>> > send > > >> >>> > an > > >> >>> > email to [email protected]. > > >> >>> > To post to this group, send email to [email protected]. > > >> >>> > Visit this group at http://groups.google.com/group/clojurescript. > > >> >>> > > >> >>> -- > > >> >>> Note that posts from new members are moderated - please be patient > > >> >>> with > > >> >>> your first post. > > >> >>> --- > > >> >>> You received this message because you are subscribed to a topic in the > > >> >>> Google Groups "ClojureScript" group. > > >> >>> To unsubscribe from this topic, visit > > >> >>> > > >> >>> https://groups.google.com/d/topic/clojurescript/xhdrGunEXPE/unsubscribe. > > >> >>> To unsubscribe from this group and all its topics, send an email to > > >> >>> [email protected]. > > >> >>> To post to this group, send email to [email protected]. > > >> >>> Visit this group at http://groups.google.com/group/clojurescript. > > >> >> > > >> >> > > >> >> -- > > >> >> Note that posts from new members are moderated - please be patient with > > >> >> your first post. > > >> >> --- > > >> >> You received this message because you are subscribed to a topic in the > > >> >> Google Groups "ClojureScript" group. > > >> >> To unsubscribe from this topic, visit > > >> >> > > >> >> https://groups.google.com/d/topic/clojurescript/xhdrGunEXPE/unsubscribe. > > >> >> To unsubscribe from this group and all its topics, send an email to > > >> >> [email protected]. > > >> >> To post to this group, send email to [email protected]. > > >> >> Visit this group at http://groups.google.com/group/clojurescript. > > >> > > > >> > > > >> > -- > > >> > Note that posts from new members are moderated - please be patient with > > >> > your > > >> > first post. > > >> > --- > > >> > You received this message because you are subscribed to the Google > > >> > Groups > > >> > "ClojureScript" group. > > >> > To unsubscribe from this group and stop receiving emails from it, send > > >> > an > > >> > email to [email protected]. > > >> > To post to this group, send email to [email protected]. > > >> > Visit this group at http://groups.google.com/group/clojurescript. > > >> > > >> -- > > >> Note that posts from new members are moderated - please be patient with > > >> your first post. > > >> --- > > >> You received this message because you are subscribed to a topic in the > > >> Google Groups "ClojureScript" group. > > >> To unsubscribe from this topic, visit > > >> https://groups.google.com/d/topic/clojurescript/xhdrGunEXPE/unsubscribe. > > >> To unsubscribe from this group and all its topics, send an email to > > >> [email protected]. > > >> To post to this group, send email to [email protected]. > > >> Visit this group at http://groups.google.com/group/clojurescript. > > > > > > > > > -- > > > Note that posts from new members are moderated - please be patient with your > > > first post. > > > --- > > > You received this message because you are subscribed to the Google Groups > > > "ClojureScript" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [email protected]. > > > To post to this group, send email to [email protected]. > > > Visit this group at http://groups.google.com/group/clojurescript. -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/clojurescript.
