Thanks for this breakdown; a lot of what you're saying about Transit is stuff I had inferred from prior announcements, but it's still enlightening to see an explicit comparison to Edn and Fressian.
On Sunday, March 15, 2015 at 7:51:55 PM UTC-7, Alex Miller wrote: > > Hi Ryan, > > To answer the big question, all of these data formats are in active use at > Cognitect and while I make no promises, I expect them all to be alive and > active for the knowable future. Each of them targets a different niche but > all of them share the qualities of transmitting extensible typed values. > > edn is the best choice for human-readable data. It is however, less > efficient to transmit and depends on writing a high-performance parser - > this is a high bar in some language environments. edn is most attractive > right now to Clojure users b/c of its proximity to Clojure itself. While it > has many advantages as an extensible readable literal data format, it's an > uphill battle to sell that against other data formats that already have > greater mindshare and tooling in other language communities. > > fressian is the highest performance option - it takes full advantage of a > number of compression tricks and has support for arbitrary user-extensible > caching. Again, it requires a fair amount of effort to write a fressian > library though so it's probably best for JVM-oriented endpoints right now. > By seeking greatest performance, fressian also makes tradeoffs that narrow > its range of use and interest group. > > transit is a pragmatic midpoint between these two. It focuses, like > fressian, on program-to-program data transmission however, the data can be > made readable (like edn) via the json-verbose mode. Like fressian, transit > contains caching capabilities but they are more limited and not > user-extensible. transit is designed primarily to have the most > high-quality implementations per lowest effort - effectively shooting for > greater reach than either edn or fressian by lower the bar to > implementation. The bar is lowered by reusing high-performance parser for > either JSON or messagepack which exist in a large number of languages. Of > particular importance is leveraging the very high performance JSON parsers > available in JavaScript runtimes, making transit viable as a browser-side > endpoint for a fraction of the effort required to write a high performance > edn or fressian endpoint. As transit explicitly seeks reach and > portability, it is naturally the format with the broadest potential usage. > > Hopefully that lays out the landscape a bit more. With respect to support, > all of these formats are supported by Cognitect. Like everyone else, we're > managing priorities across a large number of projects. Filing issues is > great, voting on issues is great, reports like this are great. External > tooling is welcome (like pretty printers, etc). Because we use these > projects inside internal Cognitect projects and products, we do not > currently have a community contribution model for most of these as there > may be impacts that are not publicly visible. That could change in the > future. > > I am not an expert on everything you mention below but at a glance it > looks like on-point feedback and I will raise it with the appropriate > people and we can follow back here or on the relevant issues as needed. > > Thanks, > Alex > > On Sunday, March 15, 2015 at 8:40:21 PM UTC-5, Ryan Schmitt wrote: >> >> I'm the author of dynamic-object >> <https://github.com/rschmitt/dynamic-object>, an open source library >> that makes Clojure's data modeling power available to Java programmers. >> This includes features like serialization and deserialization. I'll copy >> this small usage example from the README to give you a sense of how it >> works: >> >> public interface Album extends DynamicObject<Album> { >> @Key(":artist") String getArtist(); >> @Key(":album") String getAlbum(); >> @Key(":tracks") int getTracks(); >> @Key(":year") int getYear(); >> } >> >> >> String edn = "{:artist \"Meshuggah\", :album \"Chaosphere\", :tracks 8, >> :year 1998}";Album album = DynamicObject.deserialize(edn, Album.class); >> album.getArtist(); // => "Meshuggah" >> >> >> dynamic-object has always been opinionated about using Edn as the primary >> data language on the wire, for a number of reasons. For a long time, I also >> thought about adding Fressian support to dynamic-object, and I've recently >> done so on an experimental basis. (It looks like this >> <https://github.com/rschmitt/dynamic-object/blob/440ecd2f385d1cec80496ba7007bd0755023b19c/src/test/java/com/github/rschmitt/dynamicobject/FressianTest.java>.) >> >> Some time after I initially released dynamic-object, Transit was also >> released, with support for various encodings (JSON, JSON-Verbose, >> MessagePack). >> >> In working (to different extents) with these data languages, I've had >> some apprehensions about all of them. >> >> - There is a lack of tooling available for Edn, such as validators >> and pretty-printers. I spent a while looking for an Edn equivalent of >> python -mjson.tool and never found one. Clojure's built-in pprint >> function >> does not work out-of-the-box to pretty print arbitrary values, and also >> appears to handle some data structures, such as records, incorrectly. >> (pprint omits reader tags when printing records.) pprint's underlying >> implementation, cl-format, is extremely powerful and could almost >> certainly >> be used to build a validating Edn pretty-printer, but it would have an >> unacceptably long startup time. >> - There is a lack of high-quality Edn implementations for different >> languages. Because the Edn spec is not very formal or complete, there >> seems >> to be some uncertainty regarding what constitutes an Edn implementation >> in >> the first place. For instance, clojure.edn parses the Ratio type as a >> builtin, even though it is mentioned nowhere in the spec. (Issue >> <https://github.com/edn-format/edn/issues/64>.) There are also >> oddities such as the recommended C++ implementation >> <https://github.com/shaunxcode/edn-cpp> describing itself as >> "experimental." >> - Fressian's reference Java implementation is almost totally >> undocumented. This is a problem, because I'm writing a library that >> targets >> Java developers; they won't be going through the Clojure bindings (which >> are decently documented). Fressian's source code is outstanding, but it's >> still not documentation. >> - Due to the lack of documentation, it's not clear which parts of >> Fressian are actually stable. Stuart Halloway's data.fressian talk >> included >> some parentheticals about the extension points being subject to change, >> which so far they haven't, but that might only be because of the >> following >> point... >> - Fressian does not seem to have gotten any attention since the >> initial launch. People have submitted GitHub issues >> <https://github.com/Datomic/fressian/issues>, including one >> surprisingly high-quality bug report, but they have all been ignored. The >> JIRA is mostly tumbleweeds. >> - The Clojure bindings for Fressian, namely data.fressian, are >> essentially incomplete. With the exception of maps, Clojure collection >> types do not round trip, and in at least one case (vectors) that is >> because >> of a blocking issue <http://dev.clojure.org/jira/browse/DFRS-1> in >> the underlying Fressian implementation. >> - There are no documented best practices for the use of Fressian or >> some of its more advanced features like chunking. It is not clear how to >> read and write Fressian in a way that facilitates (for instance) ranged >> reads from the middle of a resource. It is not clear when checksums >> should >> be used and how they should be validated. It is not clear whether tags >> should be namespaced, or how. The only namespaced tag in data.fressian is >> for IRecord; none of the other type tags are namespaced. It's not clear >> whether this is due to bugwards compatibility. >> - Transit is advertised >> <https://github.com/cognitect/transit-format#implementations> as a >> work-in-progress. This is the main reason I haven't seriously considered >> adding Transit support to dynamic-object. >> - However, what happens when Transit is stabilized (if that ever >> happens)? Since Transit offers a msgpack encoding, will Fressian then be >> irrelevant (except for legacy use cases)? There's a FUD aspect here--I >> like >> Fressian and I want dynamic-object to support it, but I don't want to >> back >> the wrong pony and end up having to support HD-DVD and Betamax for all >> time >> (so to speak). >> - Can these formats be unified? Can Edn and Fressian encodings for >> Transit be offered? Would that even accomplish anything? >> >> I realize that none of these data languages will have the same extent of >> support and tooling as JSON or XML, but I want to ensure that >> dynamic-object's supported data languages all have attentive stewardship >> and bright futures. It's distressing that a lot of the issues with Edn and >> Fressian have not gotten much traction. Are these languages still actively >> being supported and fostered? If so, how much development activity is >> taking place on internal forks? Are any public updates planned for these >> languages any time soon? >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
