Worth considering, but my understanding is that protocol buffers do not directly embed their type information. So while there is a schema corresponding to the buffer, the application needs to know which. Presumably one would need some sort of IDL on top of protobufs to handle that.
There's also this one: https://capnproto.org/. It has a very rich schema/IDL language (including inheritance), and the RPC protocol has full support for lazy evaluation (pipeline promises), and supports incremental reads, with mmap, etc. Just seems a little out there. By the guy who designed protobufs v2. On Mon, Apr 6, 2015 at 5:08 PM, Dan Tenenbaum <dtene...@fredhutch.org> wrote: > > > ----- Original Message ----- > > From: "Michael Lawrence" <lawrence.mich...@gene.com> > > To: "Tim Triche, Jr." <tim.tri...@gmail.com> > > Cc: "Michael Lawrence" <lawrence.mich...@gene.com>, > bioc-devel@r-project.org, "Paul Shannon" <pshan...@fredhutch.org> > > Sent: Monday, April 6, 2015 4:48:41 PM > > Subject: Re: [Bioc-devel] BrowserViz and sub-protocols > > > > On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr. > > <tim.tri...@gmail.com> > > wrote: > > > > > Kitware seems to have had good experiences with WAMP, which is a > > > good > > > sign, given the complexity of their typical visualization projects > > > (stemming from old HPC projects in fluid dynamics, astrophysics, > > > realtime > > > anatomical reconstruction, etc). Thus WAMP seems like a great way > > > to > > > leverage the experiences of an open source focused, medium sized > > > shop known > > > for doing highly technical work with some similar aims to Paul's > > > pkg. > > > > > > Avro's IDL becomes onerous for large projects rather fast. The > > > lack of > > > inheritance can be excruciating when it is pushed beyond its > > > typical scope. > > > Best not to poke that sleeping dog ;-) > > > > > > > > Hmm, thanks for pointing that out. Is there no good solution for > > serializing objects, outside of simple lists and maps? > > > > Protocol buffers? > > https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket > > Dan > > > > > > > --t > > > > > > > On Apr 6, 2015, at 9:35 AM, Michael Lawrence > > > > <lawrence.mich...@gene.com> > > > wrote: > > > > > > > > Sounds like a good plan. It does seem like there is a trend away > > > > from > > > type > > > > formalism in application development. Everything is essentially > > > > JSON. > > > Have > > > > you checked out WAMP? It seems very similar to your solution; it > > > > might > > > pay > > > > off to stick with standards, even if it takes more up-front > > > > investment. > > > > It's JSON-based, so it doesn't solve the high-level type problem. > > > > Very > > > few > > > > serialization formats do. Avro has a nice IDL, but that's out of > > > > the > > > hadoop > > > > world and would be foreign to the web. There is someting to be > > > > said for > > > > simplicity, but the incurred technical debt can become > > > > burdensome. > > > > > > > > > > > >> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon > > > >> <pshan...@fredhutch.org> > > > wrote: > > > >> > > > >> Hi Michael, > > > >> > > > >> Thanks for the link to the subprotocol negotiaion discussion in > > > >> "High > > > >> Performance Browser Networking". There's a number of things in > > > >> that > > > >> discussion of which I was not aware. Very helpful. > > > >> > > > >> There may be real merit -- that is, a good trade-off between > > > >> some added > > > >> complexity and extra clarity -- to adding a mandatory > > > >> content-type > > > subfield > > > >> to the payload or to the message. Or adding full sub-protocol > > > negotiation > > > >> as the reference you supplied discusses. > > > >> > > > >> For the first release of BrowserViz (and its demo subclass, > > > >> BrowserVizDemo, along with RCyjs) I would like to keep things as > > > >> simple > > > as > > > >> possible. By which I mean: provide (as we do now) the > > > >> mechanism by > > > which > > > >> content-type can be specified, without promulgating the policy > > > >> that > > > every > > > >> message must have that payload subfield. > > > >> > > > >> In practice I have found it quick and easy in both Javascript > > > >> and R to > > > >> examine the incoming 4-field message (cmd, status, callback, > > > >> payload): > > > >> > > > >> 1) Is the cmd field something I can respond to? (Do I have a > > > >> handler > > > >> registered for that cmd?) > > > >> 2) Does the status indicate trouble? A request? > > > >> 3) If 1 & 2 suggest proceeding, then names(msg$payload) can > > > >> help me > > > >> figure out if the incoming data is structured as I expect. > > > >> > > > >> One could reasonably object that this is all "programming by (ad > > > >> hoc) > > > >> agreement", rather than by transparent, self-describing, clearly > > > >> negotiatied data and exchange standards. > > > >> > > > >> In order to arrive at such standards, to figure out how formal > > > >> we should > > > >> go, let's experiment for a release cycle. This could be > > > >> accomplished > > > >> without any changes to BrowserViz by using the status field > > > >> (status="summarizedExperimentRequest"), or by always using two > > > >> fields in > > > >> every message payload (payload$contentType, payload$content). > > > >> If a few > > > >> people collaborated on (let's say) a SummarizedExperiment > > > >> browser viz > > > >> webapp, those people could agree on how to do this, try it out, > > > >> perhaps > > > >> discover a few sub-varieties of content-type are actually > > > >> needed. Then, > > > >> when things have settled down, we could explore adding some > > > >> formality to > > > >> the process. > > > >> > > > >> How does that strike you? > > > >> > > > >> Thanks for the discussion! > > > >> > > > >> - Paul > > > >> > > > >> > > > >> > > > >> On Apr 3, 2015, at 3:11 PM, Michael Lawrence > > > >> <lawrence.mich...@gene.com > > > > > > > >> wrote: > > > >> > > > >>> The high-level type issue is sort of discussed here: > > > >>> > > > >> > > > > http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation > > > >>> > > > >>> What about extending your protocol so that the payload consists > > > >>> of two > > > >> fields: > > > >>> content-type and content, where the content-type adheres to the > > > >>> media > > > >> type specification? This is analogous to how every S4 object has > > > >> a class > > > >> attribute. > > > >>> > > > >>> Thanks for the discussion! > > > >>> > > > >>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon < > > > >> paul.thurmond.shan...@gmail.com> wrote: > > > >>> Hi Michael, > > > >>> > > > >>> Thanks for the clarification. You make a good point about > > > >>> caching: > > > >> nobody wants to have to reinvent and re-engineer that! If too > > > >> many > > > >> features like that become important, then the simplicity of > > > >> websockets > > > will > > > >> have an attendant cost. > > > >>> > > > >>> With regard to more formalism around the payload: I can > > > >>> imagine that > > > >> there will be circumstances in which that is essential. A > > > >> SummarizedExperiment browser would be very useful, and would > > > >> clearly > > > >> benefit from a standard payload structure. > > > >>> > > > >>> I figure, however, that that's above my pay-grade ;). I leave > > > >>> that up > > > >> to people like you. And I stand ready to add any features such > > > formalized > > > >> structures might require. I'd like to think (I may be naive) > > > >> that the > > > >> architects of SummarizedExperiment and heavy users of it could > > > >> devise > > > and > > > >> negotiate some standard JSON representation of the class, > > > >> translators to > > > >> and from, which all could then be used by BrowserViz without > > > >> BrowserViz > > > >> needing to know it's there. > > > >>> > > > >>> The "binary JSON" format may be useful in some circumstances: > > > >> http://bjson.org > > > >>> > > > >>> - Paul > > > >>> > > > >>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence < > > > lawrence.mich...@gene.com> > > > >> wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon < > > > >> paul.thurmond.shan...@gmail.com> wrote: > > > >>>> Hi Michael, > > > >>>> > > > >>>> Great to get your response, comments and questions. Answers > > > >>>> attempted > > > >> below. > > > >>>> > > > >>>> I think our overriding difference lies in our contrasting > > > >>>> experience > > > of > > > >> complexity. I have come to see websockets as minimal, simple, > > > >> fast and > > > >> flexible, whereas you see them quite differently. I would like > > > >> to > > > >> understand your views on this; I could be overlooking somce > > > >> important > > > and > > > >> maybe costly complicating features, perhaps because of my > > > >> fondness for > > > >> other simplifying features. > > > >>>> > > > >>>> > > > >>>> I was just referring to potential complexity. If one attempted > > > >>>> to > > > >> reimplement the useful features of HTTP (like caching), > > > >> complexity > > > would be > > > >> introduced into application code, while you really want that > > > >> complexity > > > in > > > >> the protocol implementation. HTTP seems most appropriate for > > > >> when the > > > >> server is acting as the data model. Outside of that, I see the > > > >> benefits > > > of > > > >> web sockets. > > > >>>> > > > >>>> I think you missed my question about RPC. Also, any thoughts > > > >>>> as to > > > more > > > >> formalism around the payload? We obviously have complex data > > > >> structures, > > > >> and it would be nice to communicate semantics somehow to the web > > > browser. > > > >> For example, could there be some convention for representing a > > > >> SummarizedExperiment? Could a payload contain the equivalent of > > > >> a media > > > >> type that an R/JS library could understand to marshal objects? > > > >> Could > > > there > > > >> be some way to query for the types of payload a command > > > >> supports? > > > >>>> > > > >>>> I've seen stuff like WAMP, but they seem to lack the ability > > > >>>> to > > > declare > > > >> high-level types. Maybe that's just out of style? > > > >>>> > > > >>>> > > > >>>> More below... > > > >>>> > > > >>>> I look forward to hearing back from you. > > > >>>> > > > >>>> - Paul > > > >>>> > > > >>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence < > > > lawrence.mich...@gene.com> > > > >> wrote: > > > >>>> > > > >>>>> Thanks to Val's excellent newsletter, I've had my first > > > >>>>> glance at > > > >>>>> BrowserViz. I'm glad to see something that is more flexible > > > >>>>> and > > > >> low-level > > > >>>>> than e.g. shiny. > > > >>>>> > > > >>>>> I'm curious about the motivation behind web sockets. I guess > > > >>>>> any > > > >>>>> application with an R-driven web UI actually has two UIs: the > > > >>>>> R > > > >> console and > > > >>>>> the browser. But what if the R session is headless, or if > > > >>>>> there is no > > > >> need > > > >>>>> for commands in R to affect the browser? Then the web socket > > > >>>>> layer > > > >> brings > > > >>>>> mostly unneeded complexity. > > > >>>> > > > >>>> I see websockets (like TCP sockets) as musch simpler than > > > >>>> HTTP. No > > > >> headers, > > > >>>> no explicit server and explicit client (once the connection is > > > >>>> open). > > > >>>> Could you explain the complexity you see? > > > >>>> > > > >>>>> An interesting comparison to BrowserViz is not > > > >>>>> shiny but OpenCPU. It's purely HTTP-based and still manages > > > >>>>> to > > > maintain > > > >>>>> state (not sure how efficiently). I guess one advantage of > > > >>>>> web > > > sockets > > > >> is > > > >>>>> that one can program imperatively instead of declaratively on > > > >>>>> the > > > >> server, > > > >>>>> i.e., the server can send a command to show a popup in > > > >>>>> response to > > > some > > > >>>>> event, instead of returning a "declaration" that the popup > > > >>>>> should be > > > >> shown. > > > >>>> > > > >>>> Exactly! > > > >>>> > > > >>>>> > > > >>>>> So essentially web sockets are more natural for implementing > > > >> server-side > > > >>>>> controllers (think MVC), instead of just the data model, but > > > >>>>> man, > > > it's > > > >> a > > > >>>>> shame to lose the features of HTTP. > > > >>>> > > > >>>> I'll confess: I set out to -shed- the features of HTTP. > > > >>>> Isn't it a > > > >> protocol designed for serving up web pages on demand? Not for > > > >> fast, > > > >> lighweight peer-to-peer communications? > > > >>>> > > > >>>> > > > >>>>> > > > >>>>> Ultimately, I think we want web apps that are easy to develop > > > >>>>> and > > > >> maintain, > > > >>>>> and run equally well from either a useR's session or a remote > > > >>>>> client > > > >>>>> communicating to a dedicated, headless server. Is the > > > >>>>> generality of > > > >>>>> websockets worth the complexity? > > > >>>> > > > >>>>> > > > >>>>> As an aside, it would seem relatively straight-forward to > > > >>>>> implement a > > > >>>>> simple bi-directional RPC mechanism between R and JS using > > > >>>>> the > > > standard > > > >>>>> protocol (i.e., hide details like the callback). Does that > > > >>>>> sound > > > >> reasonable? > > > >>>>> > > > >>>>> I was also a bit surprised about the need to copy/paste the > > > >>>>> JS > > > >> boilerplate. > > > >>>>> Certainly there must be javascript frameworks with a more > > > >>>>> elegant > > > >> solution > > > >>>>> to extensibility. > > > >>>> > > > >>>> You are correct on this, and I only need to correct the > > > >>>> vignette, > > > >> written many weeks ago. > > > >>>> I now provide "BrowserViz.jx", a simple Javascript module. It > > > >>>> is used > > > >> by both > > > >>>> BrowserVizDemo and RCyjs. Sorry to have sent out an obsolete > > > vignette. > > > >>>> > > > >>>>> > > > >>>>> [[alternative HTML version deleted]] > > > >>>>> > > > >>>>> _______________________________________________ > > > >>>>> Bioc-devel@r-project.org mailing list > > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > [[alternative HTML version deleted]] > > > > > > > > _______________________________________________ > > > > Bioc-devel@r-project.org mailing list > > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > > > > [[alternative HTML version deleted]] > > > > _______________________________________________ > > Bioc-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
