Bingo --t
> On Apr 6, 2015, at 5:32 PM, Michael Lawrence <lawrence.mich...@gene.com> > wrote: > > Worth considering, but my understanding is that protocol buffers do not > directly embed their type information. So while there is a schema > corresponding to the buffer, the application needs to know which. Presumably > one would need some sort of IDL on top of protobufs to handle that. > > There's also this one: https://capnproto.org/. > > It has a very rich schema/IDL language (including inheritance), and the RPC > protocol has full support for lazy evaluation (pipeline promises), and > supports incremental reads, with mmap, etc. Just seems a little out there. By > the guy who designed protobufs v2. > > >> On Mon, Apr 6, 2015 at 5:08 PM, Dan Tenenbaum <dtene...@fredhutch.org> wrote: >> >> >> ----- Original Message ----- >> > From: "Michael Lawrence" <lawrence.mich...@gene.com> >> > To: "Tim Triche, Jr." <tim.tri...@gmail.com> >> > Cc: "Michael Lawrence" <lawrence.mich...@gene.com>, >> > bioc-devel@r-project.org, "Paul Shannon" <pshan...@fredhutch.org> >> > Sent: Monday, April 6, 2015 4:48:41 PM >> > Subject: Re: [Bioc-devel] BrowserViz and sub-protocols >> > >> > On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr. >> > <tim.tri...@gmail.com> >> > wrote: >> > >> > > Kitware seems to have had good experiences with WAMP, which is a >> > > good >> > > sign, given the complexity of their typical visualization projects >> > > (stemming from old HPC projects in fluid dynamics, astrophysics, >> > > realtime >> > > anatomical reconstruction, etc). Thus WAMP seems like a great way >> > > to >> > > leverage the experiences of an open source focused, medium sized >> > > shop known >> > > for doing highly technical work with some similar aims to Paul's >> > > pkg. >> > > >> > > Avro's IDL becomes onerous for large projects rather fast. The >> > > lack of >> > > inheritance can be excruciating when it is pushed beyond its >> > > typical scope. >> > > Best not to poke that sleeping dog ;-) >> > > >> > > >> > Hmm, thanks for pointing that out. Is there no good solution for >> > serializing objects, outside of simple lists and maps? >> > >> >> Protocol buffers? >> >> https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket >> >> Dan >> >> >> > >> > > --t >> > > >> > > > On Apr 6, 2015, at 9:35 AM, Michael Lawrence >> > > > <lawrence.mich...@gene.com> >> > > wrote: >> > > > >> > > > Sounds like a good plan. It does seem like there is a trend away >> > > > from >> > > type >> > > > formalism in application development. Everything is essentially >> > > > JSON. >> > > Have >> > > > you checked out WAMP? It seems very similar to your solution; it >> > > > might >> > > pay >> > > > off to stick with standards, even if it takes more up-front >> > > > investment. >> > > > It's JSON-based, so it doesn't solve the high-level type problem. >> > > > Very >> > > few >> > > > serialization formats do. Avro has a nice IDL, but that's out of >> > > > the >> > > hadoop >> > > > world and would be foreign to the web. There is someting to be >> > > > said for >> > > > simplicity, but the incurred technical debt can become >> > > > burdensome. >> > > > >> > > > >> > > >> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon >> > > >> <pshan...@fredhutch.org> >> > > wrote: >> > > >> >> > > >> Hi Michael, >> > > >> >> > > >> Thanks for the link to the subprotocol negotiaion discussion in >> > > >> "High >> > > >> Performance Browser Networking". There's a number of things in >> > > >> that >> > > >> discussion of which I was not aware. Very helpful. >> > > >> >> > > >> There may be real merit -- that is, a good trade-off between >> > > >> some added >> > > >> complexity and extra clarity -- to adding a mandatory >> > > >> content-type >> > > subfield >> > > >> to the payload or to the message. Or adding full sub-protocol >> > > negotiation >> > > >> as the reference you supplied discusses. >> > > >> >> > > >> For the first release of BrowserViz (and its demo subclass, >> > > >> BrowserVizDemo, along with RCyjs) I would like to keep things as >> > > >> simple >> > > as >> > > >> possible. By which I mean: provide (as we do now) the >> > > >> mechanism by >> > > which >> > > >> content-type can be specified, without promulgating the policy >> > > >> that >> > > every >> > > >> message must have that payload subfield. >> > > >> >> > > >> In practice I have found it quick and easy in both Javascript >> > > >> and R to >> > > >> examine the incoming 4-field message (cmd, status, callback, >> > > >> payload): >> > > >> >> > > >> 1) Is the cmd field something I can respond to? (Do I have a >> > > >> handler >> > > >> registered for that cmd?) >> > > >> 2) Does the status indicate trouble? A request? >> > > >> 3) If 1 & 2 suggest proceeding, then names(msg$payload) can >> > > >> help me >> > > >> figure out if the incoming data is structured as I expect. >> > > >> >> > > >> One could reasonably object that this is all "programming by (ad >> > > >> hoc) >> > > >> agreement", rather than by transparent, self-describing, clearly >> > > >> negotiatied data and exchange standards. >> > > >> >> > > >> In order to arrive at such standards, to figure out how formal >> > > >> we should >> > > >> go, let's experiment for a release cycle. This could be >> > > >> accomplished >> > > >> without any changes to BrowserViz by using the status field >> > > >> (status="summarizedExperimentRequest"), or by always using two >> > > >> fields in >> > > >> every message payload (payload$contentType, payload$content). >> > > >> If a few >> > > >> people collaborated on (let's say) a SummarizedExperiment >> > > >> browser viz >> > > >> webapp, those people could agree on how to do this, try it out, >> > > >> perhaps >> > > >> discover a few sub-varieties of content-type are actually >> > > >> needed. Then, >> > > >> when things have settled down, we could explore adding some >> > > >> formality to >> > > >> the process. >> > > >> >> > > >> How does that strike you? >> > > >> >> > > >> Thanks for the discussion! >> > > >> >> > > >> - Paul >> > > >> >> > > >> >> > > >> >> > > >> On Apr 3, 2015, at 3:11 PM, Michael Lawrence >> > > >> <lawrence.mich...@gene.com >> > > > >> > > >> wrote: >> > > >> >> > > >>> The high-level type issue is sort of discussed here: >> > > >>> >> > > >> >> > > http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation >> > > >>> >> > > >>> What about extending your protocol so that the payload consists >> > > >>> of two >> > > >> fields: >> > > >>> content-type and content, where the content-type adheres to the >> > > >>> media >> > > >> type specification? This is analogous to how every S4 object has >> > > >> a class >> > > >> attribute. >> > > >>> >> > > >>> Thanks for the discussion! >> > > >>> >> > > >>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon < >> > > >> paul.thurmond.shan...@gmail.com> wrote: >> > > >>> Hi Michael, >> > > >>> >> > > >>> Thanks for the clarification. You make a good point about >> > > >>> caching: >> > > >> nobody wants to have to reinvent and re-engineer that! If too >> > > >> many >> > > >> features like that become important, then the simplicity of >> > > >> websockets >> > > will >> > > >> have an attendant cost. >> > > >>> >> > > >>> With regard to more formalism around the payload: I can >> > > >>> imagine that >> > > >> there will be circumstances in which that is essential. A >> > > >> SummarizedExperiment browser would be very useful, and would >> > > >> clearly >> > > >> benefit from a standard payload structure. >> > > >>> >> > > >>> I figure, however, that that's above my pay-grade ;). I leave >> > > >>> that up >> > > >> to people like you. And I stand ready to add any features such >> > > formalized >> > > >> structures might require. I'd like to think (I may be naive) >> > > >> that the >> > > >> architects of SummarizedExperiment and heavy users of it could >> > > >> devise >> > > and >> > > >> negotiate some standard JSON representation of the class, >> > > >> translators to >> > > >> and from, which all could then be used by BrowserViz without >> > > >> BrowserViz >> > > >> needing to know it's there. >> > > >>> >> > > >>> The "binary JSON" format may be useful in some circumstances: >> > > >> http://bjson.org >> > > >>> >> > > >>> - Paul >> > > >>> >> > > >>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence < >> > > lawrence.mich...@gene.com> >> > > >> wrote: >> > > >>> >> > > >>>> >> > > >>>> >> > > >>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon < >> > > >> paul.thurmond.shan...@gmail.com> wrote: >> > > >>>> Hi Michael, >> > > >>>> >> > > >>>> Great to get your response, comments and questions. Answers >> > > >>>> attempted >> > > >> below. >> > > >>>> >> > > >>>> I think our overriding difference lies in our contrasting >> > > >>>> experience >> > > of >> > > >> complexity. I have come to see websockets as minimal, simple, >> > > >> fast and >> > > >> flexible, whereas you see them quite differently. I would like >> > > >> to >> > > >> understand your views on this; I could be overlooking somce >> > > >> important >> > > and >> > > >> maybe costly complicating features, perhaps because of my >> > > >> fondness for >> > > >> other simplifying features. >> > > >>>> >> > > >>>> >> > > >>>> I was just referring to potential complexity. If one attempted >> > > >>>> to >> > > >> reimplement the useful features of HTTP (like caching), >> > > >> complexity >> > > would be >> > > >> introduced into application code, while you really want that >> > > >> complexity >> > > in >> > > >> the protocol implementation. HTTP seems most appropriate for >> > > >> when the >> > > >> server is acting as the data model. Outside of that, I see the >> > > >> benefits >> > > of >> > > >> web sockets. >> > > >>>> >> > > >>>> I think you missed my question about RPC. Also, any thoughts >> > > >>>> as to >> > > more >> > > >> formalism around the payload? We obviously have complex data >> > > >> structures, >> > > >> and it would be nice to communicate semantics somehow to the web >> > > browser. >> > > >> For example, could there be some convention for representing a >> > > >> SummarizedExperiment? Could a payload contain the equivalent of >> > > >> a media >> > > >> type that an R/JS library could understand to marshal objects? >> > > >> Could >> > > there >> > > >> be some way to query for the types of payload a command >> > > >> supports? >> > > >>>> >> > > >>>> I've seen stuff like WAMP, but they seem to lack the ability >> > > >>>> to >> > > declare >> > > >> high-level types. Maybe that's just out of style? >> > > >>>> >> > > >>>> >> > > >>>> More below... >> > > >>>> >> > > >>>> I look forward to hearing back from you. >> > > >>>> >> > > >>>> - Paul >> > > >>>> >> > > >>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence < >> > > lawrence.mich...@gene.com> >> > > >> wrote: >> > > >>>> >> > > >>>>> Thanks to Val's excellent newsletter, I've had my first >> > > >>>>> glance at >> > > >>>>> BrowserViz. I'm glad to see something that is more flexible >> > > >>>>> and >> > > >> low-level >> > > >>>>> than e.g. shiny. >> > > >>>>> >> > > >>>>> I'm curious about the motivation behind web sockets. I guess >> > > >>>>> any >> > > >>>>> application with an R-driven web UI actually has two UIs: the >> > > >>>>> R >> > > >> console and >> > > >>>>> the browser. But what if the R session is headless, or if >> > > >>>>> there is no >> > > >> need >> > > >>>>> for commands in R to affect the browser? Then the web socket >> > > >>>>> layer >> > > >> brings >> > > >>>>> mostly unneeded complexity. >> > > >>>> >> > > >>>> I see websockets (like TCP sockets) as musch simpler than >> > > >>>> HTTP. No >> > > >> headers, >> > > >>>> no explicit server and explicit client (once the connection is >> > > >>>> open). >> > > >>>> Could you explain the complexity you see? >> > > >>>> >> > > >>>>> An interesting comparison to BrowserViz is not >> > > >>>>> shiny but OpenCPU. It's purely HTTP-based and still manages >> > > >>>>> to >> > > maintain >> > > >>>>> state (not sure how efficiently). I guess one advantage of >> > > >>>>> web >> > > sockets >> > > >> is >> > > >>>>> that one can program imperatively instead of declaratively on >> > > >>>>> the >> > > >> server, >> > > >>>>> i.e., the server can send a command to show a popup in >> > > >>>>> response to >> > > some >> > > >>>>> event, instead of returning a "declaration" that the popup >> > > >>>>> should be >> > > >> shown. >> > > >>>> >> > > >>>> Exactly! >> > > >>>> >> > > >>>>> >> > > >>>>> So essentially web sockets are more natural for implementing >> > > >> server-side >> > > >>>>> controllers (think MVC), instead of just the data model, but >> > > >>>>> man, >> > > it's >> > > >> a >> > > >>>>> shame to lose the features of HTTP. >> > > >>>> >> > > >>>> I'll confess: I set out to -shed- the features of HTTP. >> > > >>>> Isn't it a >> > > >> protocol designed for serving up web pages on demand? Not for >> > > >> fast, >> > > >> lighweight peer-to-peer communications? >> > > >>>> >> > > >>>> >> > > >>>>> >> > > >>>>> Ultimately, I think we want web apps that are easy to develop >> > > >>>>> and >> > > >> maintain, >> > > >>>>> and run equally well from either a useR's session or a remote >> > > >>>>> client >> > > >>>>> communicating to a dedicated, headless server. Is the >> > > >>>>> generality of >> > > >>>>> websockets worth the complexity? >> > > >>>> >> > > >>>>> >> > > >>>>> As an aside, it would seem relatively straight-forward to >> > > >>>>> implement a >> > > >>>>> simple bi-directional RPC mechanism between R and JS using >> > > >>>>> the >> > > standard >> > > >>>>> protocol (i.e., hide details like the callback). Does that >> > > >>>>> sound >> > > >> reasonable? >> > > >>>>> >> > > >>>>> I was also a bit surprised about the need to copy/paste the >> > > >>>>> JS >> > > >> boilerplate. >> > > >>>>> Certainly there must be javascript frameworks with a more >> > > >>>>> elegant >> > > >> solution >> > > >>>>> to extensibility. >> > > >>>> >> > > >>>> You are correct on this, and I only need to correct the >> > > >>>> vignette, >> > > >> written many weeks ago. >> > > >>>> I now provide "BrowserViz.jx", a simple Javascript module. It >> > > >>>> is used >> > > >> by both >> > > >>>> BrowserVizDemo and RCyjs. Sorry to have sent out an obsolete >> > > vignette. >> > > >>>> >> > > >>>>> >> > > >>>>> [[alternative HTML version deleted]] >> > > >>>>> >> > > >>>>> _______________________________________________ >> > > >>>>> Bioc-devel@r-project.org mailing list >> > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel >> > > >>>> >> > > >>>> >> > > >>> >> > > >>> >> > > >> >> > > >> >> > > > >> > > > [[alternative HTML version deleted]] >> > > > >> > > > _______________________________________________ >> > > > Bioc-devel@r-project.org mailing list >> > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel >> > > >> > >> > [[alternative HTML version deleted]] >> > >> > _______________________________________________ >> > Bioc-devel@r-project.org mailing list >> > https://stat.ethz.ch/mailman/listinfo/bioc-devel >> > > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
