Protobuf begat Avro (more or less) https://avro.apache.org/docs/current/#compare
--t > On Apr 6, 2015, at 5:08 PM, Dan Tenenbaum <dtene...@fredhutch.org> wrote: > > > > ----- Original Message ----- >> From: "Michael Lawrence" <lawrence.mich...@gene.com> >> To: "Tim Triche, Jr." <tim.tri...@gmail.com> >> Cc: "Michael Lawrence" <lawrence.mich...@gene.com>, >> bioc-devel@r-project.org, "Paul Shannon" <pshan...@fredhutch.org> >> Sent: Monday, April 6, 2015 4:48:41 PM >> Subject: Re: [Bioc-devel] BrowserViz and sub-protocols >> >> On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr. >> <tim.tri...@gmail.com> >> wrote: >> >>> Kitware seems to have had good experiences with WAMP, which is a >>> good >>> sign, given the complexity of their typical visualization projects >>> (stemming from old HPC projects in fluid dynamics, astrophysics, >>> realtime >>> anatomical reconstruction, etc). Thus WAMP seems like a great way >>> to >>> leverage the experiences of an open source focused, medium sized >>> shop known >>> for doing highly technical work with some similar aims to Paul's >>> pkg. >>> >>> Avro's IDL becomes onerous for large projects rather fast. The >>> lack of >>> inheritance can be excruciating when it is pushed beyond its >>> typical scope. >>> Best not to poke that sleeping dog ;-) >> Hmm, thanks for pointing that out. Is there no good solution for >> serializing objects, outside of simple lists and maps? > > Protocol buffers? > > https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket > > Dan > > >> >>> --t >>> >>>> On Apr 6, 2015, at 9:35 AM, Michael Lawrence >>>> <lawrence.mich...@gene.com> >>> wrote: >>>> >>>> Sounds like a good plan. It does seem like there is a trend away >>>> from >>> type >>>> formalism in application development. Everything is essentially >>>> JSON. >>> Have >>>> you checked out WAMP? It seems very similar to your solution; it >>>> might >>> pay >>>> off to stick with standards, even if it takes more up-front >>>> investment. >>>> It's JSON-based, so it doesn't solve the high-level type problem. >>>> Very >>> few >>>> serialization formats do. Avro has a nice IDL, but that's out of >>>> the >>> hadoop >>>> world and would be foreign to the web. There is someting to be >>>> said for >>>> simplicity, but the incurred technical debt can become >>>> burdensome. >>>> >>>> >>>>> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon >>>>> <pshan...@fredhutch.org> >>> wrote: >>>>> >>>>> Hi Michael, >>>>> >>>>> Thanks for the link to the subprotocol negotiaion discussion in >>>>> "High >>>>> Performance Browser Networking". There's a number of things in >>>>> that >>>>> discussion of which I was not aware. Very helpful. >>>>> >>>>> There may be real merit -- that is, a good trade-off between >>>>> some added >>>>> complexity and extra clarity -- to adding a mandatory >>>>> content-type >>> subfield >>>>> to the payload or to the message. Or adding full sub-protocol >>> negotiation >>>>> as the reference you supplied discusses. >>>>> >>>>> For the first release of BrowserViz (and its demo subclass, >>>>> BrowserVizDemo, along with RCyjs) I would like to keep things as >>>>> simple >>> as >>>>> possible. By which I mean: provide (as we do now) the >>>>> mechanism by >>> which >>>>> content-type can be specified, without promulgating the policy >>>>> that >>> every >>>>> message must have that payload subfield. >>>>> >>>>> In practice I have found it quick and easy in both Javascript >>>>> and R to >>>>> examine the incoming 4-field message (cmd, status, callback, >>>>> payload): >>>>> >>>>> 1) Is the cmd field something I can respond to? (Do I have a >>>>> handler >>>>> registered for that cmd?) >>>>> 2) Does the status indicate trouble? A request? >>>>> 3) If 1 & 2 suggest proceeding, then names(msg$payload) can >>>>> help me >>>>> figure out if the incoming data is structured as I expect. >>>>> >>>>> One could reasonably object that this is all "programming by (ad >>>>> hoc) >>>>> agreement", rather than by transparent, self-describing, clearly >>>>> negotiatied data and exchange standards. >>>>> >>>>> In order to arrive at such standards, to figure out how formal >>>>> we should >>>>> go, let's experiment for a release cycle. This could be >>>>> accomplished >>>>> without any changes to BrowserViz by using the status field >>>>> (status="summarizedExperimentRequest"), or by always using two >>>>> fields in >>>>> every message payload (payload$contentType, payload$content). >>>>> If a few >>>>> people collaborated on (let's say) a SummarizedExperiment >>>>> browser viz >>>>> webapp, those people could agree on how to do this, try it out, >>>>> perhaps >>>>> discover a few sub-varieties of content-type are actually >>>>> needed. Then, >>>>> when things have settled down, we could explore adding some >>>>> formality to >>>>> the process. >>>>> >>>>> How does that strike you? >>>>> >>>>> Thanks for the discussion! >>>>> >>>>> - Paul >>>>> >>>>> >>>>> >>>>> On Apr 3, 2015, at 3:11 PM, Michael Lawrence >>>>> <lawrence.mich...@gene.com >>>> >>>>> wrote: >>>>> >>>>>> The high-level type issue is sort of discussed here: >>> http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation >>>>>> >>>>>> What about extending your protocol so that the payload consists >>>>>> of two >>>>> fields: >>>>>> content-type and content, where the content-type adheres to the >>>>>> media >>>>> type specification? This is analogous to how every S4 object has >>>>> a class >>>>> attribute. >>>>>> >>>>>> Thanks for the discussion! >>>>>> >>>>>>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon < >>>>>> paul.thurmond.shan...@gmail.com> wrote: >>>>>> Hi Michael, >>>>>> >>>>>> Thanks for the clarification. You make a good point about >>>>>> caching: >>>>> nobody wants to have to reinvent and re-engineer that! If too >>>>> many >>>>> features like that become important, then the simplicity of >>>>> websockets >>> will >>>>> have an attendant cost. >>>>>> >>>>>> With regard to more formalism around the payload: I can >>>>>> imagine that >>>>> there will be circumstances in which that is essential. A >>>>> SummarizedExperiment browser would be very useful, and would >>>>> clearly >>>>> benefit from a standard payload structure. >>>>>> >>>>>> I figure, however, that that's above my pay-grade ;). I leave >>>>>> that up >>>>> to people like you. And I stand ready to add any features such >>> formalized >>>>> structures might require. I'd like to think (I may be naive) >>>>> that the >>>>> architects of SummarizedExperiment and heavy users of it could >>>>> devise >>> and >>>>> negotiate some standard JSON representation of the class, >>>>> translators to >>>>> and from, which all could then be used by BrowserViz without >>>>> BrowserViz >>>>> needing to know it's there. >>>>>> >>>>>> The "binary JSON" format may be useful in some circumstances: >>>>> http://bjson.org >>>>>> >>>>>> - Paul >>>>>> >>>>>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence < >>> lawrence.mich...@gene.com> >>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon < >>>>> paul.thurmond.shan...@gmail.com> wrote: >>>>>>> Hi Michael, >>>>>>> >>>>>>> Great to get your response, comments and questions. Answers >>>>>>> attempted >>>>> below. >>>>>>> >>>>>>> I think our overriding difference lies in our contrasting >>>>>>> experience >>> of >>>>> complexity. I have come to see websockets as minimal, simple, >>>>> fast and >>>>> flexible, whereas you see them quite differently. I would like >>>>> to >>>>> understand your views on this; I could be overlooking somce >>>>> important >>> and >>>>> maybe costly complicating features, perhaps because of my >>>>> fondness for >>>>> other simplifying features. >>>>>>> >>>>>>> >>>>>>> I was just referring to potential complexity. If one attempted >>>>>>> to >>>>> reimplement the useful features of HTTP (like caching), >>>>> complexity >>> would be >>>>> introduced into application code, while you really want that >>>>> complexity >>> in >>>>> the protocol implementation. HTTP seems most appropriate for >>>>> when the >>>>> server is acting as the data model. Outside of that, I see the >>>>> benefits >>> of >>>>> web sockets. >>>>>>> >>>>>>> I think you missed my question about RPC. Also, any thoughts >>>>>>> as to >>> more >>>>> formalism around the payload? We obviously have complex data >>>>> structures, >>>>> and it would be nice to communicate semantics somehow to the web >>> browser. >>>>> For example, could there be some convention for representing a >>>>> SummarizedExperiment? Could a payload contain the equivalent of >>>>> a media >>>>> type that an R/JS library could understand to marshal objects? >>>>> Could >>> there >>>>> be some way to query for the types of payload a command >>>>> supports? >>>>>>> >>>>>>> I've seen stuff like WAMP, but they seem to lack the ability >>>>>>> to >>> declare >>>>> high-level types. Maybe that's just out of style? >>>>>>> >>>>>>> >>>>>>> More below... >>>>>>> >>>>>>> I look forward to hearing back from you. >>>>>>> >>>>>>> - Paul >>>>>>> >>>>>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence < >>> lawrence.mich...@gene.com> >>>>> wrote: >>>>>>> >>>>>>>> Thanks to Val's excellent newsletter, I've had my first >>>>>>>> glance at >>>>>>>> BrowserViz. I'm glad to see something that is more flexible >>>>>>>> and >>>>> low-level >>>>>>>> than e.g. shiny. >>>>>>>> >>>>>>>> I'm curious about the motivation behind web sockets. I guess >>>>>>>> any >>>>>>>> application with an R-driven web UI actually has two UIs: the >>>>>>>> R >>>>> console and >>>>>>>> the browser. But what if the R session is headless, or if >>>>>>>> there is no >>>>> need >>>>>>>> for commands in R to affect the browser? Then the web socket >>>>>>>> layer >>>>> brings >>>>>>>> mostly unneeded complexity. >>>>>>> >>>>>>> I see websockets (like TCP sockets) as musch simpler than >>>>>>> HTTP. No >>>>> headers, >>>>>>> no explicit server and explicit client (once the connection is >>>>>>> open). >>>>>>> Could you explain the complexity you see? >>>>>>> >>>>>>>> An interesting comparison to BrowserViz is not >>>>>>>> shiny but OpenCPU. It's purely HTTP-based and still manages >>>>>>>> to >>> maintain >>>>>>>> state (not sure how efficiently). I guess one advantage of >>>>>>>> web >>> sockets >>>>> is >>>>>>>> that one can program imperatively instead of declaratively on >>>>>>>> the >>>>> server, >>>>>>>> i.e., the server can send a command to show a popup in >>>>>>>> response to >>> some >>>>>>>> event, instead of returning a "declaration" that the popup >>>>>>>> should be >>>>> shown. >>>>>>> >>>>>>> Exactly! >>>>>>> >>>>>>>> >>>>>>>> So essentially web sockets are more natural for implementing >>>>> server-side >>>>>>>> controllers (think MVC), instead of just the data model, but >>>>>>>> man, >>> it's >>>>> a >>>>>>>> shame to lose the features of HTTP. >>>>>>> >>>>>>> I'll confess: I set out to -shed- the features of HTTP. >>>>>>> Isn't it a >>>>> protocol designed for serving up web pages on demand? Not for >>>>> fast, >>>>> lighweight peer-to-peer communications? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Ultimately, I think we want web apps that are easy to develop >>>>>>>> and >>>>> maintain, >>>>>>>> and run equally well from either a useR's session or a remote >>>>>>>> client >>>>>>>> communicating to a dedicated, headless server. Is the >>>>>>>> generality of >>>>>>>> websockets worth the complexity? >>>>>>> >>>>>>>> >>>>>>>> As an aside, it would seem relatively straight-forward to >>>>>>>> implement a >>>>>>>> simple bi-directional RPC mechanism between R and JS using >>>>>>>> the >>> standard >>>>>>>> protocol (i.e., hide details like the callback). Does that >>>>>>>> sound >>>>> reasonable? >>>>>>>> >>>>>>>> I was also a bit surprised about the need to copy/paste the >>>>>>>> JS >>>>> boilerplate. >>>>>>>> Certainly there must be javascript frameworks with a more >>>>>>>> elegant >>>>> solution >>>>>>>> to extensibility. >>>>>>> >>>>>>> You are correct on this, and I only need to correct the >>>>>>> vignette, >>>>> written many weeks ago. >>>>>>> I now provide "BrowserViz.jx", a simple Javascript module. It >>>>>>> is used >>>>> by both >>>>>>> BrowserVizDemo and RCyjs. Sorry to have sent out an obsolete >>> vignette. >>>>>>> >>>>>>>> >>>>>>>> [[alternative HTML version deleted]] >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Bioc-devel@r-project.org mailing list >>>>>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel >>>> >>>> [[alternative HTML version deleted]] >>>> >>>> _______________________________________________ >>>> Bioc-devel@r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel >> >> [[alternative HTML version deleted]] >> >> _______________________________________________ >> Bioc-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/bioc-devel >> _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
