Bi-directional communication over a single transport has nothing to do with the IDL. It is a new endpoint concept that allows either end to be a client and a server at the same time. Using two transports is not firewall friendly and typically not usable by most folks because of that.
Async Messaging and pub/sub are message bus style mechanisms, so wouldn't it be better to implement a message bus transport? I did some work enabling RabbitMQ and it was about 75% complete. See https://github.com/jeking3/thrift/tree/rabbitmq Thrift does not have a concept of [in, out, inout] parameters. All parameters are [in], and if data is to be returned then it is in the value returned. I don't see that changing. On Thu, May 16, 2019 at 12:35 AM John Dougrez-Lewis <jle...@lightblue.com> wrote: > > Sorry that last section did appear formatted very well. Here it is again: > > =============================== > > 1) > > A<=>B > > [async] returntype functionName (arg1, arg2) > > ====> > > A=>B > > Handletype functionName (arg1, arg2) > > B=>A > > void functionName (Handletype, returntype) > > =============================== > > 2) > > A<=>B > > [async] returntype functionName (arg1, arg2, [out] arg3 > > ====> > > A=>B > > Handletype functionName (arg1, arg2) > > B=>A > > void functionName (Handletype, returntype, arg3) > > =============================== > > 3) > > A<=>B > > [async] returntype functionName (arg1, arg2, [inout] arg3) > > ====> > > A=>B > > Handletype functionName (arg1, arg2, arg3in) > > B=>A > > void functionName (Handletype, returntype, arg3out) > > > > > -----Original Message----- > From: John Dougrez-Lewis [mailto:jle...@lightblue.com] > Sent: 16 May 2019 05:29 > To: James E. King III > Cc: dev@thrift.apache.org > Subject: RE: THRIFT-66 - Bidirectional communication > > Yes, my suggestion is for an enhancement to Thrift to make it bidirectional, > as follows: > > > 1) use 2 connections, A => B, and B => A using what is already in place. > > 2) write, by hand, a new, supporting choreography code to establish this > double connection, in the first instance for a single language > > 3) extend the IDL support this - adding attributes [asyc] method/interface, > [pub/sub] method/interface, [in/out/inout] parameter > > 4) extend the IDL generation to implement this: > > i) pre-processing the extended IDL with a new pre-processor to generate > the representations of the service definitions for both sides in terms of the > current IDL > > ii) generate the language dependent code required to hook up at of the > two ends. > > 5) implement across multiple languages > > > That gets you to the point where Thrift supports and generates bidirectional, > async & pub/sub based on IDL > > Then, optionally: > > 6) Having achieved a working implementation defined by an extended IDL, > consider re-implementing in terms of a single bidirectional transport rather > than 2 existing unidirectional independent transports. > > > The extended IDL defining a single async/pubsub A<=>B interface is used to > generate 2 interfaces: an outgoing A=>B and an async response return channel > B=>A, e.g.: > > > A<=>B > [async] returntype functionName (arg1, arg2) => A=>B > Handletype functionName (arg1, arg2) > B=>A > void functionName (Handletype, returntype) > > A<=>B > [async] returntype functionName (arg1, arg2, [out] arg3) => A=>B > Handletype functionName (arg1, arg2) > B=>A > void functionName (Handletype, returntype, arg3) > > > A<=>B > [async] returntype functionName (arg1, arg2, [inout] arg3) => A=>B > Handletype functionName (arg1, arg2, arg3in) B=>A > void functionName (Handletype, returntype, arg3out) > > > > -----Original Message----- > From: James E. King III [mailto:jk...@apache.org] > Sent: 15 May 2019 19:02 > To: jle...@lightblue.com > Cc: James E. King III; dev@thrift.apache.org > Subject: Re: THRIFT-66 - Bidirectional communication > > Re: Visio - There are no "endpoint" implementations that allow for > bi-directional communication over a single transport connection today. > Is that the Visio you were referring to? The only implementation of that > concept is buried in the THRIFT-66 attachments, and only for C#, and it was > done on a codebase about 8 years past... > > With today's thrift code if you want either end to be a client (make > requests) or a server (reply to requests) you would need to separately > instantiate a client or server on each end and have them connect to > each-other, i.e. > > A ---> B (A sends requests to a thrift server on B, B replies, on a > transport) A <--- B (B sends requests to a thrift server on A, A replies, on > a transport separate from the last one) > > It sounds like what you'd like to get to is: > > A <--> B (A and B can function as a client or as a server over the same > transport) > > Thrift cannot do the latter today, so I would recommend the former, using one > transport for each direction. > > Given they will both be on the same system, if your languages support it, > unix domain sockets are quite fast. > Otherwise if you want a shared memory solution you need to write your own > transport for that. > > - Jim > > On Wed, May 15, 2019 at 1:12 PM John Dougrez-Lewis <jle...@lightblue.com> > wrote: > > > > Hi Jim, > > > > The "oneway" route looks a bit fragile in the face of failures in the > > subsequent server-side processing which then cannot be signalled back to > > the client. > > > > The 2-way connection would be the way forward for me, particularly since it > > would work with across multiple languages. > > > > My primary use case would be a simple language bridge mechanism for a > > library to allow processes coded in one language to call, with potentially > > asynchronously returns, and pub/sub, to another process hosting a library > > coded in another language running (in the first instance) on the same box, > > communicating via IPC, preferably fast shared memory, but failing that > > sockets would do. > > > > You put a Visio diagram up back in 2010. Is the underlying source code for > > that available now ? > > > > Rather than hand-rolling the 2-way connection setup/teardown and supporting > > code, for each and every language each time, it would be nice if Framework > > code for that could be generated automatically from an enhanced and > > extended version of the IDL. > > > > Regards, > > > > John > > > > -----Original Message----- > > From: James E. King III [mailto:jk...@apache.org] > > Sent: 15 May 2019 12:05 > > To: dev@thrift.apache.org; jle...@lightblue.com > > Subject: Re: THRIFT-66 - Bidirectional communication > > > > Hello! > > > > Thrift is still a dedicated client/server model environment where clients > > can request and servers reply. The easiest way to make it 2-way today is > > to open a connection both ways. If you don't have firewalls in the way > > then you can do this effectively. The more difficult and more correct way > > to do it would be to rewrite the transport layer to use endpoints in which > > each side can be a client and/or server for any number of services (using > > TMultiplexedProtocol on top of another protocol, like TBinaryProtocol). > > This design allows one end to be a "listener", one end to be an "initiator" > > (starts the connection), and after they connect they are equal peers with > > the ability to request or reply of each-other. > > > > You can approximate asynchronous behavior by exclusively using "oneway" > > requests in your design. I'd suggest avoiding use of oneway requests with > > THttpProtocol varieties however as today there are some issues, since Http > > transport requires a response to be sent, and "oneway" dictates there is no > > reply, and most languages do not handle it well right now (there are open > > backlog issues for this). > > > > For a matrix of supported languages, protocols, transports, and server > > types, see the file LANGUAGES.md at the root of the github repository. > > > > Another idea I was toying with a while ago was to add a message bus > > transport to Thrift which would allow for things like reliable delivery and > > broadcast semantics but that also does not exist today. > > > > - Jim > > > > On Wed, May 15, 2019 at 1:05 AM John Dougrez-Lewis <jle...@lightblue.com> > > wrote: > > > > > > Hi, > > > > > > > > > > > > I was looking for a mechanism to be able to provide > > > language-agnostic API support to a hobby project I've been working on for > > > some time. > > > > > > > > > > > > By following a trail of papers, books and references, I eventually > > > came across Apache Thrift and have found and started going through > > > Randy Abernethy's new book. > > > > > > > > > > > > > > > > > > Essentially what I was looking for was support for asynchronous > > > calls, and by extension, pub/sub and two way communication across > > > and between multiple languages over some channel, preferably IPC but in > > > the worst case sockets. > > > > > > > > > > > > > > > > > > Having read the book, I can see that there is support for basic > > > synchronous RPC between a client and a server over a significant > > > number of languages and for just a very few languages, such as java, > > > some element of support for asynchronous callbacks, and otherwise > > > one-way methods that do not provide indication of subsequent failure. > > > > > > > > > > > > It appeared to me one way of extending bi-directional asynchronous > > > support would be to have the client to set itself up as a server for > > > the server at the other end to connect to, and then it would just be > > > a question of choreographing the setting up of a pair of RPC channels. > > > > > > > > > > > > An asynchronous call could be implemented by providing a synchronous > > > method that simply immediately returns a handle to the caller, and > > > the server would then continue to process the call request on a > > > background threadpool thread on the server, and the async result > > > would then be signalled by a call from the server back to the client > > > on the 2nd channel with the handle providing a context to lookup the > > > result. > > > > > > > > > > > > Pub/sub would just then be multiple calls from the server back to > > > the client. > > > > > > > > > > > > The whole thing could sit on top of the existing unidirectional RPC > > > implementation and provide full asynchronous calls & pub/sub across > > > *ALL* supported languages at probably very little additional effort, > > > with no changes to the existing code. > > > > > > > > > > > > You could then have a framework that extended the existing IDL to > > > include decoration with attributes for async & pub/sub methods & in/out > > > parameters. > > > > > > > > > > > > This extended IDL could then be pre-processed to generate > > > client-server and server-client service definitions in the existing > > > base IDL language, together with generating supporting glue code to > > > compile to provide the support for hooking up the channels between each > > > side. > > > > > > > > > > > > I note that THRIFT-66 was raised 10 years ago, but it looks like the > > > C# code was never made available for release by Dell. > > > > > > > > > > > > I have some questions: > > > > > > > > > > > > 1) What is the current state of plans for this supporting this sort > > > of > > > functionality? What issues have been encountered ? > > > > > > > > > > > > 2) Is there a document/spreadsheet somewhere showing a matrix of what > > > Transports and Protocols are supported for each language? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > > > > >
