A brief description is here: http://docs.datatorrent.com/beginner/#buffer-server
It is indeed part of apex-core; see: https://github.com/apache/apex-core/tree/master/engine/src/main/java/com/datatorrent/stram/ as well as the *stream* subdirectory. Ram On Fri, Nov 25, 2016 at 11:13 AM, Max Bridgewater <max.bridgewa...@gmail.com > wrote: > Awesome. Thanks for the additional details. I can't find any reference to > Buffer Server online. Could you give some pointers? or is it part of the > Apex core? > > Regards, > Max. > > On Fri, Nov 25, 2016 at 1:57 PM, Ashwin Chandra Putta < > ashwinchand...@gmail.com> wrote: > >> Max, >> >> Apex deploys a fast pub-sub server called buffer server in every yarn >> container it gets (except AM) before deploying operators on it. For all the >> operators which are connected downstream to operators outside the current >> container, their output ports become publishers to the buffer server. The >> downstream operators' input ports become subscribers to the buffer server. >> So there is no concept a central operator/port registry, however all the >> downstream operators do register their input ports with the buffer server. >> The serialization algorithm is kryo based. The data transport protocol is >> based on Netlet: https://github.com/DataTorrent/Netlet which is on top >> of TCP/IP. >> >> Regards, >> Ashwin. >> >> On Fri, Nov 25, 2016 at 8:33 AM, Max Bridgewater < >> max.bridgewa...@gmail.com> wrote: >> >>> Hi Folks, >>> >>> I was giving an Apex demo the other day and people asked following >>> questions: >>> >>> 1) what is the communication protocol between operators when they are on >>> distant nodes. That means, how does Apex transport the tuples from one node >>> to the other? >>> Is it a custom protocol on top of TCP/IP or is it RPC? >>> 2) What is the serialization algorithm used? >>> 3) What is the addressing scheme between operators? That means how does >>> Apex know where an operator is located and how to route data to it? Is >>> there an operator registry? If so, where does it reside? >>> >>> Thoughts? >>> >>> Thanks, >>> Max. >>> >>> >> >> >> -- >> >> Regards, >> Ashwin. >> > >
