Hi Ilya, In the case where upstream and downstream operator are in different containers, the stream is implemented through buffer server. The output port is connected to a BufferServerPublisher, which will put the data into the netlet send buffer. Downstream, the input port will be linked to a BufferServerSubscriber reservoir, which will collect the data received by netlet.
Once the operator is activated, each input port has a reservoir buffer for the tuples retrieved from the stream. These are either control tuples injected by the engine or payload tuples emitted by the upstream operator. Have a look at WindowGenerator, which is connected to the input operators. The "Node" (wrapper around the operator) is responsible to merge the tuples from the reservoir buffers into a single thread. This thread (the "operator thread") will perform the synchronization for multiple input ports and perform the callbacks on the operator. For example, it will need to synchronize the window boundary between buffers and call begin/endWindow accordingly. For other stream localities the ports will be wired differently. You have probably already seen this in StreamingContainer. See InlineStream for CONTAINER_LOCAL and OiOStream for THREAD_LOCAL. There are also different node implementations for the different types of operators (InputNode, GenericNode, UnifierNode etc.) HTH, Thomas On Wed, Oct 21, 2015 at 11:15 AM, Ganelin, Ilya <[email protected] > wrote: > Hi all - I believe at this point, I understand the high level data flow > within Apex, but I would love some pointers as to where this is implemented > in the code as well as comments on the data flow itself. Ultimately, it > would be great to capture this in a document on the Apex website. > > 1) We create operators and thence connect operators to each other > (StreamingContainer.java does some of this work) > > 2) Each operator contains an output port, output ports in turn have a > BufferServer that persists buffered output data in memory, and on disk (if > configured). Where is this in the code? > > 3) Data is written to the buffer server from a Reservoir. How does data > enter and leave the reservoir? What is responsible for this? > > 4) Data is written from the buffer server to a Reservoir. Again, how does > data enter and leave this reservoir? > > Would appreciate any input, thanks! > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > >
