We are investigating using ATS for the routing tier of a distributed database system. The client protocol is HTTP. A client sends an HTTP request to ATS which inspects the request URL and dispatches it to one or more of a collection of servers.
Routing of discrete requests between a client and a single selected server is a piece of cake and is working well. Where we are having trouble is with scatter-gather requests. Certain API calls can't be satisfied from a single server and require that the router (ATS) respond to an HTTP client request by dispatching multiple HTTP requests to the storage tier, then consolidate the results that come back. The response back to the client is sent with a multi-part MIME content type and a chunked encoding. The first server response sends an outer HTTP header and the initial return payload a MIME part. Subsequent responses just wrap the response payload with a MIME boundary and stream the results back. ATS graciously provides the chunked encoding of the output stream on our behalf. The goal is to not buffer all responses before starting to send a reply. The client is free to time out before the last server has responded and just process the results received so far. Here is basically what we did. The downstream requests are made via calling TSFetchPages where TSFetchUrlParams associates the outbound request back to the incoming request context through an intermediate object. There are two distinct callbacks registered: * ClientWriterCB is triggered when a write back to the client completes * ServerReaderCB is triggered each time a read from one of the N servers completes The first invocation of ServerReaderCB initiates a write back to the client via TSIOBufferWrite(clientConnection->vconn,...). We discovered that if each subsequent ServerReaderCB just called TSIOBufferWrite(clientConnection->vconn,...) [with the hopes of appending to the response buffer] the responses became intermixed in the client stream. It looks like TSIOBufferWrite just overwrites the contained m_write_vio's buffer reader. If the prior result wasn't completely consumed, it's just truncated. We went down a dead-end of constructing an intermediary buffer that the the ServerReaderCBs would write to and the the ClientWriterCB would consume from, but never quite worked out all the locking and callback scheduling kinks. There has to be a simpler way. How would you recommend implementing such scatter-gather logic? Ideally there would be a way to avoid the extra intermediary buffer. What I want is something akin to the iocore OneWayTunnel that takes a dynamic queue of source VConnections and one destination VConnection and does the transfer. Alternately, a way to take a TSIOBuffer from the ServerReaderCB and just append the TSIOBufferBlocks onto the list of read buffer of the client's outbound vconn would be great. Suggestions? Thanks, Tom Quiggle LinkedIn
