[CONF] Apache Qpid: Network IO Interface (page edited)

[confluence]

Page Edited : qpid : Network IO Interface Network IO Interface has been edited by Aidan Skinner (Jun 29, 2009). (View changes) Content: Objective and Scope. Overview Problem Statement Current Architecture Exclusions: / Assumptions High-Level Technical Architecture Functional Requirements Non Functional Requirements Architecture Design Overview of Design Breakdown of component parts. Testing Impact Compatibility / Migration Implications Risks Objective and Scope. Overview The broker is prone to heap exhaustion, leading to OutOfMemory errors. The servers main memory usage is storing messages on queues, Producer flow control is needed to manage that area. The other major source of memory consumption are the network buffers used to hold unprocessed AMQP frames once they have been read from the socket. These are currently able to grow uninhibited, and there is no means available to control them. Further, the transport layer is poorly implemented and difficult to work in. Improving encapsulation is an explicit goal of this work. Problem Statement For the broker to effectively manage its memory usage, it needs to be able to at least place an upper bound on the size of it's network buffers. When the broker is unable to process frames as quickly as they are being sent these buffers begin to fill up and the broker has no way to limit those. It also has no way to know how large those buffers are. Current Architecture The current MINA networking uses unbounded buffers. We replace over a dozen MINA classes, none of which have any unit test coverage. We failed to get our patches upstream and haven't attempted since then. Existing unit test coverage is minimal (approx 30%) Improving unit test coverage is difficult due to poor encapsulation Poor encapsulation has lead to tight coupling of MINA to server The current behaviour of send() leaves the potential for message loss when not using transactions and violates JMS expectations. Persistent messages which are held in either the client or servers buffers before being written to disk can be lost. MINA's internal state is currently a black box, leaving no way to determine how much memory is being used by an individual client connection. The way that we use MINA is suboptimal for our purpouses but is difficult to change due to the tight coupling Supporting alternative transport layers is impossible due to tight coupling of MINA (OSI layer 4) with the AMQP handlers (OSI layer 7). Exclusions: / Assumptions No AMQP semantics are involved. The aim of this work is purely to limt the size of the network buffers between the client producing AMQP frames and the broker processing them. It does not involve any protocol specific work. In OSI terms, this work is aimed at layer 4, not layer 7. Higher level information should be determined by the broker itself. No policy will be applied beyond blocking reads if the buffer is full. Buffers are sized uniformly across all connections Buffers are fixed at startup and do not change TCP flow control is the only mechanism used to signal to the client that it should cease to send data High-Level Technical Architecture Functional Requirements Buffer size control TCP options: nodelay, keepalive, window size etc. SSL: link level encryption, do we want to consider things like certificate validation etc here or at a higher level? Log when tcp flow control kicks in Information available about current memory usage through to JMX interface Dynamic removal of buffer bounds on server Non Functional Requirements Runtime loading of transport plugins Inactive connections do not allocate max buffer size uneccessarily (broker only, client can probably live with that) the semantics of org.apache.qpid.BasicMessageProducer.send() need to change. It may now block if there isn't enough free space to write the entire message out. The change to this methods semantics needs to be considered in the light of the stated JMS semantics and the change to support acknowledgement of publishes in AMQP 0-10 and higher. Architecture Design Common should have an interface which all transport plugins can implement and which the server and client can use. The interface would include a means to set the standard socket options and to limit it's incoming memory usage. The interface would return Overview of Design This interface should be quite simple. Methods to send, receive, flush, close, open and a method to set TCP options is likely to be sufficient. These would operate on a QpidByteBuffer, essentially MinaByteBuffer to avoid having to fix our use of expanding buffers at the same time. When processing the incoming data, one frame at a time will be processed and that frames processing will be completed before the next one is read. There will be no other data structures used to hold unprocessed frames. Breakdown of component parts. Common holds a transport layer interface which the existing MINA transport is ported too. We will also port the 0-10 client o.a.q.transport.network.io package to that interface. The server and client both use common for their network layer, and will need to be updated to use the new interface. They will need to pass through the configured socket options. The server will need to be substantively modified to push the MINA specific parts into the appropriate plugin. Testing Testing under load and handling error conditions (unexpected disconnection etc) will need to be carried out. New unit tests will need to be written to cover the transport plugins and the new interfaces. Existing test coverage in this area is minimal. Impact There is a potential effect upon performance, we will need to measure this once it has been implemented to quantify what effect, if any, it has had. Compatibility / Migration Implications Older clients connected to a new broker may suffer OOM when tcp flow control kicks in. This seems preferrable to the broker suffering OOM. Clients which upgrade their library may experience a change in behaviour of the send() method, since it may now block if the clients network buffer is full. This needs to be fully communicated. Risks MINA is quite deeply embedded in the server and will require some work to excise it fully. This is somewhat mitigated by the decision to import mina.ByteBuffer and continue using that. Differences in behaviour of transport layer may expose other bugs in the broker which were being hidden before. Inadequate test coverage, in particular the lack of representative application workloads in the performance test suite.

Powered by Atlassian Confluence (Version: 2.2.9 Build:#527 Sep 07, 2006) - Bug/feature request

Unsubscribe or edit your notifications preferences

--------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:commits-subscr...@qpid.apache.org