[CONF] Apache Qpid: Network IO Interface (page edited)

Fri, 26 Jun 2009 03:29:36 -0700

Page Edited : qpid : Network IO Interface

Network IO Interface has been edited by Aidan Skinner (Jun 26, 2009).

(View changes)

Content:
  1. Problem statement
    1. memory usage - unbounded Mina buffers
  2. Overall solution
    1. fix buffer sizes
    2. unprocessed frames in one location only
    3. full network buffers on server use tcp flow control to prevent client writing more
  3. Areas which need to be addressed
    1. one heap per session vs total job heap
    2. push back on processing network traffic to client
    3. client needs to handle pause in network processing without OOM
  4. Specific changes to address those areas
    1. broker does not retain frames outside of the network buffer.
    2. network buffers are of a fixed, configurable, size.
    3. client does not retain frames outside of the network buffer
    4. client network writes block if the buffer is full
    5. see Producer flow control for more details of the higher level aspects of this, IO work is purely focussed on the network layer, not the AMQP semantics.
  5. Current requirements which need to be carried forward
    1. Buffer size tuning
    2. TCP nodelay toggle
    3. SSL - link level encryption, what about cert validation etc?
    4. Protect-io
    5. Multi-io
    6. Write biasing
  6. Plugability
    1. Define interface
      1. Decouple protocol handling from transport
      2. Allow arbitrary plugins to be loaded at runtime ?

Tasks

  1. Common
    1. Define common transport interface
  2. Client side
    1. Port to new common interface
    2. Replace MINA ByteBuffer with nio ByteBuffer for portability
  3. Broker side
    1. Port to new common interface
    2. Replace MINA ByteBuffer with nio ByteBuffer for portability

Design Framework

  1. Objective and Scope.
    1. Overview: The broker is prone to heap exhaustion, leading to OutOfMemory errors. To protect against this Producer flow control will be implemented. In order for flow control to be effective the network transport layer needs to be able to support some new requirements.
    2. Problem Statement: For the broker to effectively manage its memory usage, it needs to be able to know how much is held in the network buffers or at least put an upper bound on it. It also needs to be able to constrain how much memory those buffers are using.
    3. Current Architecture: The current MINA networking uses unbounded buffers. When the broker is unable to process frames as quickly as they are being sent these buffers begin to fill up and the broker has no way to limit those. It also has no way to know how large those buffers are, making memory-based triggers for flow control difficult to implement.
    4. Exclusions: / Assumptions:
  2. High-Level Technical Architecture
    1. Functional Requirements:
    2. Non Functional Requirements:
    3. Architecture Design
      1. Overview of Design
      2. Breakdown of component parts.
  3. Testing Approach:

Powered by Atlassian Confluence (Version: 2.2.9 Build:#527 Sep 07, 2006) - Bug/feature request

Unsubscribe or edit your notifications preferences
--------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]

Reply via email to