Brian, A good way to post material like this is to write it up on the wiki at zeromq.org and then just provide a link by email.
-Pieter On Sat, Mar 15, 2014 at 3:22 AM, Brian Adamson <[email protected]> wrote: > As mentioned in the thread with the question regarding the > "encoder::has_data()" method deprecation, here are some notes on the > "norm_engine" adaptation of the NORM protocol for the current libzmq code. > This a shortened overview of the protocol and our implementation. The sort > of "connection free" transport model and decoupling of reliability, > congestion control, and flow control aspects that NORM provides can be > useful for a number of different ZeroMQ messaging and other application > goals for both multicast _and_ unicast. The initial "norm_engine" somewhat > replicates the capability that the PGM transport provides albeit with > automated congestion control and the highly efficient packet erasure FEC > coding ARQ protocol that NORM uses along with support for unicast as well as > multicast PUB/SUB. > > Notes for the ZeroMQ "norm_engine" > > The "norm_engine.hpp" and "norm_engine.cpp" files provide an implementation > of a NACK-Oriented Reliable Multicast (NORM) transport protocol option for > ZeroMQ. NORM is an IETF open standards protocol specified in RFC 5740 and > supporting documents. The Naval Research Laboratory (NRL) provides and open > source reference implementation that is hosted at > http://www.nrl.navy.mil/itd/ncs/products/norm. > > NORM supports reliable data delivery over IP multicast but also supports > unicast (point to point) data transfers. NORM operates on top of the User > Datagram Protocol (UDP) and supports reliability via NACK-based Automated > Repeat Request (ARQ) that uses packet erasure coding for very efficient > group communication and also provides for automated TCP-friendly congestion > control and mechanisms for support end-to-end flow control. NORM can also > be configured to provide basic UDP-like best effort transport service and > this can be enhanced by adding some amount application-settable proactive > forward error correction (FEC) packets to the transmission. I.e., by > default NORM only sends 'reactive' FEC repair packets in response to NACKs > but can also be configured to proactively send added repair packets for a > level of reliability without any feedback from the receivers. In addition > its TCP-friendly congestion control, NORM can also be configured for > fixed-rate operation and the NRL implementation supports some additional > automated congestion control options suitable for use in bit error prone > wireless communication environments. While its reliable ARQ operation is > principally NACK-based (Negative Acknowledge when packet loss is detected), > it also supports optional positive acknowledgment (ACK) from receivers that > can be used for delivery confirmation and flow control. > > By keeping the end-to-end transport aspects of reliability, congestion > control, and flow control as separable components of protocol operation, > NORM can be configured to meet a wide variety of application data delivery > needs. This ranges from "object-based" delivery of bulky content items like > files and large memory segments to messaging and even real-time, reliable > streaming. NORM has been applied in experimental applications for file > delivery, messaging (including serverless multicast chat and file/image > sharing), TCP flow proxying, and real-time video and even Voice over IP > (VoIP) streaming. > > The NRL implementation provides a procedural API compatible with C/C++ and > also provides Java and Python (pynorm) language bindings. The NORM code can > be built on Linux, BSD/MacOSX, Windows, iOS, Android, and several other Unix > operating system variants. NORM can be built as a static or shared library > (e.g. libnorm.so, .libnorm.dylib, norm.dlll, etc) and a "normApi.h" header > file is provided for the C/C++ API. The "norm_engine" addition to libzmq > uses this API. A comprehensive NORM Developer's reference guide and a > growing number of simple examples is included in the source code > distribution. NRL also hosts a NORM developer's mailing list. > > This initial use of NORM within ZeroMQ supports the ZMQ_PUB/ZMQ_SUB and > ZMQ_XPUB/ZMQ_XSUB socket types in a similar manner to the Pragmatic General > Multicast (PGM) transport option that was already present in the ZeroMQ > code. A notable difference is NORM's support of automated TCP-friendly > congestion control although NORM can also be configured to support > fixed-rate (or bounded-rate) operation if desired. Unicast addresses can > also be used for NORM session addresses and "many-to-one" sender to receiver > relationships can be established with a single NORM ZeroMQ subscriber > receiving unicast messages from multiple senders. The many additional > transport options that NORM provides (e.g. UDP-like service, etc) could also > be supported with extension of ZeroMQ socket options or establishment of > multiple NORM protocol profiles. It is hoped that the ZeroMQ community will > be interested in exploring these options. > > The current NORM code has been tested with the Python "pyzmq" API binding. > The basic transport endpoint format for the "norm_engine" is similar to that > used for PGM consisting of: > > norm://[<interfaceName>;]<addr>:<port> > > where the optional <interfaceName> field provides a means to specify a > specific host network interface for IP multicast group join and packet > transmissions. The <addr> can be an IP unicast or multicast address (NORM > supports IPv4 and IPv6). A ZeroMQ subscriber can use its loopback address > (e.g. 127.0.0,1) for unicast sessions and multiple ZeroMQ publishers can > "connect" to that subscriber. In fact it is even possible for a NORM > multicast subscriber to also terminate unicast connections for mixed-mode > (unicast and multicast) sessions. > > One aspect of the NORM protocol is that NORM session participants must use > unique "NormNodeId" identifiers for the NORM transport. The default > behavior of the NORM implementation is to auto-select a NormNodeId based on > the host IP address. However, if multiple NORM nodes are co-located on the > same host, this can be problematic. As an initial means to resolve this > issue, the "norm_engine" transport endpoint specifier also has an optional > "nodeId" field that can be inserted, with a comma delimiter, as the first > field. Thus, the full endpoint specification format is: > > norm://[<nodeId>,][<interfaceName>;]<addr>:<port> > > where both the <nodeId> and <interfaceName> are optional fields. The > NormNodeId is a 32-bit identifier. Its purpose is similar to the Real-time > Protocol (RTP) synchronization source (SSRC) identifier to provide an > address-independent means to identify the multicast source in potentially > multi-homed or asymmetric network environments. A future version of the > "norm_engine" may use a hash of the ZeroMQ Identity or other means to > establish unique NormNodeId values for multicast participants. > > The following snippets of Python "pyzmq" code illustrates simple NORM-based > subscription and publication (Note this includes the optional "node id" > field so that a loopback test on a single machine will work): > > # Subscriber > import pyzmq > context = zmq.Context() > socket = context.socket(zmq.SUB) > socket.bind("norm://2,224.1.2.3:5556") > socket.setsockopt(zmq.SUBSCRIBE, "ZMQ-Test") > while True: > string = socket.recv() > print string > > # Publisher > import pyzmq > import time > context = zmq.Context() > socket = context.socket(zmq.PUB) > socket.connect("norm://1,224.1.2.3:5556") > i = 1 > while True: > topic = "ZMQ-Test" > message = "Hello, NORM " + str(i) + " ..." > socket.send("%s %s" % (topic, message)) > i += 1 > time.sleep(1) > > Note, as with the PGM transport option, the ZeroMQ socket "connect()" and > "bind()" methods are interchangeable since NORM is a "connection-free" > protocol. I hate to call it "connectionless" since receivers do establish > state on a per-sender basis, but no explicit connection setup handshake is > required to begin data transfer. > > Regarding the "connection" establishment, NORM receivers (ZeroMQ subscribers > in this case) "synchronize" to NORM sender (ZMQ publishers) transmissions > upon reception of NORM packets. Because it was designed for IP multicast, > "late-joining" receivers can synchronize to a NORM sender that is already in > progress. NORM supports multiple receiver "synchronization policies". The > current "norm_engine" is hard coded to use the NORM_SYNC_STREAM policy. > This policy allows the newly joining receiver to request retransmission of > all content the sender has buffered/cached (e.g. possibly to the beginning > of the stream). Another relevant NORM sync policy is NORM_SYNC_CURRENT > where the receiver attempts reliable reception for current and newer content > the sender is transmitting and does not request retransmission of older > content. Synchronization policy choice may be something that future > versions of the ZeroMQ "norm_engine" may wish to expose. Similarly, > sender/receiver buffer and cache sizes (including UDP socket buffer sizes) > are configurable along with many other details of NORM transport operation. > The NORM_OBJECT_STREAM transport mode was selected because it enforces > in-order delivery of transmitted content from the sender to the receiver(s). > NORM also supports other "bulk" object data delivery models that can provide > out-of-order message/item delivery depending on network conditions. > > The following list summarizes some of key features of NORM and this > "norm_engine" extension to libzmq: > > NORM supports reliable multicast and unicast via a highly-efficient > NACK-based ARQ protocol using FEC-based packet erasure coding for repair > transmissions in response to negative acknowledgments. NORM rides on top of > UDP. It can also provide UDP-like "best effort" service and this can be > supplemented with added FEC packet erasure coding packets for something > "better than best effort" at the expense of modest transmission overhead. > NORM supports fixed-rate and automated, TCP-friendly congestion control > operation. NORM also supports some experimental congestion control options > useful for wireless network environments. The automated congestion control > operation can also be rate-bounded (min/max rate excursion) if desired. > NORM's separable reliability, congestion control, and flow control aspects > offer support for a wide range of application data delivery needs. This has > included everything from bulk file delivery via IP multicast to unicast TCP > stream proxying and even real-time quasi-reliable video and VoIP delivery. > The current "norm_engine" assumes a fairly specific form of NORM transport > options. It may be desirable to expose additional NORM features and > transport modalities via the ZeroMQ API. Some examples include: > > Different NORM receiver sync policies and sender/receiver buffering and > cache sizes > Fixed-rate vs. automated congestion control options > Specifying the NORM receiver set to the sender to allow for explicit > positive acknowledgment (ACK) based flow control and delivery confirmation. > While the current "norm_engine" default NORM delivery model is highly > reliable, at more extreme bandwidth*delay, packet loss factors, the > ACK-based flow control provides a higher guarantee of data delivery. The > ACK mechanism NORM uses easily scales well to multicast groups with 100's of > nodes. > NORM's ability to alternatively provide UDP-like "best effort" and "better > than best effort" (using packet erasure coding) delivery services for > applications. This would include control of the NORM FEC encoding > parameters. > NORM file caching, bulk file delivery options > Consideration of NORM use for other ZMQ socket types in addition to pub/sub. > > > The current NORM API is a fairly "low level" where some of the likely useful > "use patterns" (e.g. positive acknowledgment management) are left to the > application programmer to implement. The NORM API will continue to evolve > to expose additional features as needed and to provide more convenience > functions for implementing common "use patterns" in a simpler manner. The > ZeroMQ API provides a simple means to make use of NORM for different > purposes. As the ZeroMQ community considers and explores NORM use, the > evolution of the NORM API and its ZeroMQ adaptation can be expanded. > > Brian Adamson > mailto:[email protected] > 12 March 2014 > On Mar 14, 2014, at 8:11 PM, Steven McCoy <[email protected]> wrote: > > On 14 March 2014 18:43, Brian Adamson <[email protected]> wrote: >> >> Also, I have a short overview/write-up of what I've done with ZeroMQ/NORM >> that I can post to the list if there's interest? >> > > I'd like to see this. I'd like to see what other features of NORM you can > fit in and their value. > > -- > Steve-o > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
