As mentioned in the thread with the question regarding the "encoder::has_data()" method deprecation, here are some notes on the "norm_engine" adaptation of the NORM protocol for the current libzmq code. This a shortened overview of the protocol and our implementation. The sort of "connection free" transport model and decoupling of reliability, congestion control, and flow control aspects that NORM provides can be useful for a number of different ZeroMQ messaging and other application goals for both multicast _and_ unicast. The initial "norm_engine" somewhat replicates the capability that the PGM transport provides albeit with automated congestion control and the highly efficient packet erasure FEC coding ARQ protocol that NORM uses along with support for unicast as well as multicast PUB/SUB.
Notes for the ZeroMQ "norm_engine" The "norm_engine.hpp" and "norm_engine.cpp" files provide an implementation of a NACK-Oriented Reliable Multicast (NORM) transport protocol option for ZeroMQ. NORM is an IETF open standards protocol specified in RFC 5740 and supporting documents. The Naval Research Laboratory (NRL) provides and open source reference implementation that is hosted at http://www.nrl.navy.mil/itd/ncs/products/norm. NORM supports reliable data delivery over IP multicast but also supports unicast (point to point) data transfers. NORM operates on top of the User Datagram Protocol (UDP) and supports reliability via NACK-based Automated Repeat Request (ARQ) that uses packet erasure coding for very efficient group communication and also provides for automated TCP-friendly congestion control and mechanisms for support end-to-end flow control. NORM can also be configured to provide basic UDP-like best effort transport service and this can be enhanced by adding some amount application-settable proactive forward error correction (FEC) packets to the transmission. I.e., by default NORM only sends 'reactive' FEC repair packets in response to NACKs but can also be configured to proactively send added repair packets for a level of reliability without any feedback from the receivers. In addition its TCP-friendly congestion control, NORM can also be configured for fixed-rate operation and the NRL implementation supports some additional automated congestion control options suitable for use in bit error prone wireless communication environments. While its reliable ARQ operation is principally NACK-based (Negative Acknowledge when packet loss is detected), it also supports optional positive acknowledgment (ACK) from receivers that can be used for delivery confirmation and flow control. By keeping the end-to-end transport aspects of reliability, congestion control, and flow control as separable components of protocol operation, NORM can be configured to meet a wide variety of application data delivery needs. This ranges from "object-based" delivery of bulky content items like files and large memory segments to messaging and even real-time, reliable streaming. NORM has been applied in experimental applications for file delivery, messaging (including serverless multicast chat and file/image sharing), TCP flow proxying, and real-time video and even Voice over IP (VoIP) streaming. The NRL implementation provides a procedural API compatible with C/C++ and also provides Java and Python (pynorm) language bindings. The NORM code can be built on Linux, BSD/MacOSX, Windows, iOS, Android, and several other Unix operating system variants. NORM can be built as a static or shared library (e.g. libnorm.so, .libnorm.dylib, norm.dlll, etc) and a "normApi.h" header file is provided for the C/C++ API. The "norm_engine" addition to libzmq uses this API. A comprehensive NORM Developer's reference guide and a growing number of simple examples is included in the source code distribution. NRL also hosts a NORM developer's mailing list. This initial use of NORM within ZeroMQ supports the ZMQ_PUB/ZMQ_SUB and ZMQ_XPUB/ZMQ_XSUB socket types in a similar manner to the Pragmatic General Multicast (PGM) transport option that was already present in the ZeroMQ code. A notable difference is NORM's support of automated TCP-friendly congestion control although NORM can also be configured to support fixed-rate (or bounded-rate) operation if desired. Unicast addresses can also be used for NORM session addresses and "many-to-one" sender to receiver relationships can be established with a single NORM ZeroMQ subscriber receiving unicast messages from multiple senders. The many additional transport options that NORM provides (e.g. UDP-like service, etc) could also be supported with extension of ZeroMQ socket options or establishment of multiple NORM protocol profiles. It is hoped that the ZeroMQ community will be interested in exploring these options. The current NORM code has been tested with the Python "pyzmq" API binding. The basic transport endpoint format for the "norm_engine" is similar to that used for PGM consisting of: norm://[<interfaceName>;]<addr>:<port> where the optional <interfaceName> field provides a means to specify a specific host network interface for IP multicast group join and packet transmissions. The <addr> can be an IP unicast or multicast address (NORM supports IPv4 and IPv6). A ZeroMQ subscriber can use its loopback address (e.g. 127.0.0,1) for unicast sessions and multiple ZeroMQ publishers can "connect" to that subscriber. In fact it is even possible for a NORM multicast subscriber to also terminate unicast connections for mixed-mode (unicast and multicast) sessions. One aspect of the NORM protocol is that NORM session participants must use unique "NormNodeId" identifiers for the NORM transport. The default behavior of the NORM implementation is to auto-select a NormNodeId based on the host IP address. However, if multiple NORM nodes are co-located on the same host, this can be problematic. As an initial means to resolve this issue, the "norm_engine" transport endpoint specifier also has an optional "nodeId" field that can be inserted, with a comma delimiter, as the first field. Thus, the full endpoint specification format is: norm://[<nodeId>,][<interfaceName>;]<addr>:<port> where both the <nodeId> and <interfaceName> are optional fields. The NormNodeId is a 32-bit identifier. Its purpose is similar to the Real-time Protocol (RTP) synchronization source (SSRC) identifier to provide an address-independent means to identify the multicast source in potentially multi-homed or asymmetric network environments. A future version of the "norm_engine" may use a hash of the ZeroMQ Identity or other means to establish unique NormNodeId values for multicast participants. The following snippets of Python "pyzmq" code illustrates simple NORM-based subscription and publication (Note this includes the optional "node id" field so that a loopback test on a single machine will work): # Subscriber import pyzmq context = zmq.Context() socket = context.socket(zmq.SUB) socket.bind("norm://2,224.1.2.3:5556") socket.setsockopt(zmq.SUBSCRIBE, "ZMQ-Test") while True: string = socket.recv() print string # Publisher import pyzmq import time context = zmq.Context() socket = context.socket(zmq.PUB) socket.connect("norm://1,224.1.2.3:5556") i = 1 while True: topic = "ZMQ-Test" message = "Hello, NORM " + str(i) + " ..." socket.send("%s %s" % (topic, message)) i += 1 time.sleep(1) Note, as with the PGM transport option, the ZeroMQ socket "connect()" and "bind()" methods are interchangeable since NORM is a "connection-free" protocol. I hate to call it "connectionless" since receivers do establish state on a per-sender basis, but no explicit connection setup handshake is required to begin data transfer. Regarding the "connection" establishment, NORM receivers (ZeroMQ subscribers in this case) "synchronize" to NORM sender (ZMQ publishers) transmissions upon reception of NORM packets. Because it was designed for IP multicast, "late-joining" receivers can synchronize to a NORM sender that is already in progress. NORM supports multiple receiver "synchronization policies". The current "norm_engine" is hard coded to use the NORM_SYNC_STREAM policy. This policy allows the newly joining receiver to request retransmission of all content the sender has buffered/cached (e.g. possibly to the beginning of the stream). Another relevant NORM sync policy is NORM_SYNC_CURRENT where the receiver attempts reliable reception for current and newer content the sender is transmitting and does not request retransmission of older content. Synchronization policy choice may be something that future versions of the ZeroMQ "norm_engine" may wish to expose. Similarly, sender/receiver buffer and cache sizes (including UDP socket buffer sizes) are configurable along with many other details of NORM transport operation. The NORM_OBJECT_STREAM transport mode was selected because it enforces in-order delivery of transmitted content from the sender to the receiver(s). NORM also supports other "bulk" object data delivery models that can provide out-of-order message/item delivery depending on network conditions. The following list summarizes some of key features of NORM and this "norm_engine" extension to libzmq: NORM supports reliable multicast and unicast via a highly-efficient NACK-based ARQ protocol using FEC-based packet erasure coding for repair transmissions in response to negative acknowledgments. NORM rides on top of UDP. It can also provide UDP-like "best effort" service and this can be supplemented with added FEC packet erasure coding packets for something "better than best effort" at the expense of modest transmission overhead. NORM supports fixed-rate and automated, TCP-friendly congestion control operation. NORM also supports some experimental congestion control options useful for wireless network environments. The automated congestion control operation can also be rate-bounded (min/max rate excursion) if desired. NORM's separable reliability, congestion control, and flow control aspects offer support for a wide range of application data delivery needs. This has included everything from bulk file delivery via IP multicast to unicast TCP stream proxying and even real-time quasi-reliable video and VoIP delivery. The current "norm_engine" assumes a fairly specific form of NORM transport options. It may be desirable to expose additional NORM features and transport modalities via the ZeroMQ API. Some examples include: Different NORM receiver sync policies and sender/receiver buffering and cache sizes Fixed-rate vs. automated congestion control options Specifying the NORM receiver set to the sender to allow for explicit positive acknowledgment (ACK) based flow control and delivery confirmation. While the current "norm_engine" default NORM delivery model is highly reliable, at more extreme bandwidth*delay, packet loss factors, the ACK-based flow control provides a higher guarantee of data delivery. The ACK mechanism NORM uses easily scales well to multicast groups with 100's of nodes. NORM's ability to alternatively provide UDP-like "best effort" and "better than best effort" (using packet erasure coding) delivery services for applications. This would include control of the NORM FEC encoding parameters. NORM file caching, bulk file delivery options Consideration of NORM use for other ZMQ socket types in addition to pub/sub. The current NORM API is a fairly "low level" where some of the likely useful "use patterns" (e.g. positive acknowledgment management) are left to the application programmer to implement. The NORM API will continue to evolve to expose additional features as needed and to provide more convenience functions for implementing common "use patterns" in a simpler manner. The ZeroMQ API provides a simple means to make use of NORM for different purposes. As the ZeroMQ community considers and explores NORM use, the evolution of the NORM API and its ZeroMQ adaptation can be expanded. Brian Adamson mailto:[email protected] 12 March 2014 On Mar 14, 2014, at 8:11 PM, Steven McCoy <[email protected]> wrote: > On 14 March 2014 18:43, Brian Adamson <[email protected]> wrote: > Also, I have a short overview/write-up of what I've done with ZeroMQ/NORM > that I can post to the list if there's interest? > > > I'd like to see this. I'd like to see what other features of NORM you can > fit in and their value. > > -- > Steve-o > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
