On 27 May 2015 at 11:08, Ola Liljedahl <[email protected]> wrote:
> Here is my third attempt at a ODP API for MSGIO - asynchronous message > passing > based IPC for a shared nothing architecture. > > IPC/message passing can be separated into a number of different layers. > 1) Message formats and protocols (as seen by the user). > 2) API - defines syntax and semantics of MSGIO operations. > 3) Transport interface - defines a binary packet format used by MSGIO > endpoints/implementations. Cf. OSI network layer. > 4) Transport layer - implements the transport of message packet between > different endpoints. Cf. OSI link layer. > > #1 is application specific and there are many different standards (e.g. > Linux > Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases. > #3 may be implementation specific but is important for interoperatibility > (binary compatibility). If the user message (payload), operation and > parameters are encoded in a well-defined way (e.g. some kind of packet > format > which includes operation, sender and receiver addresses, payload etc), > inter-operatibility between different ODP/MSGIO implementations may be > achieved or simplified. > #4 is implementation (platform) specific. > > This proposal focuses on the API which is fundamental for source code > compatibility between different platforms (ODP implementations) - this > being > the primary purpose of ODP. A packet format for messages will be specified > as > part of the reference implementation and may be seen as a proposal for a > standard packet format for binary compatibility between different ODP/MSGIO > implementations. > > All MSGIO calls are non-blocking (with the possible exception of > odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message > transfer is asynchronous so it is not possible to return any status > operation concerning the eventual (or remote) result of an operation. > > The semantics of MSGIO message passing is that sending a message to an > endpoint > will always look like it succeeds. The appearance of endpoints is > explicitly > notified through user-defined messages specified in the odp_msgio_lookup() > call. Similarly, the disappearance (e.g. death of or otherwise lost > connection > to endpoint) is also explicitly notified through user-defined messages > specified in the odp_msgio_monitor() call. The send call does not fail > because > the addressed endpoint has disappeared, the message is just silently > discarded. > > Message delivery into the recipient address space is ordered (per priority) > and reliable. Delivery of message N implies delivery of all messages <N > (of the same priority). All messages (accepted by MSGIO send operation) > will > be delivered up to the point of endpoint termination or lost connection > where no more messages will be delivered. Actual reception (dequeueing) and > processing by the recipient is not guaranteed (use end-to-end > acknowledgements > for that). > > MSGIO endpoints can be seen as interfaces (taps) to an internal reliable > multidrop network where each endpoint has a unique address which is only > valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed > and then recreated (with the same name), the new endpoint will have a > new unique address (eventually endpoints addresses will have to be recycled > but not for a very long time). Endpoints names do not necessarily have to > be > unique. The scope of message network is not defined but it is expected that > the scope by default corresponds to an OS instance (e.g. virtual machine). > > Proposed transport packet for message I/O (not visible to the user): > uint8_t dest_addr[6]; /* Unique address of destination endpoint */ > uint8_t src_addr[6]; /* Unique address of source endpoint */ > uint16_t frametype_o; /* Outer frametype, 0x8100 */ > uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */ > uint16_t frametype_i; /* Inner frametype, TBD */ > uint8_t payload[]; /* User message */ > Looks very much like a VLAN-tagged Ethernet frame so should easily be > transported by and processed by Ethernet HW & SW Why to use Ethernet HW & SW to transport messages inside an OS instance? Usually Ethernet HW & SW (drivers?) deal with sending/receiving to/from the wire. An above requirement was referring to ordering which must be defined by the transport layer - how this format ensures ordered delivery at the destination? > . > Encoding of publish/lookup/monitor operations TBD. > > v3: > Renamed Message Bus (MBUS) to Message I/O (MSGIO). > Changed all definitions to use "msgio" instead of "mbus" prefix/infix. > odp_msgio_create(): removed default input queue parameter. > Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef(). > All input queues must now be specified using odp_msgio_inq_set() before > the MSGIO endpoint is activated. > Added odp_msgio_activate(). > Added odp_message_push_head() and odp_message_pull_head(). > Updated some function descriptions in msgio.h. > > v2: > Split off all message definitions in a separate file message.h > Renamed Inter Process Communication (IPC) to Message Bus (MBUS). > Changed all definitions to use "mbus" instead of "ipc" prefix/infix. > Renamed odp_ipc_msg_t to odp_message_t. > odp_mbus_create(): Added parameter for default input queue. Explicitly > state > that the pool must be use type ODP_EVENT_MESSAGE. > Renamed odp_ipc_resolve() to odp_mbus_lookup(). > odp_mbus_send(): Added priority parameter. > Renamed odp_ipc_sender() to odp_message_sender(). > Renamed odp_ipc_data() to odp_message_data(). > Renamed odp_ipc_length() to odp_message_length(). > Renamed odp_ipc_reset() to odp_message_length_set(). > Renamed odp_ipc_alloc() to odp_message_alloc(). > Renamed odp_ipc_free() to odp_message_free(). > odp_message_alloc(): Corrected name of invalid message handle. > Added message priorities and calls to set and remove input queues for > specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem(). > > Signed-off-by: Ola Liljedahl <[email protected]> > --- > (This document/code contribution attached is provided under the terms of > agreement LES-LTM-21309) > > include/odp/api/message.h | 199 > ++++++++++++++++++ > include/odp/api/msgio.h | 222 > +++++++++++++++++++++ > platform/linux-generic/include/odp/message.h | 39 ++++ > platform/linux-generic/include/odp/msgio.h | 40 ++++ > .../linux-generic/include/odp/plat/message_types.h | 47 +++++ > .../linux-generic/include/odp/plat/msgio_types.h | 59 ++++++ > 6 files changed, 606 insertions(+) > create mode 100644 include/odp/api/message.h > create mode 100644 include/odp/api/msgio.h > create mode 100644 platform/linux-generic/include/odp/message.h > create mode 100644 platform/linux-generic/include/odp/msgio.h > create mode 100644 platform/linux-generic/include/odp/plat/message_types.h > create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h > > diff --git a/include/odp/api/message.h b/include/odp/api/message.h > new file mode 100644 > index 0000000..76697fb > --- /dev/null > +++ b/include/odp/api/message.h > @@ -0,0 +1,199 @@ > +/* Copyright (c) 2015, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > + > +/** > + * @file > + * > + * ODP Message event type > + */ > + > +#ifndef ODP_API_MESSAGE_H_ > +#define ODP_API_MESSAGE_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +/** @defgroup odp_message ODP MESSAGE > + * @{ > + */ > + > +/** > + * @typedef odp_message_t > + * ODP message handle > + */ > + > + > +/** > + * Get address of sender (source) of message > + * > + * @param msg Message handle > + * @param[out] addr Buffer to hold sender address > + */ > +void odp_message_sender(odp_message_t msg, > + uint8_t addr[ODP_MSGIO_ADDR_SIZE]); > + > +/** > + * Message data pointer > + * > + * Return a pointer to the message data > + * > + * @param msg Message handle > + * > + * @return Pointer to the message data > + */ > +void *odp_message_data(odp_message_t msg); > + > +/** > + * Message data length > + * > + * Return length of the message data. > + * > + * @param msg Message handle > + * > + * @return Message length > + */ > +uint32_t odp_message_length(const odp_message_t msg); > + > +/** > + * Set message length > + * > + * Set length of the message data. > + * Increase message data length by moving message tail into message > tailroom. > + * The actual message data is not modified. > + * > + * @param msg Message handle > + * @param len New length > + * > + * @retval 0 on success > + * @retval <0 on error (e.g. no enough tailroom) > + */ > +int odp_message_length_set(const odp_message_t msg, uint32_t len); > + > +/** > + * Message headroom length > + * > + * Return length of the message headroom > + * > + * @param msg Message handle > + * > + * @return Headroom length > + */ > +uint32_t odp_message_headroom(const odp_message_t msg); > + > +/** > + * Push out message head > + * > + * Increase message data length by moving message head into message > headroom. > + * Message headroom is decreased with the same amount. The message head > may be > + * pushed out up to 'headroom' bytes. Message is not modified if there's > not > + * enough headroom space. > + * > + * odp_message_xxx: > + * length += len > + * headroom -= len > + * data -= len > + * > + * @param msg Message handle > + * @param len Number of bytes to push the head (0 ... headroom) > + * > + * @return The new data pointer > + * @retval NULL Requested offset exceeds available headroom > + * > + * @see odp_message_headroom(), odp_message_pull_head() > + */ > +void *odp_message_push_head(odp_message_t msg, uint32_t len); > + > +/** > + * Pull in message head > + * > + * Decrease message data length by moving message head into message data. > + * Message headroom is increased with the same amount. The message head > may be > + * pulled in up to 'length' bytes. Message is not modified if there's not > + * enough data. > + * > + * odp_message_xxx: > + * length -= len > + * headroom += len > + * data += len > + * > + * @param msg Message handle > + * @param len Number of bytes to pull the head (0 ... length) > + * > + * @return The new data pointer > + * @retval NULL Requested offset exceeds available data > + * > + * @see odp_message_headroom(), odp_message_push_head() > + */ > +void *odp_message_pull_head(odp_message_t msg, uint32_t len); > + > +/** > + * Allocate message > + * > + * Allocate a message of a specific length. > + * > + * @param pool Message pool to allocate message from > + * @param len Length of the allocated message > + * > + * @return Message handle on success > + * @retval ODP_MESSAGE_INVALID on failure and errno set > + */ > +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len); > + > +/** > + * Free message > + * > + * Free message back to the message pool it was allocated from. > + * > + * @param msg Handle of message to free > + */ > +void odp_message_free(odp_message_t msg); > + > +/** > + * Get message handle from event > + * > + * Converts an ODP_EVENT_MESSAGE type event to a message. > + * > + * @param ev Event handle representing a message. > + * > + * @return Message handle > + * > + * @see odp_event_type() > + */ > +odp_message_t odp_message_from_event(odp_event_t ev); > + > +/** > + * Convert message handle to event > + * > + * @param msg Message handle > + * > + * @return Event handle > + */ > +odp_event_t odp_message_to_event(odp_message_t msg); > + > +/** > + * Get printable value for an odp_message_t > + * > + * @param msg Message handle to be printed > + * @return uint64_t value that can be used to print/display this > + * handle > + * > + * @note This routine is intended to be used for diagnostic purposes > + * to enable applications to generate a printable value that represents > + * an odp_message_t handle. > + */ > +uint64_t odp_message_to_u64(odp_message_t msg); > + > +/** > + * @} > + */ > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h > new file mode 100644 > index 0000000..8d73021 > --- /dev/null > +++ b/include/odp/api/msgio.h > @@ -0,0 +1,222 @@ > +/* Copyright (c) 2015, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > + > +/** > + * @file > + * > + * ODP Message I/O API > + */ > + > +#ifndef ODP_API_MSGIO_H_ > +#define ODP_API_MSGIO_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +/** @defgroup odp_msgio ODP MSGIO > + * @{ > + */ > + > +/** > + * @typedef odp_msgio_t > + * ODP message I/O handle > + */ > + > +/** > + * @def ODP_MSGIO_ADDR_SIZE > + * Size of the address of a message I/O endpoint > + */ > + > +/** > + * @typedef odp_msgio_prio_t > + * ODP MSGIO message priority > + */ > + > +/** > + * @def ODP_MSGIO_PRIO_HIGHEST > + * Highest MSGIO message priority > + */ > + > +/** > + * @def ODP_MSGIO_PRIO_NORMAL > + * Normal MSGIO message priority > + */ > + > +/** > + * @def ODP_MSGIO_PRIO_LOWEST > + * Lowest MSGIO message priority > + */ > + > +/** > + * @def ODP_MSGIO_PRIO_DEFAULT > + * Default MSGIO message priority > + */ > + > + > +/** > + * Create message I/O endpoint > + * > + * Create an message I/O endpoint. The scope of the message bus is > + * not defined but it is expected that it by default encompasses the OS > + * instance but no more. > + * > + * A unique address for the endpoint is created. The addresses for other > + * endpoints can be obtained using the odp_msgio_lookup() call and > specified > + * in the odp_msgio_send() call. > + * > + * @param name Name of our endpoint > + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages > + * > + * @return Message I/O handle on success > + * @retval ODP_MSGIO_INVALID on failure and errno set > + */ > +odp_msgio_t odp_msgio_create(const char *name, > + odp_pool_t pool); > + > +/** > + * Set the input queue for a specific message priority > + * > + * @param msgio Message I/O handle > + * @param prio Message priority > + * @param queue Queue handle > + * > + * @retval 0 on success > + * @retval <0 on failure > + */ > +int odp_msgio_inq_set(odp_msgio_t msgio, > + odp_msgio_prio_t prio, > + odp_queue_t queue); > + > +/** > + * Activate message I/O endpoint > + * > + * Make this endpoint visible (odp_msgio_lookup() messages will be > returned). > + * Enable transmission and reception of messages through the endpoint. > + * > + * @param msgio Message I/O handle > + * > + * @retval 0 on success > + * @retval <0 on failure > + */ > +int odp_msgio_activate(odp_msgio_t msgio); > + > +/** > + * Destroy message I/O endpoint > + * > + * @param msgio Message I/O handle > + * > + * @retval 0 on success > + * @retval <0 on failure > + */ > +int odp_msgio_destroy(odp_msgio_t msgio); > + > +/** > + * Remove the input queue for a specific message priority > + * > + * Remove (disassociate) the matching input queue from a message I/O > endpoint. > + * The queue itself is not touched. > + * > + * @param msgio Message I/O handle > + * @param prio Message priority > + * > + * @retval 0 on success > + * @retval <0 on failure > + */ > +int odp_msgio_inq_rem(odp_msgio_t msgio, > + odp_msgio_prio_t prio); > + > +/** > + * Lookup endpoint by name > + * > + * Look up a current or future endpoint by name. > + * When the endpoint exists, return the specified message (not > necessarily the > + * same message handle) with the endpoint as the sender. > + * > + * @param msgio Message I/O handle > + * @param name Name to look up > + * @param msg Message to return > + */ > +void odp_msgio_lookup(odp_msgio_t msgio, > + const char *name, > + odp_message_t msg); > + > +/** > + * Monitor endpoint > + * > + * Monitor an endpoint (which may already have disappeared). > + * When the endpoint disappears, return the specified message (not > necessarily > + * the same message handle) with the endpoint as the sender. > + * > + * Unrecognized or invalid endpoint addresses are treated as non-existing > + * endpoints. > + * > + * @param msgio Message I/O handle > + * @param[in] addr Address of monitored endpoint > + * @param msg Message to return > + */ > +void odp_msgio_monitor(odp_msgio_t msgio, > + const uint8_t addr[ODP_MSGIO_ADDR_SIZE], > + odp_message_t msg); > + > +/** > + * Send message > + * > + * Send a message to an endpoint (which may no longer exist). > + * > + * Message delivery into the recipient address space is ordered (per > priority) > + * and reliable. Delivery of message N implies delivery of all messages <N > + * (of the same priority). > + * All messages (accepted by MSGIO) will be delivered up to the point of > + * recipient endpoint termination or lost connection where no more > messages > + * will be delivered. > + * > + * Actual reception (dequeueing) and processing by the recipient is not > + * guaranteed (use user level acknowledgements for that). > + * > + * Monitor the remote endpoint to detect the disappearance or > disconnection > + * of the endpoint. > + * > + * A message may temporarily not be accepted for transmission (e.g. due > + * to out of local buffer space), -1 will be returned and ODP errno set > + * to EAGAIN. > + * > + * @param msgio Message I/O handle > + * @param msg Message to send > + * @param prio Priority of message > + * @param[in] addr Address of recipient endpoint > + * > + * @retval 0 on success > + * @retval <0 on error > + */ > +int odp_msgio_send(odp_msgio_t msgio, > + odp_message_t msg, > + odp_msgio_prio_t prio, > + const uint8_t addr[ODP_MSGIO_ADDR_SIZE]); > + > +/** > + * Get printable value for an odp_msgio_t > + * > + * @param msgio Message I/O handle to be printed > + * @return uint64_t value that can be used to print/display this > + * handle > + * > + * @note This routine is intended to be used for diagnostic purposes > + * to enable applications to generate a printable value that represents > + * an odp_msgio_t handle. > + */ > +uint64_t odp_msgio_to_u64(odp_msgio_t msgio); > + > +/** > + * @} > + */ > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/platform/linux-generic/include/odp/message.h > b/platform/linux-generic/include/odp/message.h > new file mode 100644 > index 0000000..2a29f59 > --- /dev/null > +++ b/platform/linux-generic/include/odp/message.h > @@ -0,0 +1,39 @@ > +/* Copyright (c) 2013, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > +/** > + * @file > + * > + * ODP Message Bus > + */ > + > +#ifndef ODP_PLAT_MESSAGE_H_ > +#define ODP_PLAT_MESSAGE_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <odp/std_types.h> > +#include <odp/plat/pool_types.h> > +#include <odp/plat/message_types.h> > +#include <odp/plat/queue_types.h> > + > +/** @ingroup odp_message > + * @{ > + */ > + > +/** > + * @} > + */ > + > +#include <odp/api/message.h> > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/platform/linux-generic/include/odp/msgio.h > b/platform/linux-generic/include/odp/msgio.h > new file mode 100644 > index 0000000..f45b3a2 > --- /dev/null > +++ b/platform/linux-generic/include/odp/msgio.h > @@ -0,0 +1,40 @@ > +/* Copyright (c) 2013, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > +/** > + * @file > + * > + * ODP Message Bus > + */ > + > +#ifndef ODP_PLAT_MSGIO_H_ > +#define ODP_PLAT_MSGIO_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <odp/std_types.h> > +#include <odp/plat/pool_types.h> > +#include <odp/plat/msgio_types.h> > +#include <odp/plat/message_types.h> > +#include <odp/plat/queue_types.h> > + > +/** @ingroup odp_msgio > + * @{ > + */ > + > +/** > + * @} > + */ > + > +#include <odp/api/msgio.h> > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/platform/linux-generic/include/odp/plat/message_types.h > b/platform/linux-generic/include/odp/plat/message_types.h > new file mode 100644 > index 0000000..3a42064 > --- /dev/null > +++ b/platform/linux-generic/include/odp/plat/message_types.h > @@ -0,0 +1,47 @@ > +/* Copyright (c) 2015, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > + > +/** > + * @file > + * > + * ODP Message message type > + */ > + > +#ifndef ODP_MESSAGE_TYPES_H_ > +#define ODP_MESSAGE_TYPES_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <odp/std_types.h> > +#include <odp/plat/strong_types.h> > + > +/** @addtogroup odp_message ODP message > + * Operations on a message. > + * @{ > + */ > + > +typedef ODP_HANDLE_T(odp_message_t); > + > +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff) > + > +/** Get printable format of odp_message_t */ > +static inline uint64_t odp_message_to_u64(odp_message_t hdl) > +{ > + return _odp_pri(hdl); > +} > + > +/** > + * @} > + */ > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h > b/platform/linux-generic/include/odp/plat/msgio_types.h > new file mode 100644 > index 0000000..9d9f9da > --- /dev/null > +++ b/platform/linux-generic/include/odp/plat/msgio_types.h > @@ -0,0 +1,59 @@ > +/* Copyright (c) 2015, Linaro Limited > + * All rights reserved. > + * > + * SPDX-License-Identifier: BSD-3-Clause > + */ > + > + > +/** > + * @file > + * > + * ODP Message bus types > + */ > + > +#ifndef ODP_MSGIO_TYPES_H_ > +#define ODP_MSGIO_TYPES_H_ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <odp/std_types.h> > +#include <odp/plat/strong_types.h> > + > +/** @addtogroup odp_msgio ODP message bus > + * Operations on a message bus. > + * @{ > + */ > + > +typedef int odp_msgio_prio_t; > + > +#define ODP_MSGIO_PRIO_HIGHEST 0 > + > +#define ODP_MSGIO_PRIO_NORMAL 2 > + > +#define ODP_MSGIO_PRIO_LOWEST 3 > + > +#define ODP_MSGIO_PRIO_DEFAULT ODP_MSGIO_PRIO_NORMAL > + > +typedef ODP_HANDLE_T(odp_msgio_t); > + > +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff) > + > +#define ODP_MSGIO_ADDR_SIZE 6 > + > +/** Get printable format of odp_msgio_t */ > +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl) > +{ > + return _odp_pri(hdl); > +} > + > +/** > + * @} > + */ > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > -- > 1.9.1 > > _______________________________________________ > lng-odp mailing list > [email protected] > https://lists.linaro.org/mailman/listinfo/lng-odp >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
