Here is my third attempt at a ODP API for MSGIO - asynchronous message passing based IPC for a shared nothing architecture.
IPC/message passing can be separated into a number of different layers. 1) Message formats and protocols (as seen by the user). 2) API - defines syntax and semantics of MSGIO operations. 3) Transport interface - defines a binary packet format used by MSGIO endpoints/implementations. Cf. OSI network layer. 4) Transport layer - implements the transport of message packet between different endpoints. Cf. OSI link layer. #1 is application specific and there are many different standards (e.g. Linux Netlink, NETCONF, OpenFlow, YouNameIt) intended for different use cases. #3 may be implementation specific but is important for interoperatibility (binary compatibility). If the user message (payload), operation and parameters are encoded in a well-defined way (e.g. some kind of packet format which includes operation, sender and receiver addresses, payload etc), inter-operatibility between different ODP/MSGIO implementations may be achieved or simplified. #4 is implementation (platform) specific. This proposal focuses on the API which is fundamental for source code compatibility between different platforms (ODP implementations) - this being the primary purpose of ODP. A packet format for messages will be specified as part of the reference implementation and may be seen as a proposal for a standard packet format for binary compatibility between different ODP/MSGIO implementations. All MSGIO calls are non-blocking (with the possible exception of odp_msgio_create, odp_msgio_activate and odp_msgio_destroy). Message transfer is asynchronous so it is not possible to return any status operation concerning the eventual (or remote) result of an operation. The semantics of MSGIO message passing is that sending a message to an endpoint will always look like it succeeds. The appearance of endpoints is explicitly notified through user-defined messages specified in the odp_msgio_lookup() call. Similarly, the disappearance (e.g. death of or otherwise lost connection to endpoint) is also explicitly notified through user-defined messages specified in the odp_msgio_monitor() call. The send call does not fail because the addressed endpoint has disappeared, the message is just silently discarded. Message delivery into the recipient address space is ordered (per priority) and reliable. Delivery of message N implies delivery of all messages <N (of the same priority). All messages (accepted by MSGIO send operation) will be delivered up to the point of endpoint termination or lost connection where no more messages will be delivered. Actual reception (dequeueing) and processing by the recipient is not guaranteed (use end-to-end acknowledgements for that). MSGIO endpoints can be seen as interfaces (taps) to an internal reliable multidrop network where each endpoint has a unique address which is only valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed and then recreated (with the same name), the new endpoint will have a new unique address (eventually endpoints addresses will have to be recycled but not for a very long time). Endpoints names do not necessarily have to be unique. The scope of message network is not defined but it is expected that the scope by default corresponds to an OS instance (e.g. virtual machine). Proposed transport packet for message I/O (not visible to the user): uint8_t dest_addr[6]; /* Unique address of destination endpoint */ uint8_t src_addr[6]; /* Unique address of source endpoint */ uint16_t frametype_o; /* Outer frametype, 0x8100 */ uint16_t pcp_vid; /* 3 bits of Priority Code Point, 5 bits reserved */ uint16_t frametype_i; /* Inner frametype, TBD */ uint8_t payload[]; /* User message */ Looks very much like a VLAN-tagged Ethernet frame so should easily be transported by and processed by Ethernet HW & SW. Encoding of publish/lookup/monitor operations TBD. v3: Renamed Message Bus (MBUS) to Message I/O (MSGIO). Changed all definitions to use "msgio" instead of "mbus" prefix/infix. odp_msgio_create(): removed default input queue parameter. Removed odp_mbus_inq_setdef() and odp_mbus_inq_remdef(). All input queues must now be specified using odp_msgio_inq_set() before the MSGIO endpoint is activated. Added odp_msgio_activate(). Added odp_message_push_head() and odp_message_pull_head(). Updated some function descriptions in msgio.h. v2: Split off all message definitions in a separate file message.h Renamed Inter Process Communication (IPC) to Message Bus (MBUS). Changed all definitions to use "mbus" instead of "ipc" prefix/infix. Renamed odp_ipc_msg_t to odp_message_t. odp_mbus_create(): Added parameter for default input queue. Explicitly state that the pool must be use type ODP_EVENT_MESSAGE. Renamed odp_ipc_resolve() to odp_mbus_lookup(). odp_mbus_send(): Added priority parameter. Renamed odp_ipc_sender() to odp_message_sender(). Renamed odp_ipc_data() to odp_message_data(). Renamed odp_ipc_length() to odp_message_length(). Renamed odp_ipc_reset() to odp_message_length_set(). Renamed odp_ipc_alloc() to odp_message_alloc(). Renamed odp_ipc_free() to odp_message_free(). odp_message_alloc(): Corrected name of invalid message handle. Added message priorities and calls to set and remove input queues for specific priorities: odp_mbus_inq_set(), odp_mbus_inq_rem(). Signed-off-by: Ola Liljedahl <[email protected]> --- (This document/code contribution attached is provided under the terms of agreement LES-LTM-21309) include/odp/api/message.h | 199 ++++++++++++++++++ include/odp/api/msgio.h | 222 +++++++++++++++++++++ platform/linux-generic/include/odp/message.h | 39 ++++ platform/linux-generic/include/odp/msgio.h | 40 ++++ .../linux-generic/include/odp/plat/message_types.h | 47 +++++ .../linux-generic/include/odp/plat/msgio_types.h | 59 ++++++ 6 files changed, 606 insertions(+) create mode 100644 include/odp/api/message.h create mode 100644 include/odp/api/msgio.h create mode 100644 platform/linux-generic/include/odp/message.h create mode 100644 platform/linux-generic/include/odp/msgio.h create mode 100644 platform/linux-generic/include/odp/plat/message_types.h create mode 100644 platform/linux-generic/include/odp/plat/msgio_types.h diff --git a/include/odp/api/message.h b/include/odp/api/message.h new file mode 100644 index 0000000..76697fb --- /dev/null +++ b/include/odp/api/message.h @@ -0,0 +1,199 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + + +/** + * @file + * + * ODP Message event type + */ + +#ifndef ODP_API_MESSAGE_H_ +#define ODP_API_MESSAGE_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/** @defgroup odp_message ODP MESSAGE + * @{ + */ + +/** + * @typedef odp_message_t + * ODP message handle + */ + + +/** + * Get address of sender (source) of message + * + * @param msg Message handle + * @param[out] addr Buffer to hold sender address + */ +void odp_message_sender(odp_message_t msg, + uint8_t addr[ODP_MSGIO_ADDR_SIZE]); + +/** + * Message data pointer + * + * Return a pointer to the message data + * + * @param msg Message handle + * + * @return Pointer to the message data + */ +void *odp_message_data(odp_message_t msg); + +/** + * Message data length + * + * Return length of the message data. + * + * @param msg Message handle + * + * @return Message length + */ +uint32_t odp_message_length(const odp_message_t msg); + +/** + * Set message length + * + * Set length of the message data. + * Increase message data length by moving message tail into message tailroom. + * The actual message data is not modified. + * + * @param msg Message handle + * @param len New length + * + * @retval 0 on success + * @retval <0 on error (e.g. no enough tailroom) + */ +int odp_message_length_set(const odp_message_t msg, uint32_t len); + +/** + * Message headroom length + * + * Return length of the message headroom + * + * @param msg Message handle + * + * @return Headroom length + */ +uint32_t odp_message_headroom(const odp_message_t msg); + +/** + * Push out message head + * + * Increase message data length by moving message head into message headroom. + * Message headroom is decreased with the same amount. The message head may be + * pushed out up to 'headroom' bytes. Message is not modified if there's not + * enough headroom space. + * + * odp_message_xxx: + * length += len + * headroom -= len + * data -= len + * + * @param msg Message handle + * @param len Number of bytes to push the head (0 ... headroom) + * + * @return The new data pointer + * @retval NULL Requested offset exceeds available headroom + * + * @see odp_message_headroom(), odp_message_pull_head() + */ +void *odp_message_push_head(odp_message_t msg, uint32_t len); + +/** + * Pull in message head + * + * Decrease message data length by moving message head into message data. + * Message headroom is increased with the same amount. The message head may be + * pulled in up to 'length' bytes. Message is not modified if there's not + * enough data. + * + * odp_message_xxx: + * length -= len + * headroom += len + * data += len + * + * @param msg Message handle + * @param len Number of bytes to pull the head (0 ... length) + * + * @return The new data pointer + * @retval NULL Requested offset exceeds available data + * + * @see odp_message_headroom(), odp_message_push_head() + */ +void *odp_message_pull_head(odp_message_t msg, uint32_t len); + +/** + * Allocate message + * + * Allocate a message of a specific length. + * + * @param pool Message pool to allocate message from + * @param len Length of the allocated message + * + * @return Message handle on success + * @retval ODP_MESSAGE_INVALID on failure and errno set + */ +odp_message_t odp_message_alloc(odp_pool_t pool, uint32_t len); + +/** + * Free message + * + * Free message back to the message pool it was allocated from. + * + * @param msg Handle of message to free + */ +void odp_message_free(odp_message_t msg); + +/** + * Get message handle from event + * + * Converts an ODP_EVENT_MESSAGE type event to a message. + * + * @param ev Event handle representing a message. + * + * @return Message handle + * + * @see odp_event_type() + */ +odp_message_t odp_message_from_event(odp_event_t ev); + +/** + * Convert message handle to event + * + * @param msg Message handle + * + * @return Event handle + */ +odp_event_t odp_message_to_event(odp_message_t msg); + +/** + * Get printable value for an odp_message_t + * + * @param msg Message handle to be printed + * @return uint64_t value that can be used to print/display this + * handle + * + * @note This routine is intended to be used for diagnostic purposes + * to enable applications to generate a printable value that represents + * an odp_message_t handle. + */ +uint64_t odp_message_to_u64(odp_message_t msg); + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/include/odp/api/msgio.h b/include/odp/api/msgio.h new file mode 100644 index 0000000..8d73021 --- /dev/null +++ b/include/odp/api/msgio.h @@ -0,0 +1,222 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + + +/** + * @file + * + * ODP Message I/O API + */ + +#ifndef ODP_API_MSGIO_H_ +#define ODP_API_MSGIO_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/** @defgroup odp_msgio ODP MSGIO + * @{ + */ + +/** + * @typedef odp_msgio_t + * ODP message I/O handle + */ + +/** + * @def ODP_MSGIO_ADDR_SIZE + * Size of the address of a message I/O endpoint + */ + +/** + * @typedef odp_msgio_prio_t + * ODP MSGIO message priority + */ + +/** + * @def ODP_MSGIO_PRIO_HIGHEST + * Highest MSGIO message priority + */ + +/** + * @def ODP_MSGIO_PRIO_NORMAL + * Normal MSGIO message priority + */ + +/** + * @def ODP_MSGIO_PRIO_LOWEST + * Lowest MSGIO message priority + */ + +/** + * @def ODP_MSGIO_PRIO_DEFAULT + * Default MSGIO message priority + */ + + +/** + * Create message I/O endpoint + * + * Create an message I/O endpoint. The scope of the message bus is + * not defined but it is expected that it by default encompasses the OS + * instance but no more. + * + * A unique address for the endpoint is created. The addresses for other + * endpoints can be obtained using the odp_msgio_lookup() call and specified + * in the odp_msgio_send() call. + * + * @param name Name of our endpoint + * @param pool Pool (of type ODP_EVENT_MESSAGE) for incoming messages + * + * @return Message I/O handle on success + * @retval ODP_MSGIO_INVALID on failure and errno set + */ +odp_msgio_t odp_msgio_create(const char *name, + odp_pool_t pool); + +/** + * Set the input queue for a specific message priority + * + * @param msgio Message I/O handle + * @param prio Message priority + * @param queue Queue handle + * + * @retval 0 on success + * @retval <0 on failure + */ +int odp_msgio_inq_set(odp_msgio_t msgio, + odp_msgio_prio_t prio, + odp_queue_t queue); + +/** + * Activate message I/O endpoint + * + * Make this endpoint visible (odp_msgio_lookup() messages will be returned). + * Enable transmission and reception of messages through the endpoint. + * + * @param msgio Message I/O handle + * + * @retval 0 on success + * @retval <0 on failure + */ +int odp_msgio_activate(odp_msgio_t msgio); + +/** + * Destroy message I/O endpoint + * + * @param msgio Message I/O handle + * + * @retval 0 on success + * @retval <0 on failure + */ +int odp_msgio_destroy(odp_msgio_t msgio); + +/** + * Remove the input queue for a specific message priority + * + * Remove (disassociate) the matching input queue from a message I/O endpoint. + * The queue itself is not touched. + * + * @param msgio Message I/O handle + * @param prio Message priority + * + * @retval 0 on success + * @retval <0 on failure + */ +int odp_msgio_inq_rem(odp_msgio_t msgio, + odp_msgio_prio_t prio); + +/** + * Lookup endpoint by name + * + * Look up a current or future endpoint by name. + * When the endpoint exists, return the specified message (not necessarily the + * same message handle) with the endpoint as the sender. + * + * @param msgio Message I/O handle + * @param name Name to look up + * @param msg Message to return + */ +void odp_msgio_lookup(odp_msgio_t msgio, + const char *name, + odp_message_t msg); + +/** + * Monitor endpoint + * + * Monitor an endpoint (which may already have disappeared). + * When the endpoint disappears, return the specified message (not necessarily + * the same message handle) with the endpoint as the sender. + * + * Unrecognized or invalid endpoint addresses are treated as non-existing + * endpoints. + * + * @param msgio Message I/O handle + * @param[in] addr Address of monitored endpoint + * @param msg Message to return + */ +void odp_msgio_monitor(odp_msgio_t msgio, + const uint8_t addr[ODP_MSGIO_ADDR_SIZE], + odp_message_t msg); + +/** + * Send message + * + * Send a message to an endpoint (which may no longer exist). + * + * Message delivery into the recipient address space is ordered (per priority) + * and reliable. Delivery of message N implies delivery of all messages <N + * (of the same priority). + * All messages (accepted by MSGIO) will be delivered up to the point of + * recipient endpoint termination or lost connection where no more messages + * will be delivered. + * + * Actual reception (dequeueing) and processing by the recipient is not + * guaranteed (use user level acknowledgements for that). + * + * Monitor the remote endpoint to detect the disappearance or disconnection + * of the endpoint. + * + * A message may temporarily not be accepted for transmission (e.g. due + * to out of local buffer space), -1 will be returned and ODP errno set + * to EAGAIN. + * + * @param msgio Message I/O handle + * @param msg Message to send + * @param prio Priority of message + * @param[in] addr Address of recipient endpoint + * + * @retval 0 on success + * @retval <0 on error + */ +int odp_msgio_send(odp_msgio_t msgio, + odp_message_t msg, + odp_msgio_prio_t prio, + const uint8_t addr[ODP_MSGIO_ADDR_SIZE]); + +/** + * Get printable value for an odp_msgio_t + * + * @param msgio Message I/O handle to be printed + * @return uint64_t value that can be used to print/display this + * handle + * + * @note This routine is intended to be used for diagnostic purposes + * to enable applications to generate a printable value that represents + * an odp_msgio_t handle. + */ +uint64_t odp_msgio_to_u64(odp_msgio_t msgio); + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/message.h b/platform/linux-generic/include/odp/message.h new file mode 100644 index 0000000..2a29f59 --- /dev/null +++ b/platform/linux-generic/include/odp/message.h @@ -0,0 +1,39 @@ +/* Copyright (c) 2013, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP Message Bus + */ + +#ifndef ODP_PLAT_MESSAGE_H_ +#define ODP_PLAT_MESSAGE_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/std_types.h> +#include <odp/plat/pool_types.h> +#include <odp/plat/message_types.h> +#include <odp/plat/queue_types.h> + +/** @ingroup odp_message + * @{ + */ + +/** + * @} + */ + +#include <odp/api/message.h> + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/msgio.h b/platform/linux-generic/include/odp/msgio.h new file mode 100644 index 0000000..f45b3a2 --- /dev/null +++ b/platform/linux-generic/include/odp/msgio.h @@ -0,0 +1,40 @@ +/* Copyright (c) 2013, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP Message Bus + */ + +#ifndef ODP_PLAT_MSGIO_H_ +#define ODP_PLAT_MSGIO_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/std_types.h> +#include <odp/plat/pool_types.h> +#include <odp/plat/msgio_types.h> +#include <odp/plat/message_types.h> +#include <odp/plat/queue_types.h> + +/** @ingroup odp_msgio + * @{ + */ + +/** + * @} + */ + +#include <odp/api/msgio.h> + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/plat/message_types.h b/platform/linux-generic/include/odp/plat/message_types.h new file mode 100644 index 0000000..3a42064 --- /dev/null +++ b/platform/linux-generic/include/odp/plat/message_types.h @@ -0,0 +1,47 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + + +/** + * @file + * + * ODP Message message type + */ + +#ifndef ODP_MESSAGE_TYPES_H_ +#define ODP_MESSAGE_TYPES_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/std_types.h> +#include <odp/plat/strong_types.h> + +/** @addtogroup odp_message ODP message + * Operations on a message. + * @{ + */ + +typedef ODP_HANDLE_T(odp_message_t); + +#define ODP_MESSAGE_INVALID _odp_cast_scalar(odp_message_t, 0xffffffff) + +/** Get printable format of odp_message_t */ +static inline uint64_t odp_message_to_u64(odp_message_t hdl) +{ + return _odp_pri(hdl); +} + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp/plat/msgio_types.h b/platform/linux-generic/include/odp/plat/msgio_types.h new file mode 100644 index 0000000..9d9f9da --- /dev/null +++ b/platform/linux-generic/include/odp/plat/msgio_types.h @@ -0,0 +1,59 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + + +/** + * @file + * + * ODP Message bus types + */ + +#ifndef ODP_MSGIO_TYPES_H_ +#define ODP_MSGIO_TYPES_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp/std_types.h> +#include <odp/plat/strong_types.h> + +/** @addtogroup odp_msgio ODP message bus + * Operations on a message bus. + * @{ + */ + +typedef int odp_msgio_prio_t; + +#define ODP_MSGIO_PRIO_HIGHEST 0 + +#define ODP_MSGIO_PRIO_NORMAL 2 + +#define ODP_MSGIO_PRIO_LOWEST 3 + +#define ODP_MSGIO_PRIO_DEFAULT ODP_MSGIO_PRIO_NORMAL + +typedef ODP_HANDLE_T(odp_msgio_t); + +#define ODP_MSGIO_INVALID _odp_cast_scalar(odp_msgio_t, 0xffffffff) + +#define ODP_MSGIO_ADDR_SIZE 6 + +/** Get printable format of odp_msgio_t */ +static inline uint64_t odp_msgio_to_u64(odp_msgio_t hdl) +{ + return _odp_pri(hdl); +} + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif -- 1.9.1 _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
