Hi Ola,
Thanks for sharing this. We are also looking at IPC at Kalray, where our
ODP model is multi-process, shared-nothing, architecture.
From our point of view, the main requirements for IPC would be:
- use it to communicate between different address spaces (AS), and as
such our messages should be bigger than just a pointer to a shared mem area
- use it for control/data planes signaling but also to exchange
packets to be allowed to build packets processing pipelines
- IPC ops should be mapped to our NoC HW as much as possible,
especially for send/recv operations
From your proposal, it seems to me that what you proposed is more like
an application messaging bus such as d-bus (especially the deferred
lookup() and monitor()), with a centralized rendez-vous point for
resource management. It is certainly useful, especially for control/data
plane signaling, but I would like to have a more low-level IPC
mechanism, on top of which we could build this sort of messaging bus.
For this kind of lower-level mechanism I think pktio might be a good
match. But in that case you need to be able to identify the endpoints.
What I proposed in precedent thread, was to use a hierarchical device
naming:
- "/dev/<device>" for real devices
- "/ipc/<identifier>" for IPC
What I had in mind so far for our platform was to used {AS identifier +
pool name in the AS} as IPC identifier, which would avoid the need of a
centralized rendez-vous point.
Another slightly different subject is that we would like to extend pktio
(but it is a common requirement for IPC) to be able to open pktio for
read-only, write-only or read-write operations. This is because opening
communications on our HW allocate HW resources for each configured RX
(you need DMA endpoints to be configured). Being able to open write-only
pktio help to save those resources, and would make sense even in the
standard pktio.
How could it look like? Here are some examples:
- to open a read-only pktio in AS0:
odp_pool_t local_pool = odp_pool_create("AS0_pool", ...);
odp_pktio_open("/ipc/local", local_pool);
Notes: "/ipc/local" identifies a local, read-only ipc endpoints.
- to open a write-only pktio in AS1 to send data to AS0:
odp_pktio_open("/ipc/AS0/AS0_pool", NULL);
Notes: "/ipc/AS0/AS0_pool" identifies a remote endpoints for
writing: "AS0" identifies the address space, whereas "AS0_pool"
identifies the packet pool in AS0 address space. The fact that we do not
associate any default pool with pktio means it is write-only.
- to open a read-write pktio between AS0 and AS1 in AS1:
odp_pool_t local_pool = odp_pool_create("AS1_pool", ...);
odp_pktio_open("/AS0/AS0_pool", local_pool);
Thanks,
ben
On 05/19/2015 12:03 AM, Ola Liljedahl wrote:
As promised, here is my first attempt at a standalone API for IPC - inter
process communication in a shared nothing architecture (message passing
between processes which do not share memory).
Currently all definitions are in the file ipc.h but it is possible to
break out some message/event related definitions (everything from
odp_ipc_sender) in a separate file message.h. This would mimic the
packet_io.h/packet.h separation.
The semantics of message passing is that sending a message to an endpoint
will always look like it succeeds. The appearance of endpoints is explicitly
notified through user-defined messages specified in the odp_ipc_resolve()
call. Similarly, the disappearance (e.g. death or otherwise lost connection)
is also explicitly notified through user-defined messages specified in the
odp_ipc_monitor() call. The send call does not fail because the addressed
endpoints has disappeared.
Messages (from endpoint A to endpoint B) are delivered in order. If message
N sent to an endpoint is delivered, then all messages <N have also been
delivered. Message delivery does not guarantee actual processing by the
recipient. End-to-end acknowledgements (using messages) should be used if
this guarantee is important to the user.
IPC endpoints can be seen as interfaces (taps) to an internal reliable
multidrop network where each endpoint has a unique address which is only
valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed
and then recreated (with the same name), the new endpoint will have a
new address (eventually endpoints addresses will have to be recycled but
not for a very long time). Endpoints names do not necessarily have to be
unique.
Signed-off-by: Ola Liljedahl <[email protected]>
---
(This document/code contribution attached is provided under the terms of
agreement LES-LTM-21309)
include/odp/api/ipc.h | 261 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 261 insertions(+)
create mode 100644 include/odp/api/ipc.h
diff --git a/include/odp/api/ipc.h b/include/odp/api/ipc.h
new file mode 100644
index 0000000..3395a34
--- /dev/null
+++ b/include/odp/api/ipc.h
@@ -0,0 +1,261 @@
+/* Copyright (c) 2015, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+
+/**
+ * @file
+ *
+ * ODP IPC API
+ */
+
+#ifndef ODP_API_IPC_H_
+#define ODP_API_IPC_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/** @defgroup odp_ipc ODP IPC
+ * @{
+ */
+
+/**
+ * @typedef odp_ipc_t
+ * ODP IPC handle
+ */
+
+/**
+ * @typedef odp_ipc_msg_t
+ * ODP IPC message handle
+ */
+
+
+/**
+ * @def ODP_IPC_ADDR_SIZE
+ * Size of the address of an IPC endpoint
+ */
+
+/**
+ * Create IPC endpoint
+ *
+ * @param name Name of local IPC endpoint
+ * @param pool Pool for incoming messages
+ *
+ * @return IPC handle on success
+ * @retval ODP_IPC_INVALID on failure and errno set
+ */
+odp_ipc_t odp_ipc_create(const char *name, odp_pool_t pool);
+
+/**
+ * Destroy IPC endpoint
+ *
+ * @param ipc IPC handle
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ */
+int odp_ipc_destroy(odp_ipc_t ipc);
+
+/**
+ * Set the default input queue for an IPC endpoint
+ *
+ * @param ipc IPC handle
+ * @param queue Queue handle
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ */
+int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue);
+
+/**
+ * Remove the default input queue
+ *
+ * Remove (disassociate) the default input queue from an IPC endpoint.
+ * The queue itself is not touched.
+ *
+ * @param ipc IPC handle
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ */
+int odp_ipc_inq_remdef(odp_ipc_t ipc);
+
+/**
+ * Resolve endpoint by name
+ *
+ * Look up an existing or future endpoint by name.
+ * When the endpoint exists, return the specified message with the endpoint
+ * as the sender.
+ *
+ * @param ipc IPC handle
+ * @param name Name to resolve
+ * @param msg Message to return
+ */
+void odp_ipc_resolve(odp_ipc_t ipc,
+ const char *name,
+ odp_ipc_msg_t msg);
+
+/**
+ * Monitor endpoint
+ *
+ * Monitor an existing (potentially already dead) endpoint.
+ * When the endpoint is dead, return the specified message with the endpoint
+ * as the sender.
+ *
+ * Unrecognized or invalid endpoint addresses are treated as dead endpoints.
+ *
+ * @param ipc IPC handle
+ * @param addr Address of monitored endpoint
+ * @param msg Message to return
+ */
+void odp_ipc_monitor(odp_ipc_t ipc,
+ const uint8_t addr[ODP_IPC_ADDR_SIZE],
+ odp_ipc_msg_t msg);
+
+/**
+ * Send message
+ *
+ * Send a message to an endpoint (which may already be dead).
+ * Message delivery is ordered and reliable. All (accepted) messages will be
+ * delivered up to the point of endpoint death or lost connection.
+ * Actual reception and processing is not guaranteed (use end-to-end
+ * acknowledgements for that).
+ * Monitor the remote endpoint to detect death or lost connection.
+ *
+ * @param ipc IPC handle
+ * @param msg Message to send
+ * @param addr Address of remote endpoint
+ *
+ * @retval 0 on success
+ * @retval <0 on error
+ */
+int odp_ipc_send(odp_ipc_t ipc,
+ odp_ipc_msg_t msg,
+ const uint8_t addr[ODP_IPC_ADDR_SIZE]);
+
+/**
+ * Get address of sender (source) of message
+ *
+ * @param msg Message handle
+ * @param addr Address of sender endpoint
+ */
+void odp_ipc_sender(odp_ipc_msg_t msg,
+ uint8_t addr[ODP_IPC_ADDR_SIZE]);
+
+/**
+ * Message data pointer
+ *
+ * Return a pointer to the message data
+ *
+ * @param msg Message handle
+ *
+ * @return Pointer to the message data
+ */
+void *odp_ipc_data(odp_ipc_msg_t msg);
+
+/**
+ * Message data length
+ *
+ * Return length of the message data.
+ *
+ * @param msg Message handle
+ *
+ * @return Message length
+ */
+uint32_t odp_ipc_length(const odp_ipc_msg_t msg);
+
+/**
+ * Set message length
+ *
+ * Set length of the message data.
+ *
+ * @param msg Message handle
+ * @param len New length
+ *
+ * @retval 0 on success
+ * @retval <0 on error
+ */
+int odp_ipc_reset(const odp_ipc_msg_t msg, uint32_t len);
+
+/**
+ * Allocate message
+ *
+ * Allocate a message of a specific size.
+ *
+ * @param pool Message pool to allocate message from
+ * @param len Length of the allocated message
+ *
+ * @return IPC message handle on success
+ * @retval ODP_IPC_MSG_INVALID on failure and errno set
+ */
+odp_ipc_msg_t odp_ipc_alloc(odp_pool_t pool, uint32_t len);
+
+/**
+ * Free message
+ *
+ * Free message back to the message pool it was allocated from.
+ *
+ * @param msg Handle of message to free
+ */
+void odp_ipc_free(odp_ipc_msg_t msg);
+
+/**
+ * Get message handle from event
+ *
+ * Converts an ODP_EVENT_MESSAGE type event to a message.
+ *
+ * @param ev Event handle
+ *
+ * @return Message handle
+ *
+ * @see odp_event_type()
+ */
+odp_ipc_msg_t odp_message_from_event(odp_event_t ev);
+
+/**
+ * Convert message handle to event
+ *
+ * @param msg Message handle
+ *
+ * @return Event handle
+ */
+odp_event_t odp_message_to_event(odp_ipc_msg_t msg);
+
+/**
+ * Get printable value for an odp_ipc_t
+ *
+ * @param ipc IPC handle to be printed
+ * @return uint64_t value that can be used to print/display this
+ * handle
+ *
+ * @note This routine is intended to be used for diagnostic purposes
+ * to enable applications to generate a printable value that represents
+ * an odp_ipc_t handle.
+ */
+uint64_t odp_ipc_to_u64(odp_ipc_t ipc);
+
+/**
+ * Get printable value for an odp_ipc_msg_t
+ *
+ * @param msg Message handle to be printed
+ * @return uint64_t value that can be used to print/display this
+ * handle
+ *
+ * @note This routine is intended to be used for diagnostic purposes
+ * to enable applications to generate a printable value that represents
+ * an odp_ipc_msg_t handle.
+ */
+uint64_t odp_ipc_msg_to_u64(odp_ipc_msg_t msg);
+
+/**
+ * @}
+ */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
--
Benoît GANNE
Field Application Engineer, Kalray
+33 (0)648 125 843
_______________________________________________
lng-odp mailing list
[email protected]
https://lists.linaro.org/mailman/listinfo/lng-odp
