On 20 May 2015 at 12:47, Maxim Uvarov <[email protected]> wrote: > > > On 20 May 2015 at 13:21, Ola Liljedahl <[email protected]> wrote: >> >> On 20 May 2015 at 10:25, Maxim Uvarov <[email protected]> wrote: >> > Hello Ola, >> > >> > please find some replays bellow. >> > >> > On 19 May 2015 at 01:03, Ola Liljedahl <[email protected]> wrote: >> >> >> >> As promised, here is my first attempt at a standalone API for IPC - >> >> inter >> >> process communication in a shared nothing architecture (message passing >> >> between processes which do not share memory). >> >> >> >> Currently all definitions are in the file ipc.h but it is possible to >> >> break out some message/event related definitions (everything from >> >> odp_ipc_sender) in a separate file message.h. This would mimic the >> >> packet_io.h/packet.h separation. >> >> >> >> The semantics of message passing is that sending a message to an >> >> endpoint >> >> will always look like it succeeds. The appearance of endpoints is >> >> explicitly >> >> notified through user-defined messages specified in the >> >> odp_ipc_resolve() >> >> call. Similarly, the disappearance (e.g. death or otherwise lost >> >> connection) >> >> is also explicitly notified through user-defined messages specified in >> >> the >> >> odp_ipc_monitor() call. The send call does not fail because the >> >> addressed >> >> endpoints has disappeared. >> >> >> >> Messages (from endpoint A to endpoint B) are delivered in order. If >> >> message >> >> N sent to an endpoint is delivered, then all messages <N have also been >> >> delivered. Message delivery does not guarantee actual processing by the >> >> recipient. End-to-end acknowledgements (using messages) should be used >> >> if >> >> this guarantee is important to the user. >> >> >> >> IPC endpoints can be seen as interfaces (taps) to an internal reliable >> >> multidrop network where each endpoint has a unique address which is >> >> only >> >> valid for the lifetime of the endpoint. I.e. if an endpoint is >> >> destroyed >> >> and then recreated (with the same name), the new endpoint will have a >> >> new address (eventually endpoints addresses will have to be recycled >> >> but >> >> not for a very long time). Endpoints names do not necessarily have to >> >> be >> >> unique. >> >> >> >> Signed-off-by: Ola Liljedahl <[email protected]> >> >> --- >> >> (This document/code contribution attached is provided under the terms >> >> of >> >> agreement LES-LTM-21309) >> >> >> >> include/odp/api/ipc.h | 261 >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> 1 file changed, 261 insertions(+) >> >> create mode 100644 include/odp/api/ipc.h >> >> >> >> diff --git a/include/odp/api/ipc.h b/include/odp/api/ipc.h >> >> new file mode 100644 >> >> index 0000000..3395a34 >> >> --- /dev/null >> >> +++ b/include/odp/api/ipc.h >> >> @@ -0,0 +1,261 @@ >> >> +/* Copyright (c) 2015, Linaro Limited >> >> + * All rights reserved. >> >> + * >> >> + * SPDX-License-Identifier: BSD-3-Clause >> >> + */ >> >> + >> >> + >> >> +/** >> >> + * @file >> >> + * >> >> + * ODP IPC API >> >> + */ >> >> + >> >> +#ifndef ODP_API_IPC_H_ >> >> +#define ODP_API_IPC_H_ >> >> + >> >> +#ifdef __cplusplus >> >> +extern "C" { >> >> +#endif >> >> + >> >> +/** @defgroup odp_ipc ODP IPC >> >> + * @{ >> >> + */ >> >> + >> >> +/** >> >> + * @typedef odp_ipc_t >> >> + * ODP IPC handle >> >> + */ >> >> + >> >> +/** >> >> + * @typedef odp_ipc_msg_t >> >> + * ODP IPC message handle >> >> + */ >> >> + >> >> + >> >> +/** >> >> + * @def ODP_IPC_ADDR_SIZE >> >> + * Size of the address of an IPC endpoint >> >> + */ >> >> + >> >> +/** >> >> + * Create IPC endpoint >> >> + * >> >> + * @param name Name of local IPC endpoint >> >> + * @param pool Pool for incoming messages >> >> + * >> >> + * @return IPC handle on success >> >> + * @retval ODP_IPC_INVALID on failure and errno set >> >> + */ >> >> +odp_ipc_t odp_ipc_create(const char *name, odp_pool_t pool); >> >> + >> > >> > >> > Will it be separate pool or you want to attach existing pool to your >> > ipc? >> It should be a pool with events of type messages (ODP_EVENT_MESSAGE) >> so in practice it will likely be a separate pool. >> >> > >> > >> >> >> >> +/** >> >> + * Destroy IPC endpoint >> >> + * >> >> + * @param ipc IPC handle >> >> + * >> >> + * @retval 0 on success >> >> + * @retval <0 on failure >> >> + */ >> >> +int odp_ipc_destroy(odp_ipc_t ipc); >> >> + >> >> +/** >> >> + * Set the default input queue for an IPC endpoint >> >> + * >> >> + * @param ipc IPC handle >> >> + * @param queue Queue handle >> >> + * >> >> + * @retval 0 on success >> >> + * @retval <0 on failure >> >> + */ >> >> +int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue); >> >> + >> >> +/** >> >> + * Remove the default input queue >> >> + * >> >> + * Remove (disassociate) the default input queue from an IPC endpoint. >> >> + * The queue itself is not touched. >> >> + * >> >> + * @param ipc IPC handle >> >> + * >> >> + * @retval 0 on success >> >> + * @retval <0 on failure >> >> + */ >> >> +int odp_ipc_inq_remdef(odp_ipc_t ipc); >> >> + >> >> +/** >> >> + * Resolve endpoint by name >> >> + * >> >> + * Look up an existing or future endpoint by name. >> >> + * When the endpoint exists, return the specified message with the >> >> endpoint >> >> + * as the sender. >> >> + * >> >> + * @param ipc IPC handle >> >> + * @param name Name to resolve >> >> + * @param msg Message to return >> >> + */ >> >> +void odp_ipc_resolve(odp_ipc_t ipc, >> >> + const char *name, >> >> + odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * Monitor endpoint >> >> + * >> >> + * Monitor an existing (potentially already dead) endpoint. >> >> + * When the endpoint is dead, return the specified message with the >> >> endpoint >> >> + * as the sender. >> >> + * >> >> + * Unrecognized or invalid endpoint addresses are treated as dead >> >> endpoints. >> >> + * >> >> + * @param ipc IPC handle >> >> + * @param addr Address of monitored endpoint >> >> + * @param msg Message to return >> >> + */ >> >> +void odp_ipc_monitor(odp_ipc_t ipc, >> >> + const uint8_t addr[ODP_IPC_ADDR_SIZE], >> >> + odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * Send message >> >> + * >> >> + * Send a message to an endpoint (which may already be dead). >> >> + * Message delivery is ordered and reliable. All (accepted) messages >> >> will >> >> be >> >> + * delivered up to the point of endpoint death or lost connection. >> >> + * Actual reception and processing is not guaranteed (use end-to-end >> >> + * acknowledgements for that). >> >> + * Monitor the remote endpoint to detect death or lost connection. >> >> + * >> >> + * @param ipc IPC handle >> >> + * @param msg Message to send >> >> + * @param addr Address of remote endpoint >> >> + * >> >> + * @retval 0 on success >> >> + * @retval <0 on error >> >> + */ >> >> +int odp_ipc_send(odp_ipc_t ipc, >> >> + odp_ipc_msg_t msg, >> >> + const uint8_t addr[ODP_IPC_ADDR_SIZE]); >> >> + >> >> +/** >> >> + * Get address of sender (source) of message >> >> + * >> >> + * @param msg Message handle >> >> + * @param addr Address of sender endpoint >> >> + */ >> >> +void odp_ipc_sender(odp_ipc_msg_t msg, >> >> + uint8_t addr[ODP_IPC_ADDR_SIZE]); >> >> + >> >> +/** >> >> + * Message data pointer >> >> + * >> >> + * Return a pointer to the message data >> >> + * >> >> + * @param msg Message handle >> >> + * >> >> + * @return Pointer to the message data >> >> + */ >> >> +void *odp_ipc_data(odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * Message data length >> >> + * >> >> + * Return length of the message data. >> >> + * >> >> + * @param msg Message handle >> >> + * >> >> + * @return Message length >> >> + */ >> >> +uint32_t odp_ipc_length(const odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * Set message length >> >> + * >> >> + * Set length of the message data. >> >> + * >> >> + * @param msg Message handle >> >> + * @param len New length >> >> + * >> >> + * @retval 0 on success >> >> + * @retval <0 on error >> >> + */ >> >> +int odp_ipc_reset(const odp_ipc_msg_t msg, uint32_t len); >> > >> > >> > reset usually stand to reset to some default value. In that case it's >> > better >> > to be odp_ipc_len_set(msg, len). >> Well I copied this from packet.h where odp_packet_reset(pkt, len) >> seemed to be equivalent to what I want to achieve but I agree that >> odp_ipc_len_set() is more direct. I can change. >> >> > And reset is just void inline function to reset to default value >> > odp_ipc_reset(msg). >> > And len might be is not needed here because you transfer events. So just >> > get >> The IPC mechanism transfers messages. And different messages have >> different lengths. >> > size from event. >> The message size specified when creating a message pool that is >> associated with an IPC endpoint is basically an MTU (the largest >> message size that can be sent or received). >> >> > >> > Might be something like: >> > >> > msg.event = ev; // len will be get from event >> > msg.remote = addr; >> I don't understand what you are trying to describe here. >> >> > >> > >> > >> >> >> >> + >> >> +/** >> >> + * Allocate message >> >> + * >> >> + * Allocate a message of a specific size. >> >> + * >> >> + * @param pool Message pool to allocate message from >> >> + * @param len Length of the allocated message >> >> + * >> >> + * @return IPC message handle on success >> >> + * @retval ODP_IPC_MSG_INVALID on failure and errno set >> >> + */ >> >> +odp_ipc_msg_t odp_ipc_alloc(odp_pool_t pool, uint32_t len); >> >> + >> > >> > >> > is len sizeof(odp_ipc_msg_t) here? >> No. odp_ipc_msg_t is a handle to a message, just like odp_packet_t is >> a handle to a packet. >> The implementation of odp_ipc_msg_t is platform specific and thus >> sizeof(odp_ipc_msg_t) as well (but 32 or 64 bits is a good guess). >> >> The len parameter in odp_ipc_alloc() specifies the current/actual >> length of the message which depends on what the application wants to >> use that specific message for. >> >> > >> > >> >> >> >> +/** >> >> + * Free message >> >> + * >> >> + * Free message back to the message pool it was allocated from. >> >> + * >> >> + * @param msg Handle of message to free >> >> + */ >> >> +void odp_ipc_free(odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * Get message handle from event >> >> + * >> >> + * Converts an ODP_EVENT_MESSAGE type event to a message. >> >> + * >> >> + * @param ev Event handle >> >> + * >> >> + * @return Message handle >> >> + * >> >> + * @see odp_event_type() >> >> + */ >> >> +odp_ipc_msg_t odp_message_from_event(odp_event_t ev); >> >> + >> >> +/** >> >> + * Convert message handle to event >> >> + * >> >> + * @param msg Message handle >> >> + * >> >> + * @return Event handle >> >> + */ >> >> +odp_event_t odp_message_to_event(odp_ipc_msg_t msg); >> >> + >> > >> > >> > Most of functions look like pktio. But that version of IPC looks like >> > fully >> > software implementation. >> I think some SoC's could use HW mechanism to transfer messages (which >> are basically buffers with individual lengths) between endpoints (e.g. >> different address spaces). >> >> > Idea is good - send and receive odp events. >> Send and receive ODP *messages*, not any type of ODP events. >> >> > It looks like odp_ipc_t ipc >> > object also connected to >> > some odp_pktio_t where actual packets will be sent. And it will be >> > interesting to make queues interface >> > work with that ipc. >> > >> > Might be: >> > odp_pktio_t pktio can have ipc atribute like >> > >> > odp_ipc_pktio_set(odp_ipc_t ipc, odp_pktio_t pktio); >> Now you are down to implementation details. Why should this be part of >> the public API? >> >> >> > >> > then queue created for that pktio can send/recv packets from outside. >> I think you can already do this with the current packet_io API. No >> need to conflate this with IPC/message passing. >> >> > >> > If I understand right, that functions use queue only for ipc messages: >> > int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue); >> > int odp_ipc_inq_remdef(odp_ipc_t ipc); >> You want to be able to receive and dispatch IPC messages just like any >> other type of events so incoming messages should be put on queues. The >> queue could be specified when the IPC endpoint is created but I copied >> the packet_io API where it was possible. >> >> > and they are not linked with outside pktio. Earlier you said that >> > propose >> > to send some packets to some internal tap device. >> No not packets, messages. And the reference to "tap" was in a general >> sense, not related to Linux tap interfaces. >> >> > And in that case tap >> > device can be pktio. And then this pktio-tap connected to ipc object to >> > dispatch events from it. >> >> > >> > And probably we need better names for our ipc implementation. Mine can >> > be >> > odp_pktio_ipc and your can be odp_ipcmsg_ because the propose is >> > different. >> Actually I don't understand why your packet pipes need to change the >> ODP API. They are just pktio interfaces with different semantics. By >> using different pktio device names (which anyway are platform >> specific), you create packet pipes instead of opening physical network >> interfaces. But they behave the same from the user perspective. Or? >> > > Yes in my case is only pipes and it's just pktio with different name. No > difference to application how to use this pktio. > >> >> > >> > >> > >> > >> >> >> >> +/** >> >> + * Get printable value for an odp_ipc_t >> >> + * >> >> + * @param ipc IPC handle to be printed >> >> + * @return uint64_t value that can be used to print/display this >> >> + * handle >> >> + * >> >> + * @note This routine is intended to be used for diagnostic purposes >> >> + * to enable applications to generate a printable value that >> >> represents >> >> + * an odp_ipc_t handle. >> >> + */ >> >> +uint64_t odp_ipc_to_u64(odp_ipc_t ipc); >> >> + >> >> +/** >> >> + * Get printable value for an odp_ipc_msg_t >> >> + * >> >> + * @param msg Message handle to be printed >> >> + * @return uint64_t value that can be used to print/display this >> >> + * handle >> >> + * >> >> + * @note This routine is intended to be used for diagnostic purposes >> >> + * to enable applications to generate a printable value that >> >> represents >> >> + * an odp_ipc_msg_t handle. >> >> + */ >> >> +uint64_t odp_ipc_msg_to_u64(odp_ipc_msg_t msg); >> >> + >> >> +/** >> >> + * @} >> >> + */ >> >> + >> >> +#ifdef __cplusplus >> >> +} >> >> +#endif >> >> + >> >> +#endif >> >> -- >> >> 1.9.1 >> > >> > >> > >> > After writing replay I came with idea why not build one pktio based on >> > other >> > pktio? >> > In that case eth0_pktio = ocp_create_pktio("eth0") can be used for >> > external >> > packets exchange and >> > ipc_eth0_pktio = odp_create_pktio(eth0_pktio, ..). >> > So read from eth0_pktio will get only outside packets and read from >> > ipc_eth0_pktio can deliver only >> > ipc packets. >> There are no IPC packets, just IPC messages. >> >> > In that case you do not need to implement transport layer and >> > just do functionality above >> > on current pktio. >> IPC/message passing has different semantics (e.g. reliable/in-order) >> from network interfaces. >> IPC/message passing is for communication between local entities (e.g. >> applications on the same host), not for communicating with external >> entities over some physical network. >> So I see several serious incompatibilities and I see no reason to >> build IPC/message passing on top of some physical network interface >> and its representation in ODP. >> > > Ok, I understand that it's exchange with some messages on one machine. I > just not sure why conrtol plane and data plane be different host. Like ODL > where OVS on one machine and controller on other. The OpenFlow controller can be anywhere on the network (typically *not* on the same machine as the OpenFlow dataplane) so needs to use a proper transport protocol that works over the Internet. This is a different use case from the applications on a host which together implement some network function (e.g. eNodeB).
> > As I understand you don't what not transfer any odp events. Just ipc > messages. Btw what is inside that messages? How that message looks like? The messages exchanged between control and data plane applications are requests and replies for changing configuration, adding/removing routes, setting up/tearing down bearers and tunnels, IPsec/IKE updates etc. Also for viewing the current state of applications (e.g. read out statistics, alarms). So messages are normally quite small, maybe up to 100-200 bytes and typically well structured. Messages could be expressed as C structs. Each message will typically have a type which identifies the purpose and structure of the message. I left this out of the IPC API as I don't think it is strictly needed, it could just be a convention, store the type first in the message. The IPC mechanism itself does not use the type. > > Maxim. > > > >> >> > >> > Thanks, >> > Maxim. >> > >> > >> > >> > >> > >> >> >> >> >> >> _______________________________________________ >> >> lng-odp mailing list >> >> [email protected] >> >> https://lists.linaro.org/mailman/listinfo/lng-odp >> > >> > > > _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
