On 22 May 2015 at 15:16, Savolainen, Petri (Nokia - FI/Espoo) < [email protected]> wrote:
> Hi, > > > > Instead of message bus (mbus), I’d use terms message and message IO > (similar to packets and packet IO). > That could work as well. But the concepts named and described here actually matches quite nicely (a subset of) those of kdbus https://code.google.com/p/d-bus/source/browse/kdbus.txt?name=policy. Anyway suggestion accepted. > > > odp_msg_t == message event > But packet events are called odp_packet_t, not odp_pkt_t, and timeout events are called odp_timeout_t, not odp_tmo_t. I prefer the long name odp_message_t. Do you still insist? odp_msgio_t == message io port/interface/tap/socket/mailbox/… > > > > // create msg io port > > odp_msgio_t odp_msgio_create(…); > > > > // msg io port local address > > odp_msgio_addr_t odp_msgio_addr(odp_msgio_t msgio); > > > > > > more comments inlined … > Your inlined comments are difficult to find. They seem to be more indented that the text they comment. > > > > > *From:* ext Ola Liljedahl [mailto:[email protected]] > *Sent:* Friday, May 22, 2015 2:20 PM > *To:* Savolainen, Petri (Nokia - FI/Espoo) > *Cc:* [email protected] > *Subject:* Re: [lng-odp] [RFC] Add ipc.h > > > > On 21 May 2015 at 11:50, Savolainen, Petri (Nokia - FI/Espoo) < > [email protected]> wrote: > > > > > -----Original Message----- > > From: lng-odp [mailto:[email protected]] On Behalf Of ext > > Ola Liljedahl > > Sent: Tuesday, May 19, 2015 1:04 AM > > To: [email protected] > > Subject: [lng-odp] [RFC] Add ipc.h > > > > As promised, here is my first attempt at a standalone API for IPC - inter > > process communication in a shared nothing architecture (message passing > > between processes which do not share memory). > > > > Currently all definitions are in the file ipc.h but it is possible to > > break out some message/event related definitions (everything from > > odp_ipc_sender) in a separate file message.h. This would mimic the > > packet_io.h/packet.h separation. > > > > The semantics of message passing is that sending a message to an endpoint > > will always look like it succeeds. The appearance of endpoints is > > explicitly > > notified through user-defined messages specified in the odp_ipc_resolve() > > call. Similarly, the disappearance (e.g. death or otherwise lost > > connection) > > is also explicitly notified through user-defined messages specified in > the > > odp_ipc_monitor() call. The send call does not fail because the addressed > > endpoints has disappeared. > > > > Messages (from endpoint A to endpoint B) are delivered in order. If > > message > > N sent to an endpoint is delivered, then all messages <N have also been > > delivered. Message delivery does not guarantee actual processing by the > > Ordered is OK requirement, but "all messages <N have also been delivered" > means in practice loss less delivery (== re-tries and retransmission > windows, etc). Lossy vs loss less link should be an configuration option. > > Also what "delivered" means? > > Message: > - transmitted successfully over the link ? > - is now under control of the remote node (post office) ? > - delivered into application input queue ? > - has been dequeued from application queue ? > > > > recipient. End-to-end acknowledgements (using messages) should be used if > > this guarantee is important to the user. > > > > IPC endpoints can be seen as interfaces (taps) to an internal reliable > > multidrop network where each endpoint has a unique address which is only > > valid for the lifetime of the endpoint. I.e. if an endpoint is destroyed > > and then recreated (with the same name), the new endpoint will have a > > new address (eventually endpoints addresses will have to be recycled but > > not for a very long time). Endpoints names do not necessarily have to be > > unique. > > How widely these addresses are unique: inside one VM, multiple VMs under > the same host, multiple devices on a LAN (VLAN), ... > > I have added that the scope is expected to be an OS instance (e.g. VM). > > > > OK, it’s likely the scope mostly needed anyway. > > > > Still need to define if addressing (and protocol) is implementation > specific or standardized. > > Implementation specific for now. > I think you are suggesting implementation specific, which is fine but need > to note that it’s suitable only between two ODP instances of the same _ > *implementation*_ version. E.g. messaging between linux-generic and > odp-dpdk, or between odp-dpdk-1.1.0.0 and odp-dpdk-1.1.0.1 would not > necessary work. A protocol spec (with version numbering) would be needed to > guaranteed intra implementation version communication. > > I don't want to walk into the tar pit of defining a binary ("on-the-wire") protocol. But this can be done later. Message formats, API and transport are separate things. I am only trying to define the API here. The incompatibility issues you describe should be noted. > > > Implementation specific messaging would be sufficient for SW (ODP > instance) coming from single SW vendor, but integration of SW from multiple > vendors would need packets or proper “IPC” protocol. > > I assume you mean "message format" here. There are so many ways to define these (Netlink has already been suggested, I suspect some users may prefer e.g. NETCONF). I think any standardization here is a separate activity. The question is whether something needs to be handled on the ODP MSGIO level. The transport protocol (not seen by the applications) is defined by the ODP implementation and may very will have HW-dependencies (e.g. using HW buffers/queues/DMA). > > > > > Signed-off-by: Ola Liljedahl <[email protected]> > > --- > > (This document/code contribution attached is provided under the terms of > > agreement LES-LTM-21309) > > > > > > +/** > > + * Create IPC endpoint > > + * > > + * @param name Name of local IPC endpoint > > + * @param pool Pool for incoming messages > > + * > > + * @return IPC handle on success > > + * @retval ODP_IPC_INVALID on failure and errno set > > + */ > > +odp_ipc_t odp_ipc_create(const char *name, odp_pool_t pool); > > This creates (implicitly) the local end point address. > > Yes. Does that have to be described? > > > > Maybe to highlight that “name” is not the address. > OK > > > > > > + > > +/** > > + * Set the default input queue for an IPC endpoint > > + * > > + * @param ipc IPC handle > > + * @param queue Queue handle > > + * > > + * @retval 0 on success > > + * @retval <0 on failure > > + */ > > +int odp_ipc_inq_setdef(odp_ipc_t ipc, odp_queue_t queue); > > Multiple input queues are likely needed for different priority messages. > > I have added priorities (copied from queue.h SCHED priorities) and a > priority parameter to the send() call. > > > > packet_io.h doesn't have any API for associating a list of queues with the > different (packet) priorities so there is no template to follow. I could > invent a new call for doing this on MBUS endpoints. > > E.g. > > int odp_mbus_inq_set(odp_mbus_t mbus, odp_mbus_prio_t prio, odp_queue_t > queue); > > Call once for each priority, I think this is better than having a call > which specifies all queues at once (the number of priorities is > implementation specific). > > > > I now think that the default queue should be specified when the endpoint > is created. Messages could start pouring in immediately and might have to > be enqueued somewhere (in certain implementations, I did not experience > this problem in my prototype so did not think about it). > > > > I think it’s better to create all input queues (or actually let the > implementation create queues) before the “port” is activated, so that > already the very first incoming message goes to the right queue (priority > level). > But then the implementation will need to know the type (polled, scheduled) etc of the queues. Pass same parameters as to odp_queue_create(). In the end, odp_mbus_create() will have a million parameters. Or perhaps there should be "start" and "stop" calls so that endpoint does not get active immediately. The application can create the endpoint, do other initialization stuff (create and attach input queues) and then activate the endpoint (connect it to the message bus). Seems more flexible to me. > > BTW, odp_pktio_inq_xxx() is likely removed and handled through > classification API (and let implementation return queue handles). > OK. Then I should not mimic this API. > > > > > > > + > > +/** > > + * Resolve endpoint by name > > + * > > + * Look up an existing or future endpoint by name. > > + * When the endpoint exists, return the specified message with the > > endpoint > > + * as the sender. > > + * > > + * @param ipc IPC handle > > + * @param name Name to resolve > > + * @param msg Message to return > > + */ > > +void odp_ipc_resolve(odp_ipc_t ipc, > > + const char *name, > > + odp_ipc_msg_t msg); > > How widely these names are visible? Inside one VM, multiple VMs under the > same host, multiple devices on a LAN (VLAN), ... > > I think name service (or address resolution) are better handled in > middleware layer. If ODP provides unique addresses and message passing > mechanism, additional services can be built on top. > > We still need an API for it. How should that API look like and where > should it be declared? > > I am suggesting a definition above and that it be located in the ODP > mbus.h. Please suggest an actual alternative. > > > > It’s not necessary an API call. Application could send a “name resolution > request” message to the middleware name server, etc. The name server > address can be delivered to application in many ways (command line, config > file, init message, etc). > My original prototype used special message numbers (but special addresses could also be used, message numbers are not part of the API) to indicate resolve/lookup and monitor requests. This can still be done by the implementation. Why can't we hide this behind a standardized API? Freedom for the implementation is good but I don't think it is good to expose the application to all those different potential solutions. > > > > > > > > + > > +/** > > + * Monitor endpoint > > + * > > + * Monitor an existing (potentially already dead) endpoint. > > + * When the endpoint is dead, return the specified message with the > > endpoint > > + * as the sender. > > + * > > + * Unrecognized or invalid endpoint addresses are treated as dead > > endpoints. > > + * > > + * @param ipc IPC handle > > + * @param addr Address of monitored endpoint > > + * @param msg Message to return > > + */ > > +void odp_ipc_monitor(odp_ipc_t ipc, > > + const uint8_t addr[ODP_IPC_ADDR_SIZE], > > + odp_ipc_msg_t msg); > > Again, I'd see node health monitoring and alarms as middleware services. > > Same comment as for resolve/lookup. > > > > Again it could a message interface between application and the middleware > alarm service, etc. > > > > > > > > + > > +/** > > + * Send message > > + * > > + * Send a message to an endpoint (which may already be dead). > > + * Message delivery is ordered and reliable. All (accepted) messages > will > > be > > + * delivered up to the point of endpoint death or lost connection. > > + * Actual reception and processing is not guaranteed (use end-to-end > > + * acknowledgements for that). > > + * Monitor the remote endpoint to detect death or lost connection. > > + * > > + * @param ipc IPC handle > > + * @param msg Message to send > > + * @param addr Address of remote endpoint > > + * > > + * @retval 0 on success > > + * @retval <0 on error > > + */ > > +int odp_ipc_send(odp_ipc_t ipc, > > + odp_ipc_msg_t msg, > > Message priority parameter added. > > > > > + const uint8_t addr[ODP_IPC_ADDR_SIZE]); > > This would be used to send a message to an address, but normal > odp_queue_enq() could be used to circulate this event inside an application > (ODP instance). > > Yes. Messages are events. > > > > > > + > > +/** > > + * Get address of sender (source) of message > > + * > > + * @param msg Message handle > > + * @param addr Address of sender endpoint > > + */ > > +void odp_ipc_sender(odp_ipc_msg_t msg, > > + uint8_t addr[ODP_IPC_ADDR_SIZE]); > > + > > +/** > > + * Message data pointer > > + * > > + * Return a pointer to the message data > > + * > > + * @param msg Message handle > > + * > > + * @return Pointer to the message data > > + */ > > +void *odp_ipc_data(odp_ipc_msg_t msg); > > + > > +/** > > + * Message data length > > + * > > + * Return length of the message data. > > + * > > + * @param msg Message handle > > + * > > + * @return Message length > > + */ > > +uint32_t odp_ipc_length(const odp_ipc_msg_t msg); > > + > > +/** > > + * Set message length > > + * > > + * Set length of the message data. > > + * > > + * @param msg Message handle > > + * @param len New length > > + * > > + * @retval 0 on success > > + * @retval <0 on error > > + */ > > +int odp_ipc_reset(const odp_ipc_msg_t msg, uint32_t len); > > Per Maxim's suggestion, I renamed this call to odp_message_length_set() > (there is also an odp_message_length() call which gets the length). > > > > New length need to be in limits of the buffer size. > Yes that's why the call can fail. But perhaps this limitation needs to be documented. > Agree, that buffer size is constant, but it can large (e.g. multiple kB in > maximum). If message delivery copies data, the actual data length need to > be define (for low overhead). > Which is why we have the odp_message_length() and odp_message_length_set() calls. > > > > > When data ptr or data len is modified: push/pull head, push/pull tail > would be analogies from packet API > > Messages are not packets that you add and remove headers from. Or? > > > > Messages have structure and may very well have layers. E.g. middleware > message header is in front. The data pointer and length are updated when > message travels between ODP msgio, middleware and application. > OK so push/pull the head are needed. Any real need to push/pull the tail? We need to introduce the concept of headroom as well I guess. More and more like packets... > > > If we are going to replicate the whole packet.h API, perhaps we should > just use packets for messages. Indeed this was in my original prototype but > then I wasn't sure this was abstract and implementation independent enough. > I do envision messages to be more like buffers but with a per-buffer size > (length). > > > > Certainly all protocol flags/features are not needed and preferably no > segmentation. But dynamic data ptr/len would bring in concepts of head, > tail, etc. > > > > -Petri > > > > >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
