On Fri, Oct 17, 2014 at 11:01 AM, Alexandru Badicioiu <[email protected]> wrote: > Hi Bill, check my thoughts inline. > Thanks, > Alex > > On 17 October 2014 03:31, Bill Fischofer <[email protected]> wrote: >> >> Based on discussions we had yesterday and today, I'd like to outline the >> open issues regarding queues and synchronization/scheduling models. We'd >> like to get consensus on this in time for next week's Tuesday call. >> >> ODP identifies three different synchronization/scheduling models for >> queues: Parallel, Atomic, and Ordered. Here are my current understandings >> of what these mean: >> >> Parallel: Buffers on a parallel queue can be dequeued by the scheduler for >> any caller without restriction. This permits maximum scale-out and >> concurrency for events that are truly independent. >> >> Atomic: Buffers on an atomic queue can be dequeued by the scheduler for >> any caller. However, only one buffer from an atomic queue may be in process >> at any given time. When the scheduler dequeues a buffer from an atomic >> queue, the queue is locked and cannot dequeue further buffers until it is >> released. Releasing an atomic queue can occur in two ways: >> >> The dequeued buffer is enqueued to another queue via an odp_queue_enq() >> call. This action implicitly unlocks the atomic queue the buffer was sourced >> from. Note that this is the most common way in which atomic queues are >> unlocked. >> >> A call is made to odp_schedule_release_atomic() for the locked queue. >> This tells the scheduler that the queue's atomicity guarantee is deemed >> satisfied by the application and the queue is free to dequeue items to other >> scheduler callers. This method MUST be used if the caller consumes the >> buffer (e.g., frees it instead of enqueues it to another queue) and MAY be >> used as a performance optimization if the caller is done with any references >> to data that was serialized by the queue (e.g., the queue's context). It is >> an application programming error to release a queue prematurely as >> references subsequent to the release will not be synchronized. > > Third way - odp_buffer_free() called on a buffer which was dequeued from an > atomic queue. >> >> Ordered: Buffers on an ordered queue can be dequeued by the scheduler for >> any caller, however buffers on an ordered queue retain knowledge of their >> sequence on their source queue and this sequence will be restored whenever >> they are enqueued to a subsequent ordered queue. That is, if ordered queue >> A contains buffers A1, A2, and A3, and these are dequeued for processing by >> three different threads, then when they are subsequently enqueued to another >> ordered queue B by these threads, they will appear on B as B1, B2, and B3 >> regardless of the order in which their processing threads issued >> odp_queue_enq() calls to place them on B. > > Why ordering has to be restored only if the destination is an ordered queue? > I think it should be restored regardless of the destination queue type. Also > it should be restored if there are multiple destination queues, if the > implementation supports it.
Well then how can you inform the ODP implementation that it's ok to stop maintaining the order the packets came in? What if the flow (I understand flows make most sense to be ordered) contains a further encapsulation known by the application which can then process the packets in parallel in any given order? >> >> Implicit in these definitions is the fact that all queues associated with >> odp_pktio_t objects are ordered queues. > > Why is this implicit? The order restoration happens when buffers are > enqueued to the destination queue(s). Aren't the queues associated with > odp_pktio_t the first queues seen by a packet? If an pktio queue is > parallel, there is no requirement at all to ensure any ordering at next > enqueue. Especially in the case of Ethernet, that is the main target for ODP v1.0, where packet order is not guaranteed and rather performance (processing close to line rate) is desired, it doesn't make sense to make the input queues ordered by default. It makes sense to make the output queues from the classifier ordered by default, I guess. >> >> >> First question: Are these definitions accurate and complete? If not, then >> what are the correct definitions for these types we wish to define? >> Assuming these are correct, then there are several areas in ODP API that >> seem to need refinement: >> >> It seems that ODP buffers need at least two additional pieces of system >> meta data that are missing: >> >> Buffers need to have a last_queue that is the odp_queue_t of the last >> queue they were dequeued from. Needed so that odp_queue_enq() can unlock a >> previous atomic queue. >> >> Buffers need to retain sequence knowledge from the first ordered queue >> they were sourced from (i.e., their ingress odp_pktio_t) so this can be used >> for order restoration as they are enqueued to downstream ordered queues >> >> odp_schedule_release_atomic() currently takes a void argument and this is >> ambiguous. It should either take an odp_queue_t which is the atomic queue >> that is to be unlocked or else an odp_buffer_t and use that buffer's >> last_queue to find the queue to be unlocked. Taking an odp_buffer_t argument >> would seem to be more consistent. > > I think any argument is superfluous for odp_schedule_release_atomic , if the > application really needs to continue the buffer processing outside the > atomic context. Otherwise the context release happens on free. There can be > only one atomic context associated with a thread, at any given moment, and > the scheduler implementation has to keep track of it. Of course when freeing > the buffer the free call has to know that the atomic context of that buffer > has been previously released. >> >> Given the above definition of ordered queues, it would seem that there >> needs to be some ordered equivalent to odp_schedule_release_atomic() to >> handle the case where a buffer from an ordered queue is consumed rather than >> propagated. This is to avoid creating un-fillable gaps in the ordering >> sequence downstream. > > Any consumption will end up with calling odp_buffer_free() and free has to > inform the order restoration logic. Do we really need to extract a buffer > from the ordered flow before the consumption? > >> >> Second question: If an application generates packets (via >> odp_packet_alloc()), how are these sequenced with respect to other packets >> on downstream ordered queues? Same question for cloned/copied packets. >> > Clones/copies share the same sequencing information. The same for fragments. > If locally generated packets have to be inserted in an ordered flow they > have to explicitly request a sequencing information relevant to source queue > at the moment of insertion. Then they can be enqueued/freed similarly as the > other buffers in the flow. > >> >> Third question: If an application takes packets from a source ordered >> queue and enqueues them to different target ordered queues, how is >> sequencing handled here since by definition there are gaps in the individual >> downstream ordered queues. The simplest example is a switch or router that >> takes packets from one odp_pktio_t and sends them to multiple target >> odp_pktio_ts. A similar question arises for packets sourced from multiple >> input ordered queues going to the same target ordered queue. > > I think that order is important to be maintained per flow, not the absolute > one between unrelated flows. > I think type of the destination queues and their number also may have no > importance. Ordering should not be associated with a particular destination, > is rather a source defined thing. Ordering works between two points - > definition point (where the order is observed) and restoration point , which > can be rather logical (e.g. next enqueue) than physical (a given queue). > Order definition point is usually a queue. If it's feasible for the HW that > the order can be observed across multiple queues, then ordering can work > this way too. Maybe we can view the API this way - define order > definition/restoration points and associate queues with them. > >> >> Fourth question: How does this intersect with flow identification from the >> classifier? It would seem that the classifier should override the raw >> packet sequence and re-write this information as a flow sequence which would >> be honored as the ordering sequence by subsequent downstream ordered queues. >> Note that flows would still have the same downstream gap issues if they are >> enqueued to multiple downstream ordered queues. This would definitely arise >> in multihoming support for SCTP, though this example is not an ODP v1.0 >> consideration. >> >> We may not have a complete set of answers to all of these questions for >> ODP v1.0 but we need to be precise about what is and is not done for them in >> ODP v1.0 so that we can do accurate testing for v1.0 as well as identify the >> areas that need further work next year as we move beyond v1.0. >> >> Thanks for your thoughts and suggestions on these. If there are >> additional questions along these lines feel free to add them, but I'd really >> like to scope this discussion to what is in ODP v1.0. >> >> Bill >> >> _______________________________________________ >> lng-odp mailing list >> [email protected] >> http://lists.linaro.org/mailman/listinfo/lng-odp >> > > > _______________________________________________ > lng-odp mailing list > [email protected] > http://lists.linaro.org/mailman/listinfo/lng-odp > _______________________________________________ lng-odp mailing list [email protected] http://lists.linaro.org/mailman/listinfo/lng-odp
