On 28 March 2017 at 20:32, Bill Fischofer <[email protected]> wrote:
> On Tue, Mar 28, 2017 at 11:18 AM, Honnappa Nagarahalli
> <[email protected]> wrote:
>> On 28 March 2017 at 07:21, Bill Fischofer <[email protected]> wrote:
>>> On Tue, Mar 28, 2017 at 7:19 AM, Verma, Shally <[email protected]>
>>> wrote:
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: lng-odp [mailto:[email protected]] On Behalf Of Bill
>>>> Fischofer
>>>> Sent: 28 March 2017 16:44
>>>> To: Ola Liljedahl <[email protected]>
>>>> Cc: nd <[email protected]>; [email protected]
>>>> Subject: Re: [lng-odp] odp_queue_enq semantics
>>>>
>>>> On Tue, Mar 28, 2017 at 4:10 AM, Ola Liljedahl <[email protected]>
>>>> wrote:
>>>>> On 28 March 2017 at 10:41, Joe Savage <[email protected]> wrote:
>>>>>> Hey,
>>>>>>
>>>>>> I just wanted to clarify something about the expected behaviour of
>>>>>> odp_queue_enq. In the following code snippet, is it acceptable for
>>>>>> the assert to fire? (i.e. for a dequeue after a successful enqueue to
>>>>>> fail, with only a single thread of execution)
>>>>>>
>>>>>> odp_queue_t queue;
>>>>>> odp_event_t ev1, ev2;
>>>>>> /* ... */
>>>>>> if (odp_queue_enq(queue, ev1) == 0) {
>>>>>> ev2 = odp_queue_deq(queue);
>>>>>> assert(ev2 != ODP_EVENT_INVALID);
>>>>>> }
>>>>
>>>>
>>>> rc == 0 from odp_queue_enq() simply means that the enqueue request has
>>>> been accepted.
>>>>
>>>> odp_queue_deq() removes the first element available on the specified queue.
>>>>
>>>> As Ola points out, depending on the implementation there may be some
>>>> latency associated with queue operations so it is possible for the assert
>>>> to fire. Of course, in a multi-threaded environment some other thread may
>>>> have dequeued the event first as well, so this sort of code is inherently
>>>> brittle.
>>>>
>>>> Shally-Sounds to me that based on implementation odp_queue_enq() can be
>>>> async call so applications dequeuing events should always check against
>>>> whether it is valid intended event? And if not intended, then app should
>>>> put event back to queue? Is that understanding correct?
>>>
>>
>> odp_queue_enq() itself is not an async call. When it returns to the
>> application (that did the enqueue), the event is enqueued on the
>> queue.
>
> That is correct. If odp_queue_enq() returns success the event is
> enqueued and will appear ahead of any subsequent events added by other
> odp_queue_enq() calls.
Enqueue calls made from the same thread.
I don't think there is any guarantee of global ordering of enqueued
events from different threads, even if they synchronise through shared
memory.
> However that does not imply that it is
> instantly visible to odp_queue_deq().
Per the above:
Assume 'q' is an empty queue.
T0: odp_queue_enq(q, ev0);
T0: __atomic_store_n(&flag, 1, __ATOMIC_RELEASE);//Ensure prior memory
accesses are visible
T1: while (__atomic_load_n(&flag, __ATOMIC_ACQUIRE) != 1) ; //spin-wait
T1: ev1 = odp_queue_deq(q);
T1: assert(ev1 == ev0);
T1 is not guaranteed to be able to dequeue the event enqueued by T0 so
the assertion may fail (ev1 == ODP_EVENT_INVALID). Memory accesses and
ODP events may take different routes between the CPU's of T0 and T1
and I don't think any ordering between them is guaranteed.
Store-release does not order ODP queue operations.
But this one should hold:
Assume flag is initially 0.
T0: flag = 1;
T0: odp_queue_enq(q, ev0);
T1: ev1 = odp_queue_deq(q); //T1 dequeues the event enqueued by T0
T1: assert(flag == 1);
The assertion should hold. odp_queue_enq() should have release
ordering for all previous memory accesses by that thread and
odp_queue_deq() should have acquire ordering for all following memory
accesses by that thread.
>
>>
>>> You should check the return from odp_queue_deq() to see if it's
>>> ODP_EVENT_INVALID, but ODP does not define "peek" or "push" operations
>>> on queues, so there is no way to "put an event back" unless you call
>>> odp_queue_enq(), which adds it to the end of the queue.
>>>
>>>>
>>>>>>
>>>>>> That is, can the "success" status code from odp_queue_enq be used to
>>>>>> indicate a delayed enqueue ("This event will be added to the queue at
>>>>>> some point soon-ish"), or should it only be used to communicate an
>>>>>> immediate successful addition to the queue? The documentation seems
>>>>>> unclear on this point while the validation tests suggest the latter,
>>>>>> but I thought it worth checking up on.
>>>>> Some code in odp_scheduling test/benchmark also expects an enqueued
>>>>> event to be immediately dequeuable by the same thread.
>>>>>
>>>>> I think this is not something you can require, neither from a HW queue
>>>>> manager or from a SW implementation. HW implementations can always
>>>>> have associated latencies visible to the thread that
>>>>> enqueues/dequeues. Also for a SW implementation with loosely coupled
>>>>> parts (e.g. producer & consumer head & tail pointers) it can be
>>>>> possible for an enqueued event to not be immediately available when
>>>>> other threads are doing concurrent operations on the same queue. Only
>>>>> if a lock protects the whole data structure can you enforce one global
>>>>> view of this data structure. You don't want to use a lock.
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Joe
