With level-triggered epoll, append/enqueue operations to the local/locked queues increase performance by avoiding unnecessary atomic operations and barriers. These are necessary to avoid performance regressions when looping through ep_send_events and appending many items to a queue.
Signed-off-by: Eric Wong <[email protected]> Cc: Mathieu Desnoyers <[email protected]> Cc: Lai Jiangshan <[email protected]> Cc: Paul E. McKenney <[email protected]> Cc: Stephen Hemminger <[email protected]> Cc: Davide Libenzi <[email protected]> --- Benchmark for this coming with updated epoll patches. include/linux/wfcqueue.h | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/include/linux/wfcqueue.h b/include/linux/wfcqueue.h index 9464a0c..7eb2aaa 100644 --- a/include/linux/wfcqueue.h +++ b/include/linux/wfcqueue.h @@ -205,6 +205,49 @@ static inline bool wfcq_enqueue(struct wfcq_head *head, } /* + * __wfcq_append_local: append one local queue to another local queue + * + * No memory barriers are issued. Mutual exclusion is the responsibility + * of the caller. + * + * Returns false if the queue was empty prior to adding the node. + * Returns true otherwise. + */ +static inline bool __wfcq_append_local(struct wfcq_head *head, + struct wfcq_tail *tail, + struct wfcq_node *new_head, + struct wfcq_node *new_tail) +{ + struct wfcq_node *old_tail; + + old_tail = tail->p; + tail->p = new_tail; + old_tail->next = new_head; + + /* + * Return false if queue was empty prior to adding the node, + * else return true. + */ + return old_tail != &head->node; +} + +/* + * wfcq_enqueue_local: enqueue a node into a local wait-free queue + * + * No memory barriers are issued. Mutual exclusion is the responsibility + * of the caller. + * + * Returns false if the queue was empty prior to adding the node. + * Returns true otherwise. + */ +static inline bool wfcq_enqueue_local(struct wfcq_head *head, + struct wfcq_tail *tail, + struct wfcq_node *new_tail) +{ + return __wfcq_append_local(head, tail, new_tail, new_tail); +} + +/* * ___wfcq_busy_wait: busy-wait. */ static inline void ___wfcq_busy_wait(void) -- 1.8.2.rc3.2.g89ce8d6 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
