* 赵宇龙 ([email protected]) wrote: > Hi, > > I write a user space rcu, code at > https://github.com/zylthinking/tools/blob/master/rcu.h > https://github.com/zylthinking/tools/blob/master/rcu.c. > > I notice the main difference with liburcu should be I use a daemon thread > to do works such as waking up sleepers, while the liburcu does not. > > I wonder why we can't use such a thread. When we use it, the > rcu_read_(un)lock will not includes wake up writers any more. which will > help to improve performance. It is the cost of such a daemon thread is too > high? > > zhao yulong
Hi Zhao, The main reason for having the "wakeup writer" present in the rcu_read_unlock() path in liburcu is to be energy-efficient: we don't want any thread to consume power unless they really have to. For instance, endless busy-waiting on a variable is avoided. This is why we rely on sys_futex to wake up awaiting writers from rcu_read_unlock(). AFAIU, your rcu_daemon() thread is always active, and even though it calls "sched_yield()" to be nice to others, it will keep one CPU always powered on. One important thing to notice is that liburcu rcu_read_unlock() only calls sys_futex if a writer is waiting. Therefore, in a scenario with frequent reads and infrequent updates, rcu_read_unlock() only has to take the performance overhead of a load, test and branch, actually skipping the futex wake call. Another reason for not going for a worker thread to handle wait/wakeup between synchronize_rcu and rcu_read_unlock is to minimize impact on the application process/threading model. This is especially true for applications that rely on fork() _not_ followed by exec(): Linux actually copies a single thread of the parent (the one executing fork()), and discards all other threads. Therefore, we must be aware that adding an in-library thread will require users to handle the fork()-not-followed-by-exec() case carefully. Since we had no other choice, we rely on worker threads for call_rcu, but we don't use worker threads for the "simpler" use-case of synchronize_rcu(). A third reason for directly waking up the writer thread rather than having a worker thread dispatching this information is speed. Given the power efficiency constraints expressed above, we would have to issue one system call from the rcu_read_unlock() site to wake up the rcu_daemon() thread (so it does not have to busy-wait), and another system call to wake up the writer, involving a third thread in what should really involve only two threads. This will therefore add overhead to this signalling by requiring the scheduler to perform one extra context switch, and may involve extra communication between processors, since rcu_daemon() will likely execute on a different CPU, and will have to bring in cache lines from other processors. Finally, let's discuss the real-time aspect. For RT, we ideally want wait-free rcu_read_lock/unlock. Indeed, having a sys_futex wakeup call in rcu_read_unlock() could arguably be seen a making the unlock path less strictly wait-free (in case you would be concerned about the internal implementation of sys_futex wake not being entirely wait-free). Currently, on Linux, one way to change the behavior of rcu_read_unlock() to make it even more RT-friendly (in case you are concerned about using sys_futex() on a pure RT thread) is to undefine CONFIG_RCU_HAVE_FUTEX, thus changing the behavior of urcu/futex.h and compat_futex.c. The futex_async() call will then do busy-waiting on the FUTEX_WAIT side (waiting 10us between attempts), and do exactly _nothing_ on the wake-up side, which is certainly wait-free. This will be less energy-efficient, of course, but will provide a strictly wait-free rcu_read_unlock(). We might want to consider creating a liburcu-rt.so for real-time use-cases that prefer the non-energy-efficient wait, along with the strictly wait-free rcu_read_unlock(). Thoughts ? Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
