One typical use case of percpu_ref_tryget() family functions is as
follows,
if (percpu_ref_tryget(&p->ref)) {
/* Operate on the other fields of *p */
}
The refcount needs to be checked before operating on the other fields
of the data structure (*p), otherwise, the values gotten from the
other fields may be invalid or inconsistent. To guarantee the correct
memory ordering, percpu_ref_tryget*() needs to be the ACQUIRE
operations.
This function implements that via using smp_load_acquire() in
__ref_is_percpu() to read the percpu pointer.
Signed-off-by: "Huang, Ying" <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Ming Lei <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Miaohe Lin <[email protected]>
---
include/linux/percpu-refcount.h | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index 16c35a728b4c..9838f7ea4bf1 100644
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -165,13 +165,13 @@ static inline bool __ref_is_percpu(struct percpu_ref *ref,
* !__PERCPU_REF_ATOMIC, which may be set asynchronously, and then
* used as a pointer. If the compiler generates a separate fetch
* when using it as a pointer, __PERCPU_REF_ATOMIC may be set in
- * between contaminating the pointer value, meaning that
- * READ_ONCE() is required when fetching it.
+ * between contaminating the pointer value, smp_load_acquire()
+ * will prevent this.
*
- * The dependency ordering from the READ_ONCE() pairs
+ * The dependency ordering from the smp_load_acquire() pairs
* with smp_store_release() in __percpu_ref_switch_to_percpu().
*/
- percpu_ptr = READ_ONCE(ref->percpu_count_ptr);
+ percpu_ptr = smp_load_acquire(&ref->percpu_count_ptr);
/*
* Theoretically, the following could test just ATOMIC; however,
@@ -231,6 +231,9 @@ static inline void percpu_ref_get(struct percpu_ref *ref)
* Returns %true on success; %false on failure.
*
* This function is safe to call as long as @ref is between init and exit.
+ *
+ * This function is an ACQUIRE operation, that is, all memory operations
+ * after will appear to happen after checking the refcount.
*/
static inline bool percpu_ref_tryget_many(struct percpu_ref *ref,
unsigned long nr)
@@ -260,6 +263,9 @@ static inline bool percpu_ref_tryget_many(struct percpu_ref
*ref,
* Returns %true on success; %false on failure.
*
* This function is safe to call as long as @ref is between init and exit.
+ *
+ * This function is an ACQUIRE operation, that is, all memory operations
+ * after will appear to happen after checking the refcount.
*/
static inline bool percpu_ref_tryget(struct percpu_ref *ref)
{
@@ -280,6 +286,9 @@ static inline bool percpu_ref_tryget(struct percpu_ref *ref)
* percpu_ref_tryget_live().
*
* This function is safe to call as long as @ref is between init and exit.
+ *
+ * This function is an ACQUIRE operation, that is, all memory operations
+ * after will appear to happen after checking the refcount.
*/
static inline bool percpu_ref_tryget_live(struct percpu_ref *ref)
{
--
2.30.2
