Re: [PR] Reduce futex usage in ParkingLot (brpc)

Sun, 13 Jul 2025 06:32:48 -0700 


JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3067051607

   > 使用 #2907 和 #2916 后，futex 自旋锁竞争变更激烈了，CPU开销涨了17%左右。
   > 
   > <img alt="image" width="824" height="205" 
src="https://private-user-images.githubusercontent.com/12249635/465729698-c6e42539-1b48-4620-9163-3a58c5711a18.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3Mjk2OTgtYzZlNDI1MzktMWI0OC00NjIwLTkxNjMtM2E1OGM1NzExYTE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVlMTk2MzY3ZmIzNjhkMjAzMDgyNDVhMzNiNzk1MmU5MzRkOTE4Nzc4MTAyMjliNDNiYmYyNzNiYjRhNGFhM2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.WL5OK41ZsG6L6ZwuAcCxvypfRkO5O0JdTE894qGe6Ww";>
   > <img alt="image" width="340" height="267" 
src="https://private-user-images.githubusercontent.com/12249635/465732978-9a190d64-8ae1-4805-8de7-460bdf4605c4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzI5NzgtOWExOTBkNjQtOGFlMS00ODA1LThkZTctNDYwYmRmNDYwNWM0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVmZTUzYzc3MTg2MGQ1MmVkNmZmOTBjODVhMmU0Y2RiZGY1YjUwNjVkY2ZhZWE4NTYzYTJjMDk2NThhYjE0N2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.bQ55yIRH2ish9AI4h6tCwbtA3J7tfsGyxK81T5ADmHc";>
 <img alt="image" width="579" height="22" 
src="https://private-user-images.githubusercontent.com/12249635/465733003-e1e3c6a9-97ed-4377-8020-02e40874a83d.png?jwt=eyJhb
 
GciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzMwMDMtZTFlM2M2YTktOTdlZC00Mzc3LTgwMjAtMDJlNDA4NzRhODNkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJmNTE5YWE5ZmUyNDU1YWIxZmJjZjVlZjFmMWRhNzY4MDk2NTIwNGYyZWUyOTExNGMzZGRhOTY0NGMwZjU5ZDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ndWejjDmI8u9XEdINtR_QBJmW96lhNU6T4FMOtE_9Zk">
   > <img alt="image" width="274" height="168" 
src="https://private-user-images.githubusercontent.com/12249635/465733133-a7f6984f-5fa0-41dd-ac95-9cdf6bd42705.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzMxMzMtYTdmNjk4NGYtNWZhMC00MWRkLWFjOTUtOWNkZjZiZDQyNzA1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzZTJiNWE3MjJjM2I2ZjMwZDBhZjMwYmE1YmQxMjQwOWYxYmNhNDI1YTZlMjc1OGExYTcwOTY2OTQxMmU3MWUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.7ziAHpdAF4VDMHdvKt8SGaNs7FqmhzpamahrQn-bq5M";>
   > 我们的场景是worker数多（200），worker usage比较小，rpc qps 10W+，会主动起一些bthread。
   > 
   > 
原因估计是_waiter_num并不能可靠地获取到当前有waiter，提起返回继续去signal下一个ParkingLot，使得_pending_signal递增更频繁，导致futex_wait_private失败的概率变大，futex_wait_private的次数也变多了。
   > 
   > 我觉得_waiter_num和_pending_signal的组合操作需要原子的，才能解决这个问题。
   > 
   > @JimChengLin CC
   
   我理解这个MR 只会比之前 futex wake 更少，而不会更多。是说 waiter num 错误的返回没有 waiter，导致 parking 
lot 产生热点了吗？_pending_signal 修改次数没有变化吧。从你描述的场景来看，worker num 比较多，导致几乎永远可以有空闲 
worker 来 futex 唤醒，所以有可能的问题是激烈竞争下，最优的策略是第一个 parking lot 就唤醒，而不要尝试后面的。在这个 MR 
之前，一个线程可能就访问一个 parking lot，在这个 MR 可能需要访问陌生的 parking lot？但我很怀疑把 waiter num 和 
signal 合并后，会有什么收益，这样对条件的竞争不是更严重吗？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org

Reply via email to