CC: [email protected] CC: [email protected] TO: Peter Zijlstra <[email protected]> CC: "André Almeida" <[email protected]>
tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master head: ee1703cda8dc777e937dec172da55beaf1a74919 commit: e5c6828493b5fa6a3c4606b43e80ab6c5ec1111f futex: Split out requeue date: 6 weeks ago :::::: branch date: 26 minutes ago :::::: commit date: 6 weeks ago config: m68k-randconfig-m031-20211117 (attached as .config) compiler: m68k-linux-gcc (GCC) 11.2.0 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <[email protected]> Reported-by: Dan Carpenter <[email protected]> smatch warnings: kernel/futex/requeue.c:462 futex_requeue() warn: bitwise AND condition is false here vim +462 kernel/futex/requeue.c e5c6828493b5fa Peter Zijlstra 2021-09-23 345 e5c6828493b5fa Peter Zijlstra 2021-09-23 346 /** e5c6828493b5fa Peter Zijlstra 2021-09-23 347 * futex_requeue() - Requeue waiters from uaddr1 to uaddr2 e5c6828493b5fa Peter Zijlstra 2021-09-23 348 * @uaddr1: source futex user address e5c6828493b5fa Peter Zijlstra 2021-09-23 349 * @flags: futex flags (FLAGS_SHARED, etc.) e5c6828493b5fa Peter Zijlstra 2021-09-23 350 * @uaddr2: target futex user address e5c6828493b5fa Peter Zijlstra 2021-09-23 351 * @nr_wake: number of waiters to wake (must be 1 for requeue_pi) e5c6828493b5fa Peter Zijlstra 2021-09-23 352 * @nr_requeue: number of waiters to requeue (0-INT_MAX) e5c6828493b5fa Peter Zijlstra 2021-09-23 353 * @cmpval: @uaddr1 expected value (or %NULL) e5c6828493b5fa Peter Zijlstra 2021-09-23 354 * @requeue_pi: if we are attempting to requeue from a non-pi futex to a e5c6828493b5fa Peter Zijlstra 2021-09-23 355 * pi futex (pi to pi requeue is not supported) e5c6828493b5fa Peter Zijlstra 2021-09-23 356 * e5c6828493b5fa Peter Zijlstra 2021-09-23 357 * Requeue waiters on uaddr1 to uaddr2. In the requeue_pi case, try to acquire e5c6828493b5fa Peter Zijlstra 2021-09-23 358 * uaddr2 atomically on behalf of the top waiter. e5c6828493b5fa Peter Zijlstra 2021-09-23 359 * e5c6828493b5fa Peter Zijlstra 2021-09-23 360 * Return: e5c6828493b5fa Peter Zijlstra 2021-09-23 361 * - >=0 - on success, the number of tasks requeued or woken; e5c6828493b5fa Peter Zijlstra 2021-09-23 362 * - <0 - on error e5c6828493b5fa Peter Zijlstra 2021-09-23 363 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 364 int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, e5c6828493b5fa Peter Zijlstra 2021-09-23 365 int nr_wake, int nr_requeue, u32 *cmpval, int requeue_pi) e5c6828493b5fa Peter Zijlstra 2021-09-23 366 { e5c6828493b5fa Peter Zijlstra 2021-09-23 367 union futex_key key1 = FUTEX_KEY_INIT, key2 = FUTEX_KEY_INIT; e5c6828493b5fa Peter Zijlstra 2021-09-23 368 int task_count = 0, ret; e5c6828493b5fa Peter Zijlstra 2021-09-23 369 struct futex_pi_state *pi_state = NULL; e5c6828493b5fa Peter Zijlstra 2021-09-23 370 struct futex_hash_bucket *hb1, *hb2; e5c6828493b5fa Peter Zijlstra 2021-09-23 371 struct futex_q *this, *next; e5c6828493b5fa Peter Zijlstra 2021-09-23 372 DEFINE_WAKE_Q(wake_q); e5c6828493b5fa Peter Zijlstra 2021-09-23 373 e5c6828493b5fa Peter Zijlstra 2021-09-23 374 if (nr_wake < 0 || nr_requeue < 0) e5c6828493b5fa Peter Zijlstra 2021-09-23 375 return -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 376 e5c6828493b5fa Peter Zijlstra 2021-09-23 377 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 378 * When PI not supported: return -ENOSYS if requeue_pi is true, e5c6828493b5fa Peter Zijlstra 2021-09-23 379 * consequently the compiler knows requeue_pi is always false past e5c6828493b5fa Peter Zijlstra 2021-09-23 380 * this point which will optimize away all the conditional code e5c6828493b5fa Peter Zijlstra 2021-09-23 381 * further down. e5c6828493b5fa Peter Zijlstra 2021-09-23 382 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 383 if (!IS_ENABLED(CONFIG_FUTEX_PI) && requeue_pi) e5c6828493b5fa Peter Zijlstra 2021-09-23 384 return -ENOSYS; e5c6828493b5fa Peter Zijlstra 2021-09-23 385 e5c6828493b5fa Peter Zijlstra 2021-09-23 386 if (requeue_pi) { e5c6828493b5fa Peter Zijlstra 2021-09-23 387 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 388 * Requeue PI only works on two distinct uaddrs. This e5c6828493b5fa Peter Zijlstra 2021-09-23 389 * check is only valid for private futexes. See below. e5c6828493b5fa Peter Zijlstra 2021-09-23 390 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 391 if (uaddr1 == uaddr2) e5c6828493b5fa Peter Zijlstra 2021-09-23 392 return -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 393 e5c6828493b5fa Peter Zijlstra 2021-09-23 394 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 395 * futex_requeue() allows the caller to define the number e5c6828493b5fa Peter Zijlstra 2021-09-23 396 * of waiters to wake up via the @nr_wake argument. With e5c6828493b5fa Peter Zijlstra 2021-09-23 397 * REQUEUE_PI, waking up more than one waiter is creating e5c6828493b5fa Peter Zijlstra 2021-09-23 398 * more problems than it solves. Waking up a waiter makes e5c6828493b5fa Peter Zijlstra 2021-09-23 399 * only sense if the PI futex @uaddr2 is uncontended as e5c6828493b5fa Peter Zijlstra 2021-09-23 400 * this allows the requeue code to acquire the futex e5c6828493b5fa Peter Zijlstra 2021-09-23 401 * @uaddr2 before waking the waiter. The waiter can then e5c6828493b5fa Peter Zijlstra 2021-09-23 402 * return to user space without further action. A secondary e5c6828493b5fa Peter Zijlstra 2021-09-23 403 * wakeup would just make the futex_wait_requeue_pi() e5c6828493b5fa Peter Zijlstra 2021-09-23 404 * handling more complex, because that code would have to e5c6828493b5fa Peter Zijlstra 2021-09-23 405 * look up pi_state and do more or less all the handling e5c6828493b5fa Peter Zijlstra 2021-09-23 406 * which the requeue code has to do for the to be requeued e5c6828493b5fa Peter Zijlstra 2021-09-23 407 * waiters. So restrict the number of waiters to wake to e5c6828493b5fa Peter Zijlstra 2021-09-23 408 * one, and only wake it up when the PI futex is e5c6828493b5fa Peter Zijlstra 2021-09-23 409 * uncontended. Otherwise requeue it and let the unlock of e5c6828493b5fa Peter Zijlstra 2021-09-23 410 * the PI futex handle the wakeup. e5c6828493b5fa Peter Zijlstra 2021-09-23 411 * e5c6828493b5fa Peter Zijlstra 2021-09-23 412 * All REQUEUE_PI users, e.g. pthread_cond_signal() and e5c6828493b5fa Peter Zijlstra 2021-09-23 413 * pthread_cond_broadcast() must use nr_wake=1. e5c6828493b5fa Peter Zijlstra 2021-09-23 414 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 415 if (nr_wake != 1) e5c6828493b5fa Peter Zijlstra 2021-09-23 416 return -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 417 e5c6828493b5fa Peter Zijlstra 2021-09-23 418 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 419 * requeue_pi requires a pi_state, try to allocate it now e5c6828493b5fa Peter Zijlstra 2021-09-23 420 * without any locks in case it fails. e5c6828493b5fa Peter Zijlstra 2021-09-23 421 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 422 if (refill_pi_state_cache()) e5c6828493b5fa Peter Zijlstra 2021-09-23 423 return -ENOMEM; e5c6828493b5fa Peter Zijlstra 2021-09-23 424 } e5c6828493b5fa Peter Zijlstra 2021-09-23 425 e5c6828493b5fa Peter Zijlstra 2021-09-23 426 retry: e5c6828493b5fa Peter Zijlstra 2021-09-23 427 ret = get_futex_key(uaddr1, flags & FLAGS_SHARED, &key1, FUTEX_READ); e5c6828493b5fa Peter Zijlstra 2021-09-23 428 if (unlikely(ret != 0)) e5c6828493b5fa Peter Zijlstra 2021-09-23 429 return ret; e5c6828493b5fa Peter Zijlstra 2021-09-23 430 ret = get_futex_key(uaddr2, flags & FLAGS_SHARED, &key2, e5c6828493b5fa Peter Zijlstra 2021-09-23 431 requeue_pi ? FUTEX_WRITE : FUTEX_READ); e5c6828493b5fa Peter Zijlstra 2021-09-23 432 if (unlikely(ret != 0)) e5c6828493b5fa Peter Zijlstra 2021-09-23 433 return ret; e5c6828493b5fa Peter Zijlstra 2021-09-23 434 e5c6828493b5fa Peter Zijlstra 2021-09-23 435 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 436 * The check above which compares uaddrs is not sufficient for e5c6828493b5fa Peter Zijlstra 2021-09-23 437 * shared futexes. We need to compare the keys: e5c6828493b5fa Peter Zijlstra 2021-09-23 438 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 439 if (requeue_pi && futex_match(&key1, &key2)) e5c6828493b5fa Peter Zijlstra 2021-09-23 440 return -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 441 e5c6828493b5fa Peter Zijlstra 2021-09-23 442 hb1 = futex_hash(&key1); e5c6828493b5fa Peter Zijlstra 2021-09-23 443 hb2 = futex_hash(&key2); e5c6828493b5fa Peter Zijlstra 2021-09-23 444 e5c6828493b5fa Peter Zijlstra 2021-09-23 445 retry_private: e5c6828493b5fa Peter Zijlstra 2021-09-23 446 futex_hb_waiters_inc(hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 447 double_lock_hb(hb1, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 448 e5c6828493b5fa Peter Zijlstra 2021-09-23 449 if (likely(cmpval != NULL)) { e5c6828493b5fa Peter Zijlstra 2021-09-23 450 u32 curval; e5c6828493b5fa Peter Zijlstra 2021-09-23 451 e5c6828493b5fa Peter Zijlstra 2021-09-23 452 ret = futex_get_value_locked(&curval, uaddr1); e5c6828493b5fa Peter Zijlstra 2021-09-23 453 e5c6828493b5fa Peter Zijlstra 2021-09-23 454 if (unlikely(ret)) { e5c6828493b5fa Peter Zijlstra 2021-09-23 455 double_unlock_hb(hb1, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 456 futex_hb_waiters_dec(hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 457 e5c6828493b5fa Peter Zijlstra 2021-09-23 458 ret = get_user(curval, uaddr1); e5c6828493b5fa Peter Zijlstra 2021-09-23 459 if (ret) e5c6828493b5fa Peter Zijlstra 2021-09-23 460 return ret; e5c6828493b5fa Peter Zijlstra 2021-09-23 461 e5c6828493b5fa Peter Zijlstra 2021-09-23 @462 if (!(flags & FLAGS_SHARED)) e5c6828493b5fa Peter Zijlstra 2021-09-23 463 goto retry_private; e5c6828493b5fa Peter Zijlstra 2021-09-23 464 e5c6828493b5fa Peter Zijlstra 2021-09-23 465 goto retry; e5c6828493b5fa Peter Zijlstra 2021-09-23 466 } e5c6828493b5fa Peter Zijlstra 2021-09-23 467 if (curval != *cmpval) { e5c6828493b5fa Peter Zijlstra 2021-09-23 468 ret = -EAGAIN; e5c6828493b5fa Peter Zijlstra 2021-09-23 469 goto out_unlock; e5c6828493b5fa Peter Zijlstra 2021-09-23 470 } e5c6828493b5fa Peter Zijlstra 2021-09-23 471 } e5c6828493b5fa Peter Zijlstra 2021-09-23 472 e5c6828493b5fa Peter Zijlstra 2021-09-23 473 if (requeue_pi) { e5c6828493b5fa Peter Zijlstra 2021-09-23 474 struct task_struct *exiting = NULL; e5c6828493b5fa Peter Zijlstra 2021-09-23 475 e5c6828493b5fa Peter Zijlstra 2021-09-23 476 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 477 * Attempt to acquire uaddr2 and wake the top waiter. If we e5c6828493b5fa Peter Zijlstra 2021-09-23 478 * intend to requeue waiters, force setting the FUTEX_WAITERS e5c6828493b5fa Peter Zijlstra 2021-09-23 479 * bit. We force this here where we are able to easily handle e5c6828493b5fa Peter Zijlstra 2021-09-23 480 * faults rather in the requeue loop below. e5c6828493b5fa Peter Zijlstra 2021-09-23 481 * e5c6828493b5fa Peter Zijlstra 2021-09-23 482 * Updates topwaiter::requeue_state if a top waiter exists. e5c6828493b5fa Peter Zijlstra 2021-09-23 483 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 484 ret = futex_proxy_trylock_atomic(uaddr2, hb1, hb2, &key1, e5c6828493b5fa Peter Zijlstra 2021-09-23 485 &key2, &pi_state, e5c6828493b5fa Peter Zijlstra 2021-09-23 486 &exiting, nr_requeue); e5c6828493b5fa Peter Zijlstra 2021-09-23 487 e5c6828493b5fa Peter Zijlstra 2021-09-23 488 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 489 * At this point the top_waiter has either taken uaddr2 or e5c6828493b5fa Peter Zijlstra 2021-09-23 490 * is waiting on it. In both cases pi_state has been e5c6828493b5fa Peter Zijlstra 2021-09-23 491 * established and an initial refcount on it. In case of an e5c6828493b5fa Peter Zijlstra 2021-09-23 492 * error there's nothing. e5c6828493b5fa Peter Zijlstra 2021-09-23 493 * e5c6828493b5fa Peter Zijlstra 2021-09-23 494 * The top waiter's requeue_state is up to date: e5c6828493b5fa Peter Zijlstra 2021-09-23 495 * e5c6828493b5fa Peter Zijlstra 2021-09-23 496 * - If the lock was acquired atomically (ret == 1), then e5c6828493b5fa Peter Zijlstra 2021-09-23 497 * the state is Q_REQUEUE_PI_LOCKED. e5c6828493b5fa Peter Zijlstra 2021-09-23 498 * e5c6828493b5fa Peter Zijlstra 2021-09-23 499 * The top waiter has been dequeued and woken up and can e5c6828493b5fa Peter Zijlstra 2021-09-23 500 * return to user space immediately. The kernel/user e5c6828493b5fa Peter Zijlstra 2021-09-23 501 * space state is consistent. In case that there must be e5c6828493b5fa Peter Zijlstra 2021-09-23 502 * more waiters requeued the WAITERS bit in the user e5c6828493b5fa Peter Zijlstra 2021-09-23 503 * space futex is set so the top waiter task has to go e5c6828493b5fa Peter Zijlstra 2021-09-23 504 * into the syscall slowpath to unlock the futex. This e5c6828493b5fa Peter Zijlstra 2021-09-23 505 * will block until this requeue operation has been e5c6828493b5fa Peter Zijlstra 2021-09-23 506 * completed and the hash bucket locks have been e5c6828493b5fa Peter Zijlstra 2021-09-23 507 * dropped. e5c6828493b5fa Peter Zijlstra 2021-09-23 508 * e5c6828493b5fa Peter Zijlstra 2021-09-23 509 * - If the trylock failed with an error (ret < 0) then e5c6828493b5fa Peter Zijlstra 2021-09-23 510 * the state is either Q_REQUEUE_PI_NONE, i.e. "nothing e5c6828493b5fa Peter Zijlstra 2021-09-23 511 * happened", or Q_REQUEUE_PI_IGNORE when there was an e5c6828493b5fa Peter Zijlstra 2021-09-23 512 * interleaved early wakeup. e5c6828493b5fa Peter Zijlstra 2021-09-23 513 * e5c6828493b5fa Peter Zijlstra 2021-09-23 514 * - If the trylock did not succeed (ret == 0) then the e5c6828493b5fa Peter Zijlstra 2021-09-23 515 * state is either Q_REQUEUE_PI_IN_PROGRESS or e5c6828493b5fa Peter Zijlstra 2021-09-23 516 * Q_REQUEUE_PI_WAIT if an early wakeup interleaved. e5c6828493b5fa Peter Zijlstra 2021-09-23 517 * This will be cleaned up in the loop below, which e5c6828493b5fa Peter Zijlstra 2021-09-23 518 * cannot fail because futex_proxy_trylock_atomic() did e5c6828493b5fa Peter Zijlstra 2021-09-23 519 * the same sanity checks for requeue_pi as the loop e5c6828493b5fa Peter Zijlstra 2021-09-23 520 * below does. e5c6828493b5fa Peter Zijlstra 2021-09-23 521 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 522 switch (ret) { e5c6828493b5fa Peter Zijlstra 2021-09-23 523 case 0: e5c6828493b5fa Peter Zijlstra 2021-09-23 524 /* We hold a reference on the pi state. */ e5c6828493b5fa Peter Zijlstra 2021-09-23 525 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 526 e5c6828493b5fa Peter Zijlstra 2021-09-23 527 case 1: e5c6828493b5fa Peter Zijlstra 2021-09-23 528 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 529 * futex_proxy_trylock_atomic() acquired the user space e5c6828493b5fa Peter Zijlstra 2021-09-23 530 * futex. Adjust task_count. e5c6828493b5fa Peter Zijlstra 2021-09-23 531 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 532 task_count++; e5c6828493b5fa Peter Zijlstra 2021-09-23 533 ret = 0; e5c6828493b5fa Peter Zijlstra 2021-09-23 534 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 535 e5c6828493b5fa Peter Zijlstra 2021-09-23 536 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 537 * If the above failed, then pi_state is NULL and e5c6828493b5fa Peter Zijlstra 2021-09-23 538 * waiter::requeue_state is correct. e5c6828493b5fa Peter Zijlstra 2021-09-23 539 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 540 case -EFAULT: e5c6828493b5fa Peter Zijlstra 2021-09-23 541 double_unlock_hb(hb1, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 542 futex_hb_waiters_dec(hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 543 ret = fault_in_user_writeable(uaddr2); e5c6828493b5fa Peter Zijlstra 2021-09-23 544 if (!ret) e5c6828493b5fa Peter Zijlstra 2021-09-23 545 goto retry; e5c6828493b5fa Peter Zijlstra 2021-09-23 546 return ret; e5c6828493b5fa Peter Zijlstra 2021-09-23 547 case -EBUSY: e5c6828493b5fa Peter Zijlstra 2021-09-23 548 case -EAGAIN: e5c6828493b5fa Peter Zijlstra 2021-09-23 549 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 550 * Two reasons for this: e5c6828493b5fa Peter Zijlstra 2021-09-23 551 * - EBUSY: Owner is exiting and we just wait for the e5c6828493b5fa Peter Zijlstra 2021-09-23 552 * exit to complete. e5c6828493b5fa Peter Zijlstra 2021-09-23 553 * - EAGAIN: The user space value changed. e5c6828493b5fa Peter Zijlstra 2021-09-23 554 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 555 double_unlock_hb(hb1, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 556 futex_hb_waiters_dec(hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 557 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 558 * Handle the case where the owner is in the middle of e5c6828493b5fa Peter Zijlstra 2021-09-23 559 * exiting. Wait for the exit to complete otherwise e5c6828493b5fa Peter Zijlstra 2021-09-23 560 * this task might loop forever, aka. live lock. e5c6828493b5fa Peter Zijlstra 2021-09-23 561 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 562 wait_for_owner_exiting(ret, exiting); e5c6828493b5fa Peter Zijlstra 2021-09-23 563 cond_resched(); e5c6828493b5fa Peter Zijlstra 2021-09-23 564 goto retry; e5c6828493b5fa Peter Zijlstra 2021-09-23 565 default: e5c6828493b5fa Peter Zijlstra 2021-09-23 566 goto out_unlock; e5c6828493b5fa Peter Zijlstra 2021-09-23 567 } e5c6828493b5fa Peter Zijlstra 2021-09-23 568 } e5c6828493b5fa Peter Zijlstra 2021-09-23 569 e5c6828493b5fa Peter Zijlstra 2021-09-23 570 plist_for_each_entry_safe(this, next, &hb1->chain, list) { e5c6828493b5fa Peter Zijlstra 2021-09-23 571 if (task_count - nr_wake >= nr_requeue) e5c6828493b5fa Peter Zijlstra 2021-09-23 572 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 573 e5c6828493b5fa Peter Zijlstra 2021-09-23 574 if (!futex_match(&this->key, &key1)) e5c6828493b5fa Peter Zijlstra 2021-09-23 575 continue; e5c6828493b5fa Peter Zijlstra 2021-09-23 576 e5c6828493b5fa Peter Zijlstra 2021-09-23 577 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 578 * FUTEX_WAIT_REQUEUE_PI and FUTEX_CMP_REQUEUE_PI should always e5c6828493b5fa Peter Zijlstra 2021-09-23 579 * be paired with each other and no other futex ops. e5c6828493b5fa Peter Zijlstra 2021-09-23 580 * e5c6828493b5fa Peter Zijlstra 2021-09-23 581 * We should never be requeueing a futex_q with a pi_state, e5c6828493b5fa Peter Zijlstra 2021-09-23 582 * which is awaiting a futex_unlock_pi(). e5c6828493b5fa Peter Zijlstra 2021-09-23 583 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 584 if ((requeue_pi && !this->rt_waiter) || e5c6828493b5fa Peter Zijlstra 2021-09-23 585 (!requeue_pi && this->rt_waiter) || e5c6828493b5fa Peter Zijlstra 2021-09-23 586 this->pi_state) { e5c6828493b5fa Peter Zijlstra 2021-09-23 587 ret = -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 588 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 589 } e5c6828493b5fa Peter Zijlstra 2021-09-23 590 e5c6828493b5fa Peter Zijlstra 2021-09-23 591 /* Plain futexes just wake or requeue and are done */ e5c6828493b5fa Peter Zijlstra 2021-09-23 592 if (!requeue_pi) { e5c6828493b5fa Peter Zijlstra 2021-09-23 593 if (++task_count <= nr_wake) e5c6828493b5fa Peter Zijlstra 2021-09-23 594 futex_wake_mark(&wake_q, this); e5c6828493b5fa Peter Zijlstra 2021-09-23 595 else e5c6828493b5fa Peter Zijlstra 2021-09-23 596 requeue_futex(this, hb1, hb2, &key2); e5c6828493b5fa Peter Zijlstra 2021-09-23 597 continue; e5c6828493b5fa Peter Zijlstra 2021-09-23 598 } e5c6828493b5fa Peter Zijlstra 2021-09-23 599 e5c6828493b5fa Peter Zijlstra 2021-09-23 600 /* Ensure we requeue to the expected futex for requeue_pi. */ e5c6828493b5fa Peter Zijlstra 2021-09-23 601 if (!futex_match(this->requeue_pi_key, &key2)) { e5c6828493b5fa Peter Zijlstra 2021-09-23 602 ret = -EINVAL; e5c6828493b5fa Peter Zijlstra 2021-09-23 603 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 604 } e5c6828493b5fa Peter Zijlstra 2021-09-23 605 e5c6828493b5fa Peter Zijlstra 2021-09-23 606 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 607 * Requeue nr_requeue waiters and possibly one more in the case e5c6828493b5fa Peter Zijlstra 2021-09-23 608 * of requeue_pi if we couldn't acquire the lock atomically. e5c6828493b5fa Peter Zijlstra 2021-09-23 609 * e5c6828493b5fa Peter Zijlstra 2021-09-23 610 * Prepare the waiter to take the rt_mutex. Take a refcount e5c6828493b5fa Peter Zijlstra 2021-09-23 611 * on the pi_state and store the pointer in the futex_q e5c6828493b5fa Peter Zijlstra 2021-09-23 612 * object of the waiter. e5c6828493b5fa Peter Zijlstra 2021-09-23 613 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 614 get_pi_state(pi_state); e5c6828493b5fa Peter Zijlstra 2021-09-23 615 e5c6828493b5fa Peter Zijlstra 2021-09-23 616 /* Don't requeue when the waiter is already on the way out. */ e5c6828493b5fa Peter Zijlstra 2021-09-23 617 if (!futex_requeue_pi_prepare(this, pi_state)) { e5c6828493b5fa Peter Zijlstra 2021-09-23 618 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 619 * Early woken waiter signaled that it is on the e5c6828493b5fa Peter Zijlstra 2021-09-23 620 * way out. Drop the pi_state reference and try the e5c6828493b5fa Peter Zijlstra 2021-09-23 621 * next waiter. @this->pi_state is still NULL. e5c6828493b5fa Peter Zijlstra 2021-09-23 622 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 623 put_pi_state(pi_state); e5c6828493b5fa Peter Zijlstra 2021-09-23 624 continue; e5c6828493b5fa Peter Zijlstra 2021-09-23 625 } e5c6828493b5fa Peter Zijlstra 2021-09-23 626 e5c6828493b5fa Peter Zijlstra 2021-09-23 627 ret = rt_mutex_start_proxy_lock(&pi_state->pi_mutex, e5c6828493b5fa Peter Zijlstra 2021-09-23 628 this->rt_waiter, e5c6828493b5fa Peter Zijlstra 2021-09-23 629 this->task); e5c6828493b5fa Peter Zijlstra 2021-09-23 630 e5c6828493b5fa Peter Zijlstra 2021-09-23 631 if (ret == 1) { e5c6828493b5fa Peter Zijlstra 2021-09-23 632 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 633 * We got the lock. We do neither drop the refcount e5c6828493b5fa Peter Zijlstra 2021-09-23 634 * on pi_state nor clear this->pi_state because the e5c6828493b5fa Peter Zijlstra 2021-09-23 635 * waiter needs the pi_state for cleaning up the e5c6828493b5fa Peter Zijlstra 2021-09-23 636 * user space value. It will drop the refcount e5c6828493b5fa Peter Zijlstra 2021-09-23 637 * after doing so. this::requeue_state is updated e5c6828493b5fa Peter Zijlstra 2021-09-23 638 * in the wakeup as well. e5c6828493b5fa Peter Zijlstra 2021-09-23 639 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 640 requeue_pi_wake_futex(this, &key2, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 641 task_count++; e5c6828493b5fa Peter Zijlstra 2021-09-23 642 } else if (!ret) { e5c6828493b5fa Peter Zijlstra 2021-09-23 643 /* Waiter is queued, move it to hb2 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 644 requeue_futex(this, hb1, hb2, &key2); e5c6828493b5fa Peter Zijlstra 2021-09-23 645 futex_requeue_pi_complete(this, 0); e5c6828493b5fa Peter Zijlstra 2021-09-23 646 task_count++; e5c6828493b5fa Peter Zijlstra 2021-09-23 647 } else { e5c6828493b5fa Peter Zijlstra 2021-09-23 648 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 649 * rt_mutex_start_proxy_lock() detected a potential e5c6828493b5fa Peter Zijlstra 2021-09-23 650 * deadlock when we tried to queue that waiter. e5c6828493b5fa Peter Zijlstra 2021-09-23 651 * Drop the pi_state reference which we took above e5c6828493b5fa Peter Zijlstra 2021-09-23 652 * and remove the pointer to the state from the e5c6828493b5fa Peter Zijlstra 2021-09-23 653 * waiters futex_q object. e5c6828493b5fa Peter Zijlstra 2021-09-23 654 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 655 this->pi_state = NULL; e5c6828493b5fa Peter Zijlstra 2021-09-23 656 put_pi_state(pi_state); e5c6828493b5fa Peter Zijlstra 2021-09-23 657 futex_requeue_pi_complete(this, ret); e5c6828493b5fa Peter Zijlstra 2021-09-23 658 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 659 * We stop queueing more waiters and let user space e5c6828493b5fa Peter Zijlstra 2021-09-23 660 * deal with the mess. e5c6828493b5fa Peter Zijlstra 2021-09-23 661 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 662 break; e5c6828493b5fa Peter Zijlstra 2021-09-23 663 } e5c6828493b5fa Peter Zijlstra 2021-09-23 664 } e5c6828493b5fa Peter Zijlstra 2021-09-23 665 e5c6828493b5fa Peter Zijlstra 2021-09-23 666 /* e5c6828493b5fa Peter Zijlstra 2021-09-23 667 * We took an extra initial reference to the pi_state in e5c6828493b5fa Peter Zijlstra 2021-09-23 668 * futex_proxy_trylock_atomic(). We need to drop it here again. e5c6828493b5fa Peter Zijlstra 2021-09-23 669 */ e5c6828493b5fa Peter Zijlstra 2021-09-23 670 put_pi_state(pi_state); e5c6828493b5fa Peter Zijlstra 2021-09-23 671 e5c6828493b5fa Peter Zijlstra 2021-09-23 672 out_unlock: e5c6828493b5fa Peter Zijlstra 2021-09-23 673 double_unlock_hb(hb1, hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 674 wake_up_q(&wake_q); e5c6828493b5fa Peter Zijlstra 2021-09-23 675 futex_hb_waiters_dec(hb2); e5c6828493b5fa Peter Zijlstra 2021-09-23 676 return ret ? ret : task_count; e5c6828493b5fa Peter Zijlstra 2021-09-23 677 } e5c6828493b5fa Peter Zijlstra 2021-09-23 678 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/[email protected]
.config.gz
Description: application/gzip
_______________________________________________ kbuild mailing list -- [email protected] To unsubscribe send an email to [email protected]
