Issue created by Sebastian Huber:
https://gitlab.rtems.org/rtems/rtos/rtems/-/work_items/5733
Assignee: Sebastian Huber
## Summary
This is a severe bug which should be fixed in all production systems.
A thread timeout can cancel a wait which the thread started after that timeout
expired. The score has no way to name the wait which a timeout belongs to.
The thread then gets a wrong `STATUS_TIMEOUT` for an unrelated wait, or the
watchdog tree of a processor becomes corrupt and a later clock tick stops every
processor.
## The mechanism
`_Watchdog_Do_tickle()` extracts the node from the tree, sets it inactive and
releases the lock of the header. It calls the service routine after that. It
never acquires the `Timer.Lock` of the thread, which is the lock the arm and
the removal of a thread timer use.
For a thread timer the routine is `_Thread_Timeout()`. That routine calls
`_Thread_Continue()` with `STATUS_TIMEOUT`.
Between the release of the header lock and the call, another party can satisfy
the wait. The `_Thread_Timer_remove()` of that party finds an inactive node
and reports success, although the routine is still in flight. The score has no
cancel and drain for a thread timer. The thread leaves its wait, continues,
and enters a new wait.
`_Thread_Continue()` then acquires the wait lock of the new wait. It tests
`wait_flags != THREAD_WAIT_STATE_READY` and derives the wait class from the
value it reads. Both tests pass for any wait, so it cancels the new one.
The thread wait flags cannot separate one wait of a thread from the next. The
state runs `READY`, `INTEND_TO_BLOCK`, `BLOCKED`, `READY` and starts again. It
is a generation counter of three values which resets at every wait.
## What the cancel leaves behind
| The new wait of the thread | Result |
| -------------------------- | -------------------------------------- |
| carries a timeout | the watchdog it armed, still scheduled |
| carries no timeout | a report of a timeout which never ran |
A leaked watchdog stays in the tree of a processor. The next timed wait of
that thread inserts the same node a second time. The red-black tree of that
processor is then corrupt. A later clock tick either traps in
`_RBTree_Extract()` or turns in an endless loop with the lock of the tree held.
The second case stops every processor.
## Exposure
Every wait which `_Thread_Continue()` ends is exposed. That covers each call
which blocks on a thread queue, `rtems_event_receive()`,
`rtems_task_wake_after()`, `rtems_task_wake_when()` and the POSIX counterparts
of these calls.
On an SMP configuration the party which satisfies the wait is an interrupt
handler or a thread on another processor. The window is open on every timed
wait.
On a uniprocessor configuration the window opens through `_TOD_Set()`. That
routine tickles the realtime header of each processor in task context.
`rtems_clock_set()` holds a mutex and an ISR lock, and
`_Timecounter_Set_clock()`
releases that ISR lock before the loop. Interrupts and thread dispatch are both
enabled while the loop calls `_Thread_Timeout()`.
## Affected versions
| Change | Date | Releases
|
| --------------------------------------------- | ---------- |
--------------------- |
| `score: Add thread wait flags` | 2015-03-04 | 4.11 and later
|
| `score: Replace watchdog handler implementation` | 2016-02-18 | 5.1 and later
|
| `score: Add _Thread_Continue()` | 2017-10-19 | 5.1 and later
|
The mechanism above is present in 5.1, 5.2, 5.3, 6.1, 6.2 and the development
head. Release 4.11 carries the thread wait flags but a different watchdog
handler. Nobody checked 4.11 against this defect.
## How to construct the defect without a race
The `_TOD_Set()` path makes the window deterministic on a single processor.
1. A high priority task blocks on a semaphore with an absolute realtime
timeout.
2. A low priority task calls `rtems_clock_set()` with a time past that expiry.
3. `_TOD_Set()` extracts the watchdog, releases the header lock and calls
`_Thread_Timeout()`.
4. An interrupt releases the semaphore inside that window.
5. The high priority task preempts, returns from the wait and enters a second
wait.
6. The low priority task resumes and cancels the second wait.
Without the defect the second wait stays intact. With the defect it reports a
timeout, and it leaks its watchdog when it carries one.
## The repair
The thread wait flags word is an `unsigned int` and uses bits 0 to 15 only.
Bits 16 to 31 hold a generation which steps at every wait start. The watchdog
carries the generation which the arm wrote. `_Watchdog_Do_tickle()` reads that
value under the header lock and passes it to the service routine. This is the
only transport which survives a second arm of the same node, because the arm of
the next wait overwrites every field of the node.
`_Thread_Timeout()` compares the value against the current generation under the
wait lock. On a mismatch it returns and touches nothing.
The bug report was created using Claude Code.
--
View it on GitLab: https://gitlab.rtems.org/rtems/rtos/rtems/-/work_items/5733
You're receiving this email because of your account on gitlab.rtems.org.
Unsubscribe from this thread:
https://gitlab.rtems.org/-/sent_notifications/5-87dswag3ss2c6tl7w2xqxeg92-1d/unsubscribe
| Manage all notifications: https://gitlab.rtems.org/-/profile/notifications |
Help: https://gitlab.rtems.org/help
_______________________________________________
bugs mailing list
[email protected]
http://lists.rtems.org/mailman/listinfo/bugs
