vrmay23 opened a new pull request, #20394:
URL: https://github.com/apache/nuttx/pull/20394
## Summary
**Why.** NuttX is a small, minimalist RTOS. The documentation is not small
and
not organised, and that gap is what this PR fixes.
Right now `Documentation/` has **19 entries at the top level**, and two of
them
are not really sections, they are boxes where pages were put because there
was
no better place. "OS Components" has 21 pages. "Implementation Details" has
37.
The names do not tell you what is inside.
And the names do not match the contents. "Architecture APIs" lists 18 pages,
but only 2 of them are architecture APIs (`arch.rst` and `board.rst`). One of
the other 16 is *Congestion Control NewReno* — a TCP algorithm, sitting next
to
`up_initialize()`.
The Guides index starts with a `.. todo::` asking for tutorial pages to be
written, then lists 57 pages in no particular order. That `todo` is one of
**35** that
are published on the live site, because `conf.py` has
`todo_include_todos = True`.
So the site grew page by page, with no plan. If you come to port a board, or
to understand how the scheduler picks a thread, you meet a list of categories
that match nothing you can see in the source.
**What this does.** It files every page under the part of the source tree it
talks about. `sched/` is documented in `os/scheduling/`, `fs/` in
`os/filesystem/`, `drivers/` in `os/drivers/`, and so on. The top level goes
from 19 entries to **10**, and each one has a sentence saying what it is for.
Nothing here is invented. The structure is the one every NuttX developer
already knows, because it is the source tree.
**What else changed.** Reading each page against the code showed places where
the page and the code disagree. When that happened, the code won. Some
examples:
* `net/netdev/` is a registry you reach from both sides — `psock_ioctl()` is
in
`netdev_ioctl.c`, and drivers come in through `netdev_register()`. It is
not
a layer in the data path. The old ASCII diagram drew it as a separate
column.
* `RPTUN` does `select RPMSG_VIRTIO`, so RPMSG is the layer and RPTUN is one
way to carry it. The page had it the other way round.
* `SCHED_FIFO` is not the default when round robin is on.
`nxthread_setup_scheduler()` picks the policy with a compile-time test, so
`CONFIG_RR_INTERVAL` changes it for the whole system.
* `g_waitingforsemaphore`, `g_waitingformqnotempty` and
`g_waitingformqnotfull` appear in five comments and are declared nowhere.
Waiters now queue inside the object they wait on, through
`g_tasklisttable[]` and `TLIST_ATTR_OFFSET`.
**50 pages** have new or rewritten text. They were empty, or a `.. todo::`,
or
they said something the code contradicts. Every other page keeps the text
that
is already in master.
The diff is big — 1039 files — because moving a page changes every link that
points to it. But the text change is small, and you can check exactly how
small:
| | pages |
|---|---|
| never touched | 782 |
| changed by the move only | 447 |
| moved, byte for byte the same | 291 |
| **new or rewritten text** | **50** |
| halves of one page that was split | 2 |
| new, no text (just a toctree) | 1 |
| **total `.rst` in HEAD** | **1573** |
"Changed by the move only" is not an opinion. Take both versions, delete
everything a move touches — link targets, paths, URLs, tag lines, toctree
blocks, table borders — and what is left is byte for byte the same. No
sentence
was reworded. No section was renamed.
**No page was deleted, and no URL breaks.** `Documentation/redirects.py` has
**520 rules**, and every target was checked to point at a page that exists.
The
NuttX docs are linked from issues, mailing lists and slides going back years,
so breaking those links would not be worth it.
## Impact
* **New feature / existing feature changed?** NO for code. The diff touches
`Documentation/` and nothing else — no `.c`, no `.h`, no build file, no
workflow. 1039 files, all of them under `Documentation/`.
* **Impact on user?** YES, small, and handled. Doc URLs change. The 520
redirect rules keep every old path working, and each one was verified.
* **Impact on build?** NO for the firmware build. The doc build uses the
same
inputs and options as before and is clean under `-W`.
* **Impact on hardware?** NO. No arch, board or driver source is touched.
* **Impact on documentation?** YES — this is the change itself.
* **Impact on security?** NO.
* **Impact on compatibility?** NO. This is not a breaking change, and we
tested it: old URLs still resolve, the build is clean, no source file
changes.
* **Anything else?** You do not have to read a thousand files to trust the
"no text changed" claim. There is a script for it, see Testing.
## Testing
I confirm that changes are verified on local setup and works as intended:
* Build host: Linux x86_64, Python 3.12.7, Sphinx 6.2.1, docutils 0.19.
`Documentation/Pipfile` asks for `Sphinx ~= 6.0` and `docutils == 0.19`.
* Target: none. No source outside `Documentation/` is touched.
* `./tools/checkpatch.sh -g <base>...HEAD` exits 0. The only Python in the
diff is `conf.py`, `redirects.py` and the tag-index extension, and black,
isort and flake8 are all clean on them.
**How you can check this yourself.** All of this runs from the repo root and
needs only git and Python. None of it asks you to trust this text.
1. **The "only the layout changed" claim.** The script is posted as the first
comment on this PR — it is a one-off checking tool, not something that
belongs in the tree. It takes the base commit and checks every `.rst` page
in HEAD. For each page
not in the declared list, it deletes everything a move touches from both
versions and requires what is left to be identical. It exits non-zero and
names the page if any page disagrees:
```
python3 validate_reorg.py <base-commit>
```
It also tests each new page for text instead of assuming. If you forget to
declare a page, it fails. That check exists because an earlier version
trusted the list, and an independent audit found a page the list had
missed.
2. **The build.** Sphinx runs with `-W`, so a warning is an error:
```
cd Documentation && make html
```
3. **The redirects.**
```
cd Documentation && python3 -c "
import sys, os, posixpath; sys.path.insert(0, '.')
from redirects import redirects
bad = [o for o, v in redirects.items()
if not any(os.path.exists(
posixpath.normpath(posixpath.join(posixpath.dirname(o),
v))[:-5] + e)
for e in ('.rst', '.md'))]
print(len(redirects), 'rules,', len(bad), 'broken')"
```
4. **Provenance**, if you want to be sure nothing strange got in:
```
# every external host in added lines
git diff <base>..HEAD -- Documentation/ | grep '^+' |
grep -oE 'https?://[^ )>"]+' | sed -E 's|(https?://[^/]+).*|\1|' | sort
| uniq -c
# anything touched outside Documentation/ (empty here)
git diff <base>..HEAD --name-only | grep -v '^Documentation/'
# invisible characters in added lines
git diff <base>..HEAD | grep '^+' | grep -P
'[\x{200B}-\x{200D}\x{FEFF}\x{2060}\x{00A0}]'
```
The last one finds exactly one line, in `ReleaseNotes/NuttX-12.0.0.md`.
The
non-breaking space in it is already in master; that line is touched only
to
escape a Markdown bracket when the release notes were renamed to `.md`.
Testing logs before change:
```
19 top-level entries; guides/index.rst opens with a .. todo::
"Architecture APIs" lists 18 pages, 2 of them architecture APIs
35 .. todo:: directives published on the site
```
Testing logs after change:
```
$ cd Documentation && make html
build succeeded.
2529 pages, 0 warnings under -W, 0 documents outside a toctree
$ python3 validate_reorg.py <base>
-- .rst PAGES IN HEAD (1573) --
never touched 782
changed by the move only 447
moved, byte for byte identical 291
new or rewritten text (the 50) 50
halves of one split page 2
new, no text 1
TOTAL 1573
APPROVED
$ # redirects
520 rules, 0 broken
$ ./tools/checkpatch.sh -g <base>...HEAD
(exit 0)
```
**An independent audit.** Before opening this PR, the branch was audited by
someone who was given the repo and the script, and asked to check the list
instead of trusting it. The audit pulled **1043 factual claims** out of the
50
pages — paths, symbols, Kconfig names, counts, and every label and arrow in
the
SVG diagrams — and wrote one shell command per claim to prove or disprove it
against the tree. Result: **991 confirmed, 40 refuted, 9 not checkable, 3
ambiguous**.
Everything that was really wrong is fixed in this branch. For example:
* Two pages said 59 files under `arch/` call `netdev_register()`. It is 64.
* One page said `g_inactivetasks` is the only task list that is not
prioritised. Three lists have `attr = 0`, and a fourth state has no list.
* `ReleaseNotes/index` said every release has a tag. Tagging starts at
`nuttx-4.14`; the 46 releases from 0.1.0 to 0.4.13 have none.
* Three labels in `policies.svg` used `ss_*`; the struct fields are
`sched_ss_*`.
* Some Kconfig symbols named on the pages do not exist any more:
`CONFIG_SYSLOG_RAMLOG` (the words are swapped, it is
`CONFIG_RAMLOG_SYSLOG`),
`CONFIG_RAMLOG_CONSOLE` (removed in `dcaaf2d912`),
`CONFIG_SYSLOG_SERIAL_CONSOLE` (removed in `553f12b4e8`),
`CONFIG_RAMLOG_NPOLLWAITERS` (removed in `046dd38c55`).
The audit also found one page with text that the list did not declare. It is
declared now — that is why the number is 50 and not 49 — and the script no
longer takes the list on trust.
**On binaries:** no binary file content changed. 36 were moved and nothing
else. 7 are new, and all 7 are the SVG diagrams drawn for this change —
`build_modes`, `interrupt_flow`, `memory_models`, `net_stack`, `policies`,
`task_states` and `system_map`. They are hand-written XML: no editor
metadata,
no scripts, no `data:` URIs, and no external host except the W3C SVG
namespace.
**Merge state.** The branch is merged with `master` at `53ac762e79`. The
merge
brought 23 new doc pages. 20 stayed where they were added. Three moved to
follow their code:
| added as | now in | why |
|---|---|---|
| `components/drivers/character/aie.rst` | `os/drivers/character/` | the
page says `drivers/aie` is an upper-half character driver |
| `components/drivers/special/sensors/tc74.rst` |
`os/drivers/special/sensors/` | it is a thermal sensor, and it moves with the
other sensor pages |
| `implementation/chroot.rst` | `os/filesystem/` | the code is
`fs/vfs/fs_chroot.c` and the symbol is `FS_CHROOT` in `fs/Kconfig` |
Five `:doc:` links in the newly merged pages pointed at the old layout and
failed the `-W` build. All five targets exist, only the paths had moved.
**What comes after this PR.** Two more are planned, kept separate on purpose
so each can be judged on its own:
*Patch 2 — code fixes.* Reading each page against the code found things that
are wrong in the source, not in the docs. They touch `.c` and `.h`, so they
do
not belong here:
* `include/nuttx/mutex.h` says `nxmutex_unlock()` and `nxrmutex_unlock()` may
be called from an interrupt handler. `sem_post.c` has
`DEBUGASSERT(!up_interrupt_context())`.
* `include/nuttx/board.h:557` describes `LED_STACKCREATED` as "Idle stack
created". The idle stack is the one case that does not set it.
* `drivers/misc/Kconfig:30` says `DEV_ASCII` gives 0x21-0x7f. The code gives
0x20-0x7e and turns the space into a newline.
* `CONFIG_FS_PROCFS_EXCLUDE_CLK` is honoured in seven places and declared in
no
Kconfig, unlike the other twenty-one `FS_PROCFS_EXCLUDE_*` options.
* `sched_setscheduler.c` accepts `SCHED_OTHER` with no guard, but compiles
its
`case` only under `CONFIG_RR_INTERVAL > 0`, and the switch has no
`default`.
With round robin off, asking for `SCHED_OTHER` on a sporadic thread returns
`OK`, sets the priority, and leaves the thread sporadic.
* Five comments name task lists that do not exist any more
(`g_waitingforsemaphore` and the two message-queue ones). That stale
comment
is where the documentation error came from, which is why it is worth fixing
at the source.
*Patch 3 — a CI check for board documentation.* A script that fails the build
when a board under `boards/` has no page under `Documentation/platforms/`, or
when a board page has tags the tag index cannot use. It is what stops the
structure from drifting again. It was written together with this work but it
is a new merge requirement for anyone adding a board, which is a policy
question and not a documentation one, so it goes on its own where it can be
discussed properly.
## PR verification Self-Check
* [x] This PR introduces only one functional change.
* [x] I have updated all required description fields above.
* [x] My PR adheres to Contributing Guidelines and Documentation.
* [ ] My PR is still work in progress (not ready for review).
* [x] My PR is ready for review and can be safely merged into a codebase.
---
Signed-off-by: Vinicius May <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
