Hi Peter/Jonathan, Em Fri, 15 Aug 2025 10:11:09 +0100 Jonathan Cameron <jonathan.came...@huawei.com> escreveu:
> On Thu, 14 Aug 2025 18:13:15 +0100 > Peter Maydell <peter.mayd...@linaro.org> wrote: > > > Earlier this year, the Linux kernel's kernel-doc script was rewritten > > from the old Perl version into a shiny and hopefully more maintainable > > Python version. This commit series updates our copy of this script > > to the latest kernel version. I have tested it by comparing the > > generated HTML documentation and checking that there are no > > unexpected changes. Nice! Yeah, I had a branch here doing something similar for QEMU, but got sidetracked by other things and didn't have time to address a couple of issues. I'm glad you find the time for it. > > Luckily we are carrying very few local modifications to the Perl > > script, so this is fairly straightforward. The structure of the > > patchset is: > > * a minor update to the kerneldoc.py Sphinx extension so it > > will work with both old and new kernel-doc script output > > * a fix to a doc comment markup error that I noticed while comparing > > the HTML output from the two versions of the script > > * import the new Python script, unmodified from the kernel's version > > (conveniently the kernel calls it kernel-doc.py, so it doesn't > > clash with the existing script) > > * make the changes to that library code that correspond to the > > two local QEMU-specific changes we carry To make it easier to maintain and keep in sync with Kernel upstream, perhaps we can try to change Kernel upstream to make easier for QEMU to have a class override for the kdoc parser, allowing it to just sync with Linux upstream, while having its own set of rules on a separate file. A RFC on that sense is welcomed. Otherwise, I'll try to spare some time to think on a good way for doing that. > > * tell sphinx to use the Python version > > * delete the Perl script (I have put a diff of our local mods > > to the Perl script in the commit message of this commit, for > > posterity) > > > > The diffstat looks big, but almost all of it is "import the > > kernel's new script that we trust and don't need to review in > > detail" and "delete the old script". One thing that should be noticed is that Jonathan Corbet is currently doing several cleanups at the Python script, simplifying some regular expressions, avoiding them when str.replace() does the job and adding comments. The end goal is to make it easier for developers to understand and help maintaining its code. So, it is probably worth backporting Linux upstream changes after the end of Kernel 6.17 cycle. > > Given Mauro is somewhat active in qemu as well, +CC for information > if nothing else. > > Jonathan > > > > > > > My immediate motivation for doing this update is that I noticed > > that the submitter of https://gitlab.com/qemu-project/qemu/-/issues/3077 > > is using a Perl that complains about a construct in the perl script, > > which prompted me to check if the kernel folks had already fixed > > it, which it turned out that they had, by rewriting the whole thing :-) > > More generally, if we don't do this update, then we're effectively > > going to drift down the same path we did with checkpatch.pl, where > > we have our own version that diverges from the kernel's version > > and we have to maintain it ourselves. > > > > We should also update the Sphinx plugin itself (i.e. > > docs/sphinx/kerneldoc.py), but because I did not need to do > > that to update the main kernel-doc script, I have left that as > > a separate todo item. The Kernel Sphinx plugin after the change is IMHO (*) a lot cleaner than before, and hendles better kernel-doc warnings, as they are now using Sphinx logger class. (*) I'm a little bit suspicious when talking about it, as I did the changes there too ;-) - Btw, one important point to notice: if you picked the latest version of kernel-doc, it currently requires at least Python 3.6 (3.7 is the recommended minimal one). It does check that, silently bailing out if Python < 3.6. With Python 3.6, it emits a warning, as the parameter order for structs and functions won't match the original order, as the script assumes 3.7+ dict behavior where the insert order is preserved. So, at QEMU build instructions, I would add a notice asking for at least 3.7 to build docs. > > > > Testing > > ------- > > > > I looked at the HTML output of the old kernel-doc script versus the > > new one, using the following diff command which mechanically excludes > > a couple of "same minor change" everywhere diffs, and eyeballing the > > resulting ~150 lines of diff. > > > > diff -w -I '^<div class="kernelindent docutils container">$' -I '^</div>$' > > -I '^<p><strong>Definition</strong>' -r -u -x searchindex.js > > build/x86/docs-old-kerneldoc/manual build/x86/docs/manual > > > > The HTML changes are: > > > > (1) some paras now have ID tags, eg: > > -<p><strong>Functions operating on arrays of bits</strong></p> > > +<p id="functions-operating-on-arrays-of-bits"><strong>Functions operating > > on arrays of bits</strong></p> > > > > (2) Some extra named <div>s, eg: > > +<div class="kernelindent docutils container"> > > <p><strong>Parameters</strong></p> > > <dl class="simple"> > > <dt><code class="docutils literal notranslate"><span > > class="pre">long</span> <span class="pre">nr</span></code></dt><dd><p>the > > bit to set</p> > > @@ -144,12 +145,14 @@ > > <dt><code class="docutils literal notranslate"><span > > class="pre">unsigned</span> <span class="pre">long</span> <span > > class="pre">*addr</span></code></dt><dd><p>the address to start counting > > from</p> > > </dd> > > </dl> > > +</div> > > > > (3) The new version correctly parses the multi-line Return: block for > > the memory_translate_iotlb() doc comment. You can see that the > > old HTML here had dt/dd markup, and it mis-renders in the HTML at > > https://www.qemu.org/docs/master/devel/memory.html#c.memory_translate_iotlb > > > > <p><strong>Return</strong></p> > > -<dl class="simple"> > > -<dt>On success, return the MemoryRegion containing the > > <strong>iotlb</strong> translated</dt><dd><p>addr. The MemoryRegion must > > not be > > accessed after rcu_read_unlock. > > +<p>On success, return the MemoryRegion containing the > > <strong>iotlb</strong> translated > > +addr. The MemoryRegion must not be accessed after rcu_read_unlock. > > On failure, return NULL, setting <strong>errp</strong> with error.</p> > > -</dd> > > -</dl> > > +</div> > > > > "Definition" sections now get output with a trailing colon: > > > > -<p><strong>Definition</strong></p> > > +<div class="kernelindent docutils container"> > > +<p><strong>Definition</strong>:</p> > > > > This seems like it might be a bug in kernel-doc since the Parameters, > > Return, etc sections don't get the trailing colon. I don't think it's > > important enough to worry about. > > > > thanks > > -- PMM > > > > Peter Maydell (8): > > docs/sphinx/kerneldoc.py: Handle new LINENO syntax > > tests/qtest/libqtest.h: Remove stray space from doc comment > > scripts: Import Python kerneldoc from Linux kernel > > scripts/kernel-doc: strip QEMU_ from function definitions > > scripts/kernel-doc: tweak for QEMU coding standards > > scripts/kerneldoc: Switch to the Python kernel-doc script > > scripts/kernel-doc: Delete the old Perl kernel-doc script > > MAINTAINERS: Put kernel-doc under the "docs build machinery" section I'll review the actual patches later. > > > > MAINTAINERS | 2 + > > docs/conf.py | 4 +- > > docs/sphinx/kerneldoc.py | 7 +- > > tests/qtest/libqtest.h | 2 +- > > .editorconfig | 2 +- > > scripts/kernel-doc | 2442 ------------------------------- > > scripts/kernel-doc.py | 325 ++++ > > scripts/lib/kdoc/kdoc_files.py | 291 ++++ > > scripts/lib/kdoc/kdoc_item.py | 42 + > > scripts/lib/kdoc/kdoc_output.py | 749 ++++++++++ > > scripts/lib/kdoc/kdoc_parser.py | 1670 +++++++++++++++++++++ > > scripts/lib/kdoc/kdoc_re.py | 270 ++++ > > 12 files changed, 3355 insertions(+), 2451 deletions(-) > > delete mode 100755 scripts/kernel-doc > > create mode 100755 scripts/kernel-doc.py > > create mode 100644 scripts/lib/kdoc/kdoc_files.py > > create mode 100644 scripts/lib/kdoc/kdoc_item.py > > create mode 100644 scripts/lib/kdoc/kdoc_output.py > > create mode 100644 scripts/lib/kdoc/kdoc_parser.py > > create mode 100644 scripts/lib/kdoc/kdoc_re.py > > > Thanks, Mauro