Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
FYI Am 07.06.2016 um 09:54 schrieb Daniel Vetter : > On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser > wrote: >> From: "Heiser, Markus" >> > I'm still not sold on the vintage-kerneldoc idea. At least in my > experience (after first converting to asciidoc and now to sphinx) this > is a non-issue. Yes, there's the oddball misrendering, but that's no > worse than the oddball typo. Since *vintage* and reST mode is an option of the ".. kernel-doc:" directive it is compareabel. E.g 80211.html produce markup errors: * 230 in "reST" mode * 6 in *vintage* "kernel-doc" mode 95% of the errors caused by: Emphasis “*”: like *emphasis* or **emphasis strong** Leading “_” : is a anchor in reST markup (_foo). Trailing “_: is a reference in reST markup (foo_). interpreted text: “`” inline literals: “``” substitution references: “|” which qouted when the parser runs in vintage "kernel-doc" mode. --Markus-- -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
On Fri, 10 Jun 2016, Markus Heiser wrote: > Am 08.06.2016 um 21:49 schrieb Jonathan Corbet : >> But please, >> at this point, let's build on Jani's work and go from there. Things have >> waited for long enough while we've gone around on this; I think what we >> have is a good starting point. > > I'am willing to contribute, but take my POV: I have finished all > including migration of **all** DocBook to reST, so why should I > throw it all away? Nobody is asking you to throw it away; we're asking you to rebase that work, with some modifications, on top of docs-next, which now has my work merged. > are your friends. You will see that all requirements to get > productive are well done. Within the next days I will > add more features, which has been requested on the ML. OTOH, if you do keep working on a baseline that no longer applies to docs-next, you will be wasting your efforts. > nevertheless which kind of implementation is used, the parsers are all > exchangeable. There is only the user interface which has to be stable > and this is the ".. kernel-doc:" directive. > > I implemented a python version of the kernel-doc parser with an (python) > API, so why should we fiddle with perl and pipes when implementing > a ".. kernel-doc:" directive? No matter how hacky the perl script seems, it has been used on the kernel sources for nearly two decades. Sure, it has accumulated more than a little cruft along the way, but also a lot of corner cases and gotchas have been ironed out. I wouldn't dream of replacing that until we've migrated a bunch of documents using it, and can confirm the replacement produces the same output. Also, there may still be users for the perl script outside of the Sphinx pipeline. I'd like to make sure we don't end up having to maintain *two* homebrew scripted C parsers before adding a new one. >> Along these lines, I don't currently have a strong opinion on the >> big-files vs. little-files question. I *do*, however, like the idea of >> trying to create one coherent kernel document rather than perpetuation our >> current collection of independent book silos. Initially it will certainly >> look like the LDP-based books that people used to duct-tape together back >> in the 90's, but it should improve over time. > > Placing all DocBooks in one huge sphinx-project does not scales well > and brings to additional dependencies. To solve this problem > I use intersphinx and a index-page where all books are referred. On the modifications, I'll repeat that I strongly object to maintaining that arbitrary distinction between "books" and other files. We had that with DocBook, and we'd be more than happy to get rid of it. I object to adding a separate Sphinx project and duplicating a configuration file per document. I object to splitting the documents too much into small bits and pieces, for the reasons I explained in my earlier email [1]. If someone wants to have separate projects with dedicated configuration for special needs, they can add them. But I maintain that most documents do not have that need, and everything is simpler without. [1] http://mid.gmane.org/87wpm15q7q@intel.com > Read chapter "Getting started with reST" from [1]. > [1] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO How about: 1) Write a file in rst, save it somewhere under Documentation. 2) Reference the file in Documentation/index.rst. 3) Be happy. I'll hand it to you that you have more experience in Sphinx than I do, but I like to think I know my fellow kernel developers better. The above is what they want to hear. They don't want a single hurdle more. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Am 08.06.2016 um 21:49 schrieb Jonathan Corbet : > So I've finally gotten a chance to make another pass over this stuff. > > Markus, your enthusiasm is great; I'm hoping you'll do great things > helping us to improve the kernel's documentation toolchain. With 7 years DocBook and 8 years reST experience this is my opportunity to give linux something back ;-) > But please, > at this point, let's build on Jani's work and go from there. Things have > waited for long enough while we've gone around on this; I think what we > have is a good starting point. I'am willing to contribute, but take my POV: I have finished all including migration of **all** DocBook to reST, so why should I throw it all away? Pull it from: https://github.com/return42/linux.git linux-doc-reST The kernel-doc HOWTO [1], the Template Book [2] and make books-help are your friends. You will see that all requirements to get productive are well done. Within the next days I will add more features, which has been requested on the ML. [1] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO [2] http://return42.github.io/sphkerneldoc/books/template-book > > On the specifics, Daniel already covered most of it pretty well. > > On Tue, 7 Jun 2016 09:54:21 +0200 > Daniel Vetter wrote: > >> I think next steps would be: >> - rebase flat-table onto Jani's work and relicense under gplv2 > > This I would really like to see. is already done. > >> - look into rewriting kernel-doc in python more as a long-term project > > There is nobody who would like to dump the Perl kernel-doc more than I > would; it wasn't pretty to begin with and hasn't improved over the years. > I, too, had thought about redoing it, but I, too, concluded that it wasn't > the highest of priorities. > > Please do keep this around, we may want it before too long. I have some > sympathy for Daniel's suggestion to look into using LLVM; we could also > maybe stay a little closer to our roots and use the sparse library. But > there might also be value in a Python version that doesn't add more > dependencies to the docs toolchain. We need to think about this, but I > don't think we need to answer it now. nevertheless which kind of implementation is used, the parsers are all exchangeable. There is only the user interface which has to be stable and this is the ".. kernel-doc:" directive. I implemented a python version of the kernel-doc parser with an (python) API, so why should we fiddle with perl and pipes when implementing a ".. kernel-doc:" directive? > >> - start converting docs instead - I really want to start reaping >> benefits of all this work as soon as possible. > > Absolutely. pull above, there are all converted DocBooks are in. > > Along these lines, I don't currently have a strong opinion on the > big-files vs. little-files question. I *do*, however, like the idea of > trying to create one coherent kernel document rather than perpetuation our > current collection of independent book silos. Initially it will certainly > look like the LDP-based books that people used to duct-tape together back > in the 90's, but it should improve over time. Placing all DocBooks in one huge sphinx-project does not scales well and brings to additional dependencies. To solve this problem I use intersphinx and a index-page where all books are referred. Read chapter "Getting started with reST" from [1]. --Markus-- > > Thanks, > > jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
So I've finally gotten a chance to make another pass over this stuff. Markus, your enthusiasm is great; I'm hoping you'll do great things helping us to improve the kernel's documentation toolchain. But please, at this point, let's build on Jani's work and go from there. Things have waited for long enough while we've gone around on this; I think what we have is a good starting point. On the specifics, Daniel already covered most of it pretty well. On Tue, 7 Jun 2016 09:54:21 +0200 Daniel Vetter wrote: > I think next steps would be: > - rebase flat-table onto Jani's work and relicense under gplv2 This I would really like to see. > - look into rewriting kernel-doc in python more as a long-term project There is nobody who would like to dump the Perl kernel-doc more than I would; it wasn't pretty to begin with and hasn't improved over the years. I, too, had thought about redoing it, but I, too, concluded that it wasn't the highest of priorities. Please do keep this around, we may want it before too long. I have some sympathy for Daniel's suggestion to look into using LLVM; we could also maybe stay a little closer to our roots and use the sparse library. But there might also be value in a Python version that doesn't add more dependencies to the docs toolchain. We need to think about this, but I don't think we need to answer it now. > - start converting docs instead - I really want to start reaping > benefits of all this work as soon as possible. Absolutely. Along these lines, I don't currently have a strong opinion on the big-files vs. little-files question. I *do*, however, like the idea of trying to create one coherent kernel document rather than perpetuation our current collection of independent book silos. Initially it will certainly look like the LDP-based books that people used to duct-tape together back in the 90's, but it should improve over time. Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Am 07.06.2016 um 13:09 schrieb Jani Nikula : > On Tue, 07 Jun 2016, Markus Heiser wrote: >> Am 07.06.2016 um 10:59 schrieb Jani Nikula : >>> One of the key arguments against too much splitting that hasn't been >>> mentioned is that despite all the fine output Sphinx can produce, we >>> have plenty of people who couldn't care less about running Sphinx to get >>> readable documentation. They will grep and read the plain text files >>> directly, and that's a large part of the appeal of any lightweight >>> markup. >> >> But they have read XML or compiled DocBook XML? ... Sphinx brings a >> search engine with its html. >> >>> When you split up a file into snippets that no longer tell a coherent >>> story independently, you've failed. >> >> Chapters are breaking stories? >> >>> For the .txt files under Documentation, we mostly do not want to split >>> them up any more if and when they're converted to rst. For the .tmpl >>> files under Documentation/DocBook, each rst file split off from there >>> should still be a sensible document on its own, with the filename >>> telling what it's about. This will be the main benefit of this whole >>> exercise for the people who do not care about Sphinx - instead of >>> reading (read: ignoring) DocBook XML, they can now read the rst files. >> >> Sorry but in IMO this suggestion is backward, if someone don't be able >> to build HTML documents he should at least be able to use the >> internet [1] :-o >> >> [1] http://return42.github.io/sphkerneldoc/articles/books.html > > I don't think you understand the target audience very well. May be, help me .. I can't see the use case ... is emac's incremental search your use case? ... possibly we are talking past each other * chunking is my recommendation on books, e.g. chapters is a good point to split a book .. beside: books which make use of the kernel-doc directive are not *well grep-able* in their reST format .. * not every .txt file should be chunked ... and folders with the 00-INDEX are already chunked. The only thing I want to say is, that I recommend chunking, but at the end it should be an author decision, not a handicap of the build system. -- Markus -- > > BR, > Jani. > > > -- > Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Hi all,
sorry I forgot to mentioning that I have not yet add the Makefile.reST
to the root Makefile. To test, please give the srctree environment in the
command line. E.g:
cd /share/linux/Documentation
srctree=/share/linux make -f Makefile.reST books/kernel-doc-HOWTO.html
-- Markus --
Am 06.06.2016 um 18:32 schrieb Markus Heiser :
> From: "Heiser, Markus"
>
> Hi Jonathan, Jani, all -
>
> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>
>git://github.com/return42/linux.git linux-doc-reST
>
> The specification of the implementation is the (new) kernel-doc-HOWTO book
> which
> is a part of the patch series (also online avaliable at [2]).
>
> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
> taking reST extension into account. It serves several purposes:
>
> * user guide for kernel developers
> * specification of the kernel-doc parser
> * test case
>
> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO,
> the
> kernel-doc perl script is not needed for reST builds. The python variant of
> the
> kernel-doc parser supports the markup modes:
>
> * kernel-doc (vintage)
> * reST
>
> This series also includes the reST directives describe in the
> kernel-doc-HOWTO:
>
> * kernel-doc (uses/imports the parser from kernel_doc.py)
> * flat-table (table notation with meaningfull diffs)
>
> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
> this folder a minimal python infrastructure is provided.
>
> The basic sphinx-build infrastructure consists of:
>
> * Documentation/Makefile.reST & Documentation/conf.py
>
> Makefile and basic 'sphinx config' file to build the various reST
> documents and output formats. Provides the basic sphinx-doc build
> infrastructure including the *sphinx-subprojects* feature. With this
> feature each book can be build and distributed stand-alone.
>
> * Documentation/books
>
> In this folder, the books with reST markup are placed. To
> provide *sphinx-subprojects*, each book has its one folder and
> a (optional) Documentation/books/{book-name}.conf file
> which *overwrites* the basic configuration from Documentation/conf.py
>
> Any comments are welcome, but please read [2] first.
>
> With regards
>
> -- Markus --
>
> [1] https://return42.github.io/sphkerneldoc
> [2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
>
>
>
> Heiser, Markus (7):
> python: add scripts/site-packages
> sphinx-doc: add basic sphinx-build infrastructure
> kernel-doc-HOWTO: add kernel-doc specification
> linuxdoc: add python package linuxdoc
> kernel-doc parser: inital python implementation
> kernel-doc directive: initial implementation
> flat-table directive: initial implementation
>
> Documentation/.gitignore |2 +
> Documentation/Makefile.reST| 111 +
> Documentation/books/.gitignore |0
> Documentation/books/kernel-doc-HOWTO.conf | 200 ++
> .../books/kernel-doc-HOWTO/all-in-a-tumble-src.rst | 12 +
> .../books/kernel-doc-HOWTO/all-in-a-tumble.h | 271 ++
> .../books/kernel-doc-HOWTO/all-in-a-tumble.rst | 11 +
> Documentation/books/kernel-doc-HOWTO/csv_table.txt |6 +
> Documentation/books/kernel-doc-HOWTO/index.rst | 46 +
> .../kernel-doc-HOWTO/kernel-doc-components.rst | 35 +
> .../kernel-doc-HOWTO/kernel-doc-directive.rst | 199 ++
> .../books/kernel-doc-HOWTO/kernel-doc-examples.rst | 16 +
> .../books/kernel-doc-HOWTO/kernel-doc-intro.rst| 85 +
> .../books/kernel-doc-HOWTO/kernel-doc-syntax.rst | 171 ++
> .../kernel-doc-HOWTO/reST-kernel-doc-mode.rst | 120 +
> Documentation/books/kernel-doc-HOWTO/refs.txt | 15 +
> .../books/kernel-doc-HOWTO/table-markup.rst| 499
> .../books/kernel-doc-HOWTO/test/parser_test.h | 332 +++
> .../kernel-doc-HOWTO/vintage-kernel-doc-mode.rst | 100 +
> Documentation/conf.py | 357 +++
> Documentation/sphinx-static/theme_overrides.css| 45 +
> Documentation/sphinx-tex/Makefile | 88 +
> scripts/site-python/.gitignore |1 +
> scripts/site-python/README | 14 +
> scripts/site-python/linuxdoc/__init__.py |8 +
> scripts/site-python/linuxdoc/kernel_doc.py | 2764
> scripts/site-python/linuxdoc/rstFlatTable.py | 356 +++
> scripts/site-python/linuxdoc/rstKernelDoc.py | 369 +++
> 28 files changed, 6233 insertions(+)
> create mode 100644 Documentation/.gitignore
> create mode 100644 Documentation/Makefile.reST
> create mode 100644 Documentation/books/.gitignore
> create mode 100644 Documentation/books/kernel-doc-HOWTO.conf
> create mode 100644
> Documentation/books/kernel-doc-HOWTO/all-in-a-tumble-src.rst
> create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble.h
> create mode 100644 Documentat
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
On Tue, 07 Jun 2016, Markus Heiser wrote: > Am 07.06.2016 um 10:59 schrieb Jani Nikula : >> One of the key arguments against too much splitting that hasn't been >> mentioned is that despite all the fine output Sphinx can produce, we >> have plenty of people who couldn't care less about running Sphinx to get >> readable documentation. They will grep and read the plain text files >> directly, and that's a large part of the appeal of any lightweight >> markup. > > But they have read XML or compiled DocBook XML? ... Sphinx brings a > search engine with its html. > >> When you split up a file into snippets that no longer tell a coherent >> story independently, you've failed. > > Chapters are breaking stories? > >> For the .txt files under Documentation, we mostly do not want to split >> them up any more if and when they're converted to rst. For the .tmpl >> files under Documentation/DocBook, each rst file split off from there >> should still be a sensible document on its own, with the filename >> telling what it's about. This will be the main benefit of this whole >> exercise for the people who do not care about Sphinx - instead of >> reading (read: ignoring) DocBook XML, they can now read the rst files. > > Sorry but in IMO this suggestion is backward, if someone don't be able > to build HTML documents he should at least be able to use the > internet [1] :-o > > [1] http://return42.github.io/sphkerneldoc/articles/books.html I don't think you understand the target audience very well. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Am 07.06.2016 um 10:59 schrieb Jani Nikula : > On Tue, 07 Jun 2016, Daniel Vetter wrote: >> I think getting the flat-table directive landed would be great. We >> have a similarly annoying table in gpu.tmpl where the ascii-art tables >> fall short and are annoying. We want to split it up, but that's not a >> short-term solution suitable for conversion. Atm it's licensed under >> gplv3, which is incompatible with the kernel, so need to fix that. > > Agreed. see last mail >> I'm still not sold on the vintage-kerneldoc idea. At least in my >> experience (after first converting to asciidoc and now to sphinx) this >> is a non-issue. Yes, there's the oddball misrendering, but that's no >> worse than the oddball typo. And a really good way for >> drive-by-contributors and newbies to send in a few useful patches. >> Personally I'm totally happy to have that bit of ugliness, same way I >> don't reject patches just because they trip over checkpatch.pl in some >> minor detail. The downside otoh means we'll have to forever maintain 2 >> versions of kernel-doc, and I don't want that. Jani&me have already >> ripped out some of the more bonghits features of kernel-doc (like >> anything with a \w: is a heading and other nonsense like that). The >> only thing we really need to keep as special is the overall kerneldoc >> layout, and the special referencing using &, %, @ and friends. > > Agreed. see last mail >> Reimplementing kernel-doc in python is also something Jani&I >> discussed. But I think it only makes sense if we go ahead and also >> modernize the parser, i.e. use something like python-clang to parse C, >> and build a modern parser-combinators thing for the kernel-doc itself. >> The regex state machinery is a horror show, whether it's written in >> cargo-culted perl or python. > > I think the only sensible way to change the parser is to get everything > working using the perl version first, and then you can diff the results > against the python version. That's about the only validation of the new > parser that actually matters in the real world. > > At this time, I'd put this way down in the list of priorities. I tested the python version of the parser by running it over nearly the whole kernel-tree .. yes this is not a side by side test, but I also tested it by partial side by side comparing the output of [1] [1] https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc I spend over a week in testing ... IMO in the meantime the quality of the python parser is a bit over the perl one. But at the end, I have never seen a "error free implementation". >> I agree that we need to update the kernel-doc howto (and Jani's >> working on that already too), but I think there's no need to split up >> that finely. Maybe that's a bit against sphinx best practices, but >> right now we have rather large documents (both plain text and >> docbook). Splitting definitely makes sense in some cases (e.g. >> gpu.tmpl is only this large because we couldn't do cross-template >> referencing), but by-and-large I think the current splitting is mostly >> reasonable. > > One of the key arguments against too much splitting that hasn't been > mentioned is that despite all the fine output Sphinx can produce, we > have plenty of people who couldn't care less about running Sphinx to get > readable documentation. They will grep and read the plain text files > directly, and that's a large part of the appeal of any lightweight > markup. But they have read XML or compiled DocBook XML? ... Sphinx brings a search engine with its html. > When you split up a file into snippets that no longer tell a coherent > story independently, you've failed. Chapters are breaking stories? > For the .txt files under Documentation, we mostly do not want to split > them up any more if and when they're converted to rst. For the .tmpl > files under Documentation/DocBook, each rst file split off from there > should still be a sensible document on its own, with the filename > telling what it's about. This will be the main benefit of this whole > exercise for the people who do not care about Sphinx - instead of > reading (read: ignoring) DocBook XML, they can now read the rst files. Sorry but in IMO this suggestion is backward, if someone don't be able to build HTML documents he should at least be able to use the internet [1] :-o [1] http://return42.github.io/sphkerneldoc/articles/books.html > > BR, > Jani. > > -- > Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Am 07.06.2016 um 09:54 schrieb Daniel Vetter :
> On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser
> wrote:
>> From: "Heiser, Markus"
>>
>> Hi Jonathan, Jani, all -
>>
>> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>>
>>git://github.com/return42/linux.git linux-doc-reST
>>
>> The specification of the implementation is the (new) kernel-doc-HOWTO book
>> which
>> is a part of the patch series (also online avaliable at [2]).
>>
>> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
>> taking reST extension into account. It serves several purposes:
>>
>> * user guide for kernel developers
>> * specification of the kernel-doc parser
>> * test case
>>
>> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO,
>> the
>> kernel-doc perl script is not needed for reST builds. The python variant of
>> the
>> kernel-doc parser supports the markup modes:
>>
>> * kernel-doc (vintage)
>> * reST
>>
>> This series also includes the reST directives describe in the
>> kernel-doc-HOWTO:
>>
>> * kernel-doc (uses/imports the parser from kernel_doc.py)
>> * flat-table (table notation with meaningfull diffs)
>>
>> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
>> this folder a minimal python infrastructure is provided.
>>
>> The basic sphinx-build infrastructure consists of:
>>
>> * Documentation/Makefile.reST & Documentation/conf.py
>>
>> Makefile and basic 'sphinx config' file to build the various reST
>> documents and output formats. Provides the basic sphinx-doc build
>> infrastructure including the *sphinx-subprojects* feature. With this
>> feature each book can be build and distributed stand-alone.
>>
>> * Documentation/books
>>
>> In this folder, the books with reST markup are placed. To
>> provide *sphinx-subprojects*, each book has its one folder and
>> a (optional) Documentation/books/{book-name}.conf file
>> which *overwrites* the basic configuration from Documentation/conf.py
>>
>> Any comments are welcome, but please read [2] first.
>
> I think getting the flat-table directive landed would be great. We
> have a similarly annoying table in gpu.tmpl where the ascii-art tables
> fall short and are annoying. We want to split it up, but that's not a
> short-term solution suitable for conversion. Atm it's licensed under
> gplv3, which is incompatible with the kernel, so need to fix that.
No problem, changed to gplv2
https://github.com/return42/linux/commit/97ee23e74384200fe79fe4557954a897725949e3
is this OK?
> I'm still not sold on the vintage-kerneldoc idea. At least in my
> experience (after first converting to asciidoc and now to sphinx) this
> is a non-issue. Yes, there's the oddball misrendering, but that's no
> worse than the oddball typo. And a really good way for
> drive-by-contributors and newbies to send in a few useful patches.
> Personally I'm totally happy to have that bit of ugliness, same way I
> don't reject patches just because they trip over checkpatch.pl in some
> minor detail. The downside otoh means we'll have to forever maintain 2
> versions of kernel-doc, and I don't want that.
This can be stopped by a reST lint process similar to what Jani mentioned.
> Jani&me have already
> ripped out some of the more bonghits features of kernel-doc (like
> anything with a \w: is a heading and other nonsense like that).
Yes, I removed it in the reST mode of the parser. This is also
a reason to differentiate between *vintage* and reST or you will
break tons of kernel-doc comments, which are valid in the term of
the *vintage* style.
> The only thing we really need to keep as special is the overall kerneldoc
> layout, and the special referencing using &, %, @ and friends.
Ok, I will try to add this highlighting feature to the reST mode.
I left it out because I feared that there are any collision with
reST markup, but with a closer look the fear might be unfounded.
> Reimplementing kernel-doc in python is also something Jani&I
> discussed. But I think it only makes sense if we go ahead and also
> modernize the parser, i.e. use something like python-clang to parse C,
> and build a modern parser-combinators thing for the kernel-doc itself.
> The regex state machinery is a horror show, whether it's written in
> cargo-culted perl or python.
Yes clang might be better solution and be more flexible on parsing
C, but it brings also much more decencies, but no one has tested
it against the kernel tree (I can't judge it, I have no experience
with) ... the regular expressions are proofed.
But anyway, if someone implements a C parser based on clang we can
use it, but before the python implementation is the best we have ;-)
> I agree that we need to update the kernel-doc howto (and Jani's
> working on that already too), but I think there's no need to split up
> that finely. Maybe that's a bit against sphinx best practices, but
> right now we have rather large documents (both plain
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
On Tue, 07 Jun 2016, Daniel Vetter wrote: > I think getting the flat-table directive landed would be great. We > have a similarly annoying table in gpu.tmpl where the ascii-art tables > fall short and are annoying. We want to split it up, but that's not a > short-term solution suitable for conversion. Atm it's licensed under > gplv3, which is incompatible with the kernel, so need to fix that. Agreed. > I'm still not sold on the vintage-kerneldoc idea. At least in my > experience (after first converting to asciidoc and now to sphinx) this > is a non-issue. Yes, there's the oddball misrendering, but that's no > worse than the oddball typo. And a really good way for > drive-by-contributors and newbies to send in a few useful patches. > Personally I'm totally happy to have that bit of ugliness, same way I > don't reject patches just because they trip over checkpatch.pl in some > minor detail. The downside otoh means we'll have to forever maintain 2 > versions of kernel-doc, and I don't want that. Jani&me have already > ripped out some of the more bonghits features of kernel-doc (like > anything with a \w: is a heading and other nonsense like that). The > only thing we really need to keep as special is the overall kerneldoc > layout, and the special referencing using &, %, @ and friends. Agreed. > Reimplementing kernel-doc in python is also something Jani&I > discussed. But I think it only makes sense if we go ahead and also > modernize the parser, i.e. use something like python-clang to parse C, > and build a modern parser-combinators thing for the kernel-doc itself. > The regex state machinery is a horror show, whether it's written in > cargo-culted perl or python. I think the only sensible way to change the parser is to get everything working using the perl version first, and then you can diff the results against the python version. That's about the only validation of the new parser that actually matters in the real world. At this time, I'd put this way down in the list of priorities. > I agree that we need to update the kernel-doc howto (and Jani's > working on that already too), but I think there's no need to split up > that finely. Maybe that's a bit against sphinx best practices, but > right now we have rather large documents (both plain text and > docbook). Splitting definitely makes sense in some cases (e.g. > gpu.tmpl is only this large because we couldn't do cross-template > referencing), but by-and-large I think the current splitting is mostly > reasonable. One of the key arguments against too much splitting that hasn't been mentioned is that despite all the fine output Sphinx can produce, we have plenty of people who couldn't care less about running Sphinx to get readable documentation. They will grep and read the plain text files directly, and that's a large part of the appeal of any lightweight markup. When you split up a file into snippets that no longer tell a coherent story independently, you've failed. For the .txt files under Documentation, we mostly do not want to split them up any more if and when they're converted to rst. For the .tmpl files under Documentation/DocBook, each rst file split off from there should still be a sensible document on its own, with the filename telling what it's about. This will be the main benefit of this whole exercise for the people who do not care about Sphinx - instead of reading (read: ignoring) DocBook XML, they can now read the rst files. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser wrote:
> From: "Heiser, Markus"
>
> Hi Jonathan, Jani, all -
>
> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>
> git://github.com/return42/linux.git linux-doc-reST
>
> The specification of the implementation is the (new) kernel-doc-HOWTO book
> which
> is a part of the patch series (also online avaliable at [2]).
>
> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
> taking reST extension into account. It serves several purposes:
>
> * user guide for kernel developers
> * specification of the kernel-doc parser
> * test case
>
> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO,
> the
> kernel-doc perl script is not needed for reST builds. The python variant of
> the
> kernel-doc parser supports the markup modes:
>
> * kernel-doc (vintage)
> * reST
>
> This series also includes the reST directives describe in the
> kernel-doc-HOWTO:
>
> * kernel-doc (uses/imports the parser from kernel_doc.py)
> * flat-table (table notation with meaningfull diffs)
>
> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
> this folder a minimal python infrastructure is provided.
>
> The basic sphinx-build infrastructure consists of:
>
> * Documentation/Makefile.reST & Documentation/conf.py
>
> Makefile and basic 'sphinx config' file to build the various reST
> documents and output formats. Provides the basic sphinx-doc build
> infrastructure including the *sphinx-subprojects* feature. With this
> feature each book can be build and distributed stand-alone.
>
> * Documentation/books
>
> In this folder, the books with reST markup are placed. To
> provide *sphinx-subprojects*, each book has its one folder and
> a (optional) Documentation/books/{book-name}.conf file
> which *overwrites* the basic configuration from Documentation/conf.py
>
> Any comments are welcome, but please read [2] first.
I think getting the flat-table directive landed would be great. We
have a similarly annoying table in gpu.tmpl where the ascii-art tables
fall short and are annoying. We want to split it up, but that's not a
short-term solution suitable for conversion. Atm it's licensed under
gplv3, which is incompatible with the kernel, so need to fix that.
I'm still not sold on the vintage-kerneldoc idea. At least in my
experience (after first converting to asciidoc and now to sphinx) this
is a non-issue. Yes, there's the oddball misrendering, but that's no
worse than the oddball typo. And a really good way for
drive-by-contributors and newbies to send in a few useful patches.
Personally I'm totally happy to have that bit of ugliness, same way I
don't reject patches just because they trip over checkpatch.pl in some
minor detail. The downside otoh means we'll have to forever maintain 2
versions of kernel-doc, and I don't want that. Jani&me have already
ripped out some of the more bonghits features of kernel-doc (like
anything with a \w: is a heading and other nonsense like that). The
only thing we really need to keep as special is the overall kerneldoc
layout, and the special referencing using &, %, @ and friends.
Reimplementing kernel-doc in python is also something Jani&I
discussed. But I think it only makes sense if we go ahead and also
modernize the parser, i.e. use something like python-clang to parse C,
and build a modern parser-combinators thing for the kernel-doc itself.
The regex state machinery is a horror show, whether it's written in
cargo-culted perl or python.
I agree that we need to update the kernel-doc howto (and Jani's
working on that already too), but I think there's no need to split up
that finely. Maybe that's a bit against sphinx best practices, but
right now we have rather large documents (both plain text and
docbook). Splitting definitely makes sense in some cases (e.g.
gpu.tmpl is only this large because we couldn't do cross-template
referencing), but by-and-large I think the current splitting is mostly
reasonable. A better demonstration vehicle is imo converting all
existing .tmpl to .rst. And also making sure the new kernel-doc is
still able to parse all existing kernel-doc comments, even when
they're not pulled into .tmpl files. Jani has done that (using
hacked-up versions of the sphinx-lint script) for his tree, making
sure there's no unexpected changes anywhere.
I think next steps would be:
- rebase flat-table onto Jani's work and relicense under gplv2
- look into rewriting kernel-doc in python more as a long-term project
- start converting docs instead - I really want to start reaping
benefits of all this work as soon as possible.
Thanks, Daniel
>
> With regards
>
> -- Markus --
>
> [1] https://return42.github.io/sphkerneldoc
> [2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
>
>
>
> Heiser, Markus (7):
> python: add scripts/site-packages
> sphinx-doc: add basic sphinx-build infrastructure
> kernel-doc-HOWTO: add kernel
[PATCH 0/7] add reST/sphinx-doc to linux documentation
From: "Heiser, Markus"
Hi Jonathan, Jani, all -
I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
git://github.com/return42/linux.git linux-doc-reST
The specification of the implementation is the (new) kernel-doc-HOWTO book which
is a part of the patch series (also online avaliable at [2]).
The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
taking reST extension into account. It serves several purposes:
* user guide for kernel developers
* specification of the kernel-doc parser
* test case
The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO, the
kernel-doc perl script is not needed for reST builds. The python variant of the
kernel-doc parser supports the markup modes:
* kernel-doc (vintage)
* reST
This series also includes the reST directives describe in the kernel-doc-HOWTO:
* kernel-doc (uses/imports the parser from kernel_doc.py)
* flat-table (table notation with meaningfull diffs)
The Python/Sphinx extensions placed in the scripts/site-packages folder. With
this folder a minimal python infrastructure is provided.
The basic sphinx-build infrastructure consists of:
* Documentation/Makefile.reST & Documentation/conf.py
Makefile and basic 'sphinx config' file to build the various reST
documents and output formats. Provides the basic sphinx-doc build
infrastructure including the *sphinx-subprojects* feature. With this
feature each book can be build and distributed stand-alone.
* Documentation/books
In this folder, the books with reST markup are placed. To
provide *sphinx-subprojects*, each book has its one folder and
a (optional) Documentation/books/{book-name}.conf file
which *overwrites* the basic configuration from Documentation/conf.py
Any comments are welcome, but please read [2] first.
With regards
-- Markus --
[1] https://return42.github.io/sphkerneldoc
[2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
Heiser, Markus (7):
python: add scripts/site-packages
sphinx-doc: add basic sphinx-build infrastructure
kernel-doc-HOWTO: add kernel-doc specification
linuxdoc: add python package linuxdoc
kernel-doc parser: inital python implementation
kernel-doc directive: initial implementation
flat-table directive: initial implementation
Documentation/.gitignore |2 +
Documentation/Makefile.reST| 111 +
Documentation/books/.gitignore |0
Documentation/books/kernel-doc-HOWTO.conf | 200 ++
.../books/kernel-doc-HOWTO/all-in-a-tumble-src.rst | 12 +
.../books/kernel-doc-HOWTO/all-in-a-tumble.h | 271 ++
.../books/kernel-doc-HOWTO/all-in-a-tumble.rst | 11 +
Documentation/books/kernel-doc-HOWTO/csv_table.txt |6 +
Documentation/books/kernel-doc-HOWTO/index.rst | 46 +
.../kernel-doc-HOWTO/kernel-doc-components.rst | 35 +
.../kernel-doc-HOWTO/kernel-doc-directive.rst | 199 ++
.../books/kernel-doc-HOWTO/kernel-doc-examples.rst | 16 +
.../books/kernel-doc-HOWTO/kernel-doc-intro.rst| 85 +
.../books/kernel-doc-HOWTO/kernel-doc-syntax.rst | 171 ++
.../kernel-doc-HOWTO/reST-kernel-doc-mode.rst | 120 +
Documentation/books/kernel-doc-HOWTO/refs.txt | 15 +
.../books/kernel-doc-HOWTO/table-markup.rst| 499
.../books/kernel-doc-HOWTO/test/parser_test.h | 332 +++
.../kernel-doc-HOWTO/vintage-kernel-doc-mode.rst | 100 +
Documentation/conf.py | 357 +++
Documentation/sphinx-static/theme_overrides.css| 45 +
Documentation/sphinx-tex/Makefile | 88 +
scripts/site-python/.gitignore |1 +
scripts/site-python/README | 14 +
scripts/site-python/linuxdoc/__init__.py |8 +
scripts/site-python/linuxdoc/kernel_doc.py | 2764
scripts/site-python/linuxdoc/rstFlatTable.py | 356 +++
scripts/site-python/linuxdoc/rstKernelDoc.py | 369 +++
28 files changed, 6233 insertions(+)
create mode 100644 Documentation/.gitignore
create mode 100644 Documentation/Makefile.reST
create mode 100644 Documentation/books/.gitignore
create mode 100644 Documentation/books/kernel-doc-HOWTO.conf
create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble-src.rst
create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble.h
create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble.rst
create mode 100644 Documentation/books/kernel-doc-HOWTO/csv_table.txt
create mode 100644 Documentation/books/kernel-doc-HOWTO/index.rst
create mode 100644
Documentation/books/kernel-doc-HOWTO/kernel-doc-components.rst
create mode 100644
Documentation/books/kernel-doc-HOWTO/kernel-doc-directive.rst
create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-examples.rst
create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-intro.rst
cr
