Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-15 Thread Markus Heiser 
FYI

Am 07.06.2016 um 09:54 schrieb Daniel Vetter :

> On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser  
> wrote:
>> From: "Heiser, Markus" 
>> 
> I'm still not sold on the vintage-kerneldoc idea. At least in my
> experience (after first converting to asciidoc and now to sphinx) this
> is a non-issue. Yes, there's the oddball misrendering, but that's no
> worse than the oddball typo.

Since *vintage* and reST mode is an option of the ".. kernel-doc:" 
directive it is compareabel. E.g 80211.html produce markup errors:

* 230 in "reST" mode
* 6 in *vintage* "kernel-doc" mode

95% of the errors caused by:

Emphasis “*”: like *emphasis* or **emphasis strong**
Leading “_” : is a anchor in reST markup (_foo).
Trailing “_: is a reference in reST markup (foo_).
interpreted text: “`”
inline literals: “``”
substitution references: “|”

which qouted when the parser runs in vintage "kernel-doc" mode.

--Markus--
 

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-10 Thread Markus Heiser 

Am 08.06.2016 um 21:49 schrieb Jonathan Corbet :

> So I've finally gotten a chance to make another pass over this stuff.
> 
> Markus, your enthusiasm is great; I'm hoping you'll do great things
> helping us to improve the kernel's documentation toolchain.  

With 7 years DocBook and 8 years reST experience this is my
opportunity to give linux something back ;-)

> But please,
> at this point, let's build on Jani's work and go from there.  Things have
> waited for long enough while we've gone around on this; I think what we
> have is a good starting point.

I'am willing to contribute, but take my POV: I have finished all
including migration of **all** DocBook to reST, so why should I
throw it all away?

Pull it from:

 https://github.com/return42/linux.git linux-doc-reST 

The kernel-doc HOWTO [1], the Template Book [2] and 

  make books-help 

are your friends. You will see that all requirements to get 
productive are well done. Within the next days I will 
add more features, which has been requested on the ML.

[1] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
[2] http://return42.github.io/sphkerneldoc/books/template-book

> 
> On the specifics, Daniel already covered most of it pretty well.
> 
> On Tue, 7 Jun 2016 09:54:21 +0200
> Daniel Vetter  wrote:
> 
>> I think next steps would be:
>> - rebase flat-table onto Jani's work and relicense under gplv2
> 
> This I would really like to see.

is already done.

> 
>> - look into rewriting kernel-doc in python more as a long-term project
> 
> There is nobody who would like to dump the Perl kernel-doc more than I
> would; it wasn't pretty to begin with and hasn't improved over the years.
> I, too, had thought about redoing it, but I, too, concluded that it wasn't
> the highest of priorities.
> 
> Please do keep this around, we may want it before too long.  I have some
> sympathy for Daniel's suggestion to look into using LLVM; we could also
> maybe stay a little closer to our roots and use the sparse library.  But
> there might also be value in a Python version that doesn't add more
> dependencies to the docs toolchain.  We need to think about this, but I
> don't think we need to answer it now.

nevertheless which kind of implementation is used, the parsers are all
exchangeable. There is only the user interface which has to be stable 
and this is the ".. kernel-doc:" directive.

I implemented a python version of the kernel-doc parser with an (python)
API, so why should we fiddle with perl and pipes when implementing
a ".. kernel-doc:" directive?

> 
>> - start converting docs instead - I really want to start reaping
>> benefits of all this work as soon as possible.
> 
> Absolutely.

pull above, there are all converted DocBooks are in.

> 
> Along these lines, I don't currently have a strong opinion on the
> big-files vs. little-files question.  I *do*, however, like the idea of
> trying to create one coherent kernel document rather than perpetuation our
> current collection of independent book silos.  Initially it will certainly
> look like the LDP-based books that people used to duct-tape together back
> in the 90's, but it should improve over time.

Placing all DocBooks in one huge sphinx-project does not scales well 
and brings to additional dependencies. To solve this problem
I use intersphinx and a index-page where all books are referred.

Read chapter "Getting started with reST" from [1].

--Markus--

> 
> Thanks,
> 
> jon

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-08 Thread Jonathan Corbet 
So I've finally gotten a chance to make another pass over this stuff.

Markus, your enthusiasm is great; I'm hoping you'll do great things
helping us to improve the kernel's documentation toolchain.  But please,
at this point, let's build on Jani's work and go from there.  Things have
waited for long enough while we've gone around on this; I think what we
have is a good starting point.

On the specifics, Daniel already covered most of it pretty well.

On Tue, 7 Jun 2016 09:54:21 +0200
Daniel Vetter  wrote:

> I think next steps would be:
> - rebase flat-table onto Jani's work and relicense under gplv2

This I would really like to see.

> - look into rewriting kernel-doc in python more as a long-term project

There is nobody who would like to dump the Perl kernel-doc more than I
would; it wasn't pretty to begin with and hasn't improved over the years.
I, too, had thought about redoing it, but I, too, concluded that it wasn't
the highest of priorities.

Please do keep this around, we may want it before too long.  I have some
sympathy for Daniel's suggestion to look into using LLVM; we could also
maybe stay a little closer to our roots and use the sparse library.  But
there might also be value in a Python version that doesn't add more
dependencies to the docs toolchain.  We need to think about this, but I
don't think we need to answer it now.

> - start converting docs instead - I really want to start reaping
> benefits of all this work as soon as possible.

Absolutely.

Along these lines, I don't currently have a strong opinion on the
big-files vs. little-files question.  I *do*, however, like the idea of
trying to create one coherent kernel document rather than perpetuation our
current collection of independent book silos.  Initially it will certainly
look like the LDP-based books that people used to duct-tape together back
in the 90's, but it should improve over time.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Markus Heiser 

Am 07.06.2016 um 13:09 schrieb Jani Nikula :

> On Tue, 07 Jun 2016, Markus Heiser  wrote:
>> Am 07.06.2016 um 10:59 schrieb Jani Nikula :
>>> One of the key arguments against too much splitting that hasn't been
>>> mentioned is that despite all the fine output Sphinx can produce, we
>>> have plenty of people who couldn't care less about running Sphinx to get
>>> readable documentation. They will grep and read the plain text files
>>> directly, and that's a large part of the appeal of any lightweight
>>> markup.
>> 
>> But they have read XML or compiled DocBook XML? ... Sphinx brings a
>> search engine with its html.
>> 
>>> When you split up a file into snippets that no longer tell a coherent
>>> story independently, you've failed.
>> 
>> Chapters are breaking stories?
>> 
>>> For the .txt files under Documentation, we mostly do not want to split
>>> them up any more if and when they're converted to rst. For the .tmpl
>>> files under Documentation/DocBook, each rst file split off from there
>>> should still be a sensible document on its own, with the filename
>>> telling what it's about. This will be the main benefit of this whole
>>> exercise for the people who do not care about Sphinx - instead of
>>> reading (read: ignoring) DocBook XML, they can now read the rst files.
>> 
>> Sorry but in IMO this suggestion is backward, if someone don't be able
>> to build HTML documents he should at least be able to use the
>> internet [1] :-o
>> 
>> [1] http://return42.github.io/sphkerneldoc/articles/books.html
> 
> I don't think you understand the target audience very well.

May be, help me .. I can't see the use case ... is emac's incremental search
your use case? ... possibly we are talking past each other

* chunking is my recommendation on books, e.g. chapters is a good point 
  to split a book .. beside: books which make use of the kernel-doc directive
  are not *well grep-able* in their reST format ..

* not every .txt file should be chunked ... and folders with the 00-INDEX
  are already chunked.

The only thing I want to say is, that I recommend chunking, but 
at the end it should be an author decision, not a handicap of the 
build system.

-- Markus --

> 
> BR,
> Jani.
> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Markus Heiser 
Hi all,

sorry I forgot to mentioning that I have not yet add the Makefile.reST
to the root Makefile. To test, please give the srctree environment in the
command line. E.g:

  cd /share/linux/Documentation
  srctree=/share/linux make -f Makefile.reST books/kernel-doc-HOWTO.html

-- Markus --



Am 06.06.2016 um 18:32 schrieb Markus Heiser :

> From: "Heiser, Markus" 
> 
> Hi Jonathan, Jani, all -
> 
> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
> 
>git://github.com/return42/linux.git linux-doc-reST
> 
> The specification of the implementation is the (new) kernel-doc-HOWTO book 
> which
> is a part of the patch series (also online avaliable at [2]).
> 
> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
> taking reST extension into account. It serves several purposes:
> 
> * user guide for kernel developers
> * specification of the kernel-doc parser
> * test case
> 
> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO, 
> the
> kernel-doc perl script is not needed for reST builds.  The python variant of 
> the
> kernel-doc parser supports the markup modes:
> 
> * kernel-doc (vintage)
> * reST
> 
> This series also includes the reST directives describe in the 
> kernel-doc-HOWTO:
> 
> * kernel-doc (uses/imports the parser from kernel_doc.py)
> * flat-table (table notation with meaningfull diffs)
> 
> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
> this folder a minimal python infrastructure is provided.
> 
> The basic sphinx-build infrastructure consists of:
> 
> * Documentation/Makefile.reST & Documentation/conf.py
> 
>  Makefile and basic 'sphinx config' file to build the various reST
>  documents and output formats. Provides the basic sphinx-doc build
>  infrastructure including the *sphinx-subprojects* feature. With this
>  feature each book can be build and distributed stand-alone.
> 
> * Documentation/books
> 
>  In this folder, the books with reST markup are placed. To
>  provide *sphinx-subprojects*, each book has its one folder and
>  a (optional) Documentation/books/{book-name}.conf file
>  which *overwrites* the basic configuration from Documentation/conf.py
> 
> Any comments are welcome, but please read [2] first.
> 
> With regards
> 
>  -- Markus --
> 
> [1] https://return42.github.io/sphkerneldoc
> [2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
> 
> 
> 
> Heiser, Markus (7):
>  python: add scripts/site-packages
>  sphinx-doc: add basic sphinx-build infrastructure
>  kernel-doc-HOWTO: add kernel-doc specification
>  linuxdoc: add python package linuxdoc
>  kernel-doc parser: inital python implementation
>  kernel-doc directive: initial implementation
>  flat-table directive: initial implementation
> 
> Documentation/.gitignore   |2 +
> Documentation/Makefile.reST|  111 +
> Documentation/books/.gitignore |0
> Documentation/books/kernel-doc-HOWTO.conf  |  200 ++
> .../books/kernel-doc-HOWTO/all-in-a-tumble-src.rst |   12 +
> .../books/kernel-doc-HOWTO/all-in-a-tumble.h   |  271 ++
> .../books/kernel-doc-HOWTO/all-in-a-tumble.rst |   11 +
> Documentation/books/kernel-doc-HOWTO/csv_table.txt |6 +
> Documentation/books/kernel-doc-HOWTO/index.rst |   46 +
> .../kernel-doc-HOWTO/kernel-doc-components.rst |   35 +
> .../kernel-doc-HOWTO/kernel-doc-directive.rst  |  199 ++
> .../books/kernel-doc-HOWTO/kernel-doc-examples.rst |   16 +
> .../books/kernel-doc-HOWTO/kernel-doc-intro.rst|   85 +
> .../books/kernel-doc-HOWTO/kernel-doc-syntax.rst   |  171 ++
> .../kernel-doc-HOWTO/reST-kernel-doc-mode.rst  |  120 +
> Documentation/books/kernel-doc-HOWTO/refs.txt  |   15 +
> .../books/kernel-doc-HOWTO/table-markup.rst|  499 
> .../books/kernel-doc-HOWTO/test/parser_test.h  |  332 +++
> .../kernel-doc-HOWTO/vintage-kernel-doc-mode.rst   |  100 +
> Documentation/conf.py  |  357 +++
> Documentation/sphinx-static/theme_overrides.css|   45 +
> Documentation/sphinx-tex/Makefile  |   88 +
> scripts/site-python/.gitignore |1 +
> scripts/site-python/README |   14 +
> scripts/site-python/linuxdoc/__init__.py   |8 +
> scripts/site-python/linuxdoc/kernel_doc.py | 2764 
> scripts/site-python/linuxdoc/rstFlatTable.py   |  356 +++
> scripts/site-python/linuxdoc/rstKernelDoc.py   |  369 +++
> 28 files changed, 6233 insertions(+)
> create mode 100644 Documentation/.gitignore
> create mode 100644 Documentation/Makefile.reST
> create mode 100644 Documentation/books/.gitignore
> create mode 100644 Documentation/books/kernel-doc-HOWTO.conf
> create mode 100644 
> Documentation/books/kernel-doc-HOWTO/all-in-a-tumble-src.rst
> create mode 100644

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Jani Nikula 
On Tue, 07 Jun 2016, Markus Heiser  wrote:
> Am 07.06.2016 um 10:59 schrieb Jani Nikula :
>> One of the key arguments against too much splitting that hasn't been
>> mentioned is that despite all the fine output Sphinx can produce, we
>> have plenty of people who couldn't care less about running Sphinx to get
>> readable documentation. They will grep and read the plain text files
>> directly, and that's a large part of the appeal of any lightweight
>> markup.
>
> But they have read XML or compiled DocBook XML? ... Sphinx brings a
> search engine with its html.
>
>> When you split up a file into snippets that no longer tell a coherent
>> story independently, you've failed.
>
> Chapters are breaking stories?
>
>> For the .txt files under Documentation, we mostly do not want to split
>> them up any more if and when they're converted to rst. For the .tmpl
>> files under Documentation/DocBook, each rst file split off from there
>> should still be a sensible document on its own, with the filename
>> telling what it's about. This will be the main benefit of this whole
>> exercise for the people who do not care about Sphinx - instead of
>> reading (read: ignoring) DocBook XML, they can now read the rst files.
>
> Sorry but in IMO this suggestion is backward, if someone don't be able
> to build HTML documents he should at least be able to use the
> internet [1] :-o
>
> [1] http://return42.github.io/sphkerneldoc/articles/books.html

I don't think you understand the target audience very well.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Markus Heiser 

Am 07.06.2016 um 10:59 schrieb Jani Nikula :

> On Tue, 07 Jun 2016, Daniel Vetter  wrote:
>> I think getting the flat-table directive landed would be great. We
>> have a similarly annoying table in gpu.tmpl where the ascii-art tables
>> fall short and are annoying. We want to split it up, but that's not a
>> short-term solution suitable for conversion. Atm it's licensed under
>> gplv3, which is incompatible with the kernel, so need to fix that.
> 
> Agreed.

see last mail 

>> I'm still not sold on the vintage-kerneldoc idea. At least in my
>> experience (after first converting to asciidoc and now to sphinx) this
>> is a non-issue. Yes, there's the oddball misrendering, but that's no
>> worse than the oddball typo. And a really good way for
>> drive-by-contributors and newbies to send in a few useful patches.
>> Personally I'm totally happy to have that bit of ugliness, same way I
>> don't reject patches just because they trip over checkpatch.pl in some
>> minor detail. The downside otoh means we'll have to forever maintain 2
>> versions of kernel-doc, and I don't want that. Jani have already
>> ripped out some of the more bonghits features of kernel-doc (like
>> anything with a \w: is a heading and other nonsense like that). The
>> only thing we really need to keep as special is the overall kerneldoc
>> layout, and the special referencing using &, %, @ and friends.
> 
> Agreed.

see last mail

>> Reimplementing kernel-doc in python is also something Jani
>> discussed. But I think it only makes sense if we go ahead and also
>> modernize the parser, i.e. use something like python-clang to parse C,
>> and build a modern parser-combinators thing for the kernel-doc itself.
>> The regex state machinery is a horror show, whether it's written in
>> cargo-culted perl or python.
> 
> I think the only sensible way to change the parser is to get everything
> working using the perl version first, and then you can diff the results
> against the python version. That's about the only validation of the new
> parser that actually matters in the real world.
> 
> At this time, I'd put this way down in the list of priorities.

I tested the python version of the parser by running it over nearly
the whole kernel-tree .. yes this is not a side by side test, but I also
tested it by partial side by side comparing the output of [1]

[1] https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc 

I spend over a week in testing ... IMO in the meantime the quality
of the python parser is a bit over the perl one. But at the end,
I have never seen a "error free implementation". 


>> I agree that we need to update the kernel-doc howto (and Jani's
>> working on that already too), but I think there's no need to split up
>> that finely. Maybe that's a bit against sphinx best practices, but
>> right now we have rather large documents (both plain text and
>> docbook). Splitting definitely makes sense in some cases (e.g.
>> gpu.tmpl is only this large because we couldn't do cross-template
>> referencing), but by-and-large I think the current splitting is mostly
>> reasonable.
> 
> One of the key arguments against too much splitting that hasn't been
> mentioned is that despite all the fine output Sphinx can produce, we
> have plenty of people who couldn't care less about running Sphinx to get
> readable documentation. They will grep and read the plain text files
> directly, and that's a large part of the appeal of any lightweight
> markup.

But they have read XML or compiled DocBook XML? ... Sphinx brings a
search engine with its html.

> When you split up a file into snippets that no longer tell a coherent
> story independently, you've failed.

Chapters are breaking stories?

> For the .txt files under Documentation, we mostly do not want to split
> them up any more if and when they're converted to rst. For the .tmpl
> files under Documentation/DocBook, each rst file split off from there
> should still be a sensible document on its own, with the filename
> telling what it's about. This will be the main benefit of this whole
> exercise for the people who do not care about Sphinx - instead of
> reading (read: ignoring) DocBook XML, they can now read the rst files.

Sorry but in IMO this suggestion is backward, if someone don't be able
to build HTML documents he should at least be able to use the
internet [1] :-o

[1] http://return42.github.io/sphkerneldoc/articles/books.html

> 
> BR,
> Jani.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Markus Heiser 

Am 07.06.2016 um 09:54 schrieb Daniel Vetter :

> On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser  
> wrote:
>> From: "Heiser, Markus" 
>> 
>> Hi Jonathan, Jani, all -
>> 
>> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>> 
>>git://github.com/return42/linux.git linux-doc-reST
>> 
>> The specification of the implementation is the (new) kernel-doc-HOWTO book 
>> which
>> is a part of the patch series (also online avaliable at [2]).
>> 
>> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
>> taking reST extension into account. It serves several purposes:
>> 
>> * user guide for kernel developers
>> * specification of the kernel-doc parser
>> * test case
>> 
>> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO, 
>> the
>> kernel-doc perl script is not needed for reST builds.  The python variant of 
>> the
>> kernel-doc parser supports the markup modes:
>> 
>> * kernel-doc (vintage)
>> * reST
>> 
>> This series also includes the reST directives describe in the 
>> kernel-doc-HOWTO:
>> 
>> * kernel-doc (uses/imports the parser from kernel_doc.py)
>> * flat-table (table notation with meaningfull diffs)
>> 
>> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
>> this folder a minimal python infrastructure is provided.
>> 
>> The basic sphinx-build infrastructure consists of:
>> 
>> * Documentation/Makefile.reST & Documentation/conf.py
>> 
>>  Makefile and basic 'sphinx config' file to build the various reST
>>  documents and output formats. Provides the basic sphinx-doc build
>>  infrastructure including the *sphinx-subprojects* feature. With this
>>  feature each book can be build and distributed stand-alone.
>> 
>> * Documentation/books
>> 
>>  In this folder, the books with reST markup are placed. To
>>  provide *sphinx-subprojects*, each book has its one folder and
>>  a (optional) Documentation/books/{book-name}.conf file
>>  which *overwrites* the basic configuration from Documentation/conf.py
>> 
>> Any comments are welcome, but please read [2] first.
> 
> I think getting the flat-table directive landed would be great. We
> have a similarly annoying table in gpu.tmpl where the ascii-art tables
> fall short and are annoying. We want to split it up, but that's not a
> short-term solution suitable for conversion. Atm it's licensed under
> gplv3, which is incompatible with the kernel, so need to fix that.

No problem, changed to gplv2

https://github.com/return42/linux/commit/97ee23e74384200fe79fe4557954a897725949e3

is this OK?

> I'm still not sold on the vintage-kerneldoc idea. At least in my
> experience (after first converting to asciidoc and now to sphinx) this
> is a non-issue. Yes, there's the oddball misrendering, but that's no
> worse than the oddball typo. And a really good way for
> drive-by-contributors and newbies to send in a few useful patches.
> Personally I'm totally happy to have that bit of ugliness, same way I
> don't reject patches just because they trip over checkpatch.pl in some
> minor detail. The downside otoh means we'll have to forever maintain 2
> versions of kernel-doc, and I don't want that.

This can be stopped by a reST lint process similar to what Jani mentioned.

> Jani have already
> ripped out some of the more bonghits features of kernel-doc (like
> anything with a \w: is a heading and other nonsense like that).

Yes, I removed it in the reST mode of the parser. This is also
a reason to differentiate between *vintage* and reST or you will
break tons of kernel-doc comments, which are valid in the term of 
the *vintage* style.

> The only thing we really need to keep as special is the overall kerneldoc
> layout, and the special referencing using &, %, @ and friends.

Ok, I will try to add this highlighting feature to the reST mode.
I left it out because I feared that there are any collision with
reST markup, but with a closer look the fear might be unfounded.


> Reimplementing kernel-doc in python is also something Jani
> discussed. But I think it only makes sense if we go ahead and also
> modernize the parser, i.e. use something like python-clang to parse C,
> and build a modern parser-combinators thing for the kernel-doc itself.
> The regex state machinery is a horror show, whether it's written in
> cargo-culted perl or python.

Yes clang might be better solution and be more flexible on parsing
C, but it brings also much more decencies, but no one has tested
it against the kernel tree (I can't judge it, I have no experience 
with) ... the regular expressions are proofed.
 
But anyway, if someone implements a C parser based on clang we can
use it, but before the python implementation is the best we have ;-)

> I agree that we need to update the kernel-doc howto (and Jani's
> working on that already too), but I think there's no need to split up
> that finely. Maybe that's a bit against sphinx

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Jani Nikula 
On Tue, 07 Jun 2016, Daniel Vetter  wrote:
> I think getting the flat-table directive landed would be great. We
> have a similarly annoying table in gpu.tmpl where the ascii-art tables
> fall short and are annoying. We want to split it up, but that's not a
> short-term solution suitable for conversion. Atm it's licensed under
> gplv3, which is incompatible with the kernel, so need to fix that.

Agreed.

> I'm still not sold on the vintage-kerneldoc idea. At least in my
> experience (after first converting to asciidoc and now to sphinx) this
> is a non-issue. Yes, there's the oddball misrendering, but that's no
> worse than the oddball typo. And a really good way for
> drive-by-contributors and newbies to send in a few useful patches.
> Personally I'm totally happy to have that bit of ugliness, same way I
> don't reject patches just because they trip over checkpatch.pl in some
> minor detail. The downside otoh means we'll have to forever maintain 2
> versions of kernel-doc, and I don't want that. Jani have already
> ripped out some of the more bonghits features of kernel-doc (like
> anything with a \w: is a heading and other nonsense like that). The
> only thing we really need to keep as special is the overall kerneldoc
> layout, and the special referencing using &, %, @ and friends.

Agreed.

> Reimplementing kernel-doc in python is also something Jani
> discussed. But I think it only makes sense if we go ahead and also
> modernize the parser, i.e. use something like python-clang to parse C,
> and build a modern parser-combinators thing for the kernel-doc itself.
> The regex state machinery is a horror show, whether it's written in
> cargo-culted perl or python.

I think the only sensible way to change the parser is to get everything
working using the perl version first, and then you can diff the results
against the python version. That's about the only validation of the new
parser that actually matters in the real world.

At this time, I'd put this way down in the list of priorities.

> I agree that we need to update the kernel-doc howto (and Jani's
> working on that already too), but I think there's no need to split up
> that finely. Maybe that's a bit against sphinx best practices, but
> right now we have rather large documents (both plain text and
> docbook). Splitting definitely makes sense in some cases (e.g.
> gpu.tmpl is only this large because we couldn't do cross-template
> referencing), but by-and-large I think the current splitting is mostly
> reasonable.

One of the key arguments against too much splitting that hasn't been
mentioned is that despite all the fine output Sphinx can produce, we
have plenty of people who couldn't care less about running Sphinx to get
readable documentation. They will grep and read the plain text files
directly, and that's a large part of the appeal of any lightweight
markup.

When you split up a file into snippets that no longer tell a coherent
story independently, you've failed.

For the .txt files under Documentation, we mostly do not want to split
them up any more if and when they're converted to rst. For the .tmpl
files under Documentation/DocBook, each rst file split off from there
should still be a sensible document on its own, with the filename
telling what it's about. This will be the main benefit of this whole
exercise for the people who do not care about Sphinx - instead of
reading (read: ignoring) DocBook XML, they can now read the rst files.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation

2016-06-07 Thread Daniel Vetter 
On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser  wrote:
> From: "Heiser, Markus" 
>
> Hi Jonathan, Jani, all -
>
> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>
> git://github.com/return42/linux.git linux-doc-reST
>
> The specification of the implementation is the (new) kernel-doc-HOWTO book 
> which
> is a part of the patch series (also online avaliable at [2]).
>
> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
> taking reST extension into account. It serves several purposes:
>
> * user guide for kernel developers
> * specification of the kernel-doc parser
> * test case
>
> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO, 
> the
> kernel-doc perl script is not needed for reST builds.  The python variant of 
> the
> kernel-doc parser supports the markup modes:
>
>  * kernel-doc (vintage)
>  * reST
>
> This series also includes the reST directives describe in the 
> kernel-doc-HOWTO:
>
> * kernel-doc (uses/imports the parser from kernel_doc.py)
> * flat-table (table notation with meaningfull diffs)
>
> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
> this folder a minimal python infrastructure is provided.
>
> The basic sphinx-build infrastructure consists of:
>
> * Documentation/Makefile.reST & Documentation/conf.py
>
>   Makefile and basic 'sphinx config' file to build the various reST
>   documents and output formats. Provides the basic sphinx-doc build
>   infrastructure including the *sphinx-subprojects* feature. With this
>   feature each book can be build and distributed stand-alone.
>
> * Documentation/books
>
>   In this folder, the books with reST markup are placed. To
>   provide *sphinx-subprojects*, each book has its one folder and
>   a (optional) Documentation/books/{book-name}.conf file
>   which *overwrites* the basic configuration from Documentation/conf.py
>
> Any comments are welcome, but please read [2] first.

I think getting the flat-table directive landed would be great. We
have a similarly annoying table in gpu.tmpl where the ascii-art tables
fall short and are annoying. We want to split it up, but that's not a
short-term solution suitable for conversion. Atm it's licensed under
gplv3, which is incompatible with the kernel, so need to fix that.

I'm still not sold on the vintage-kerneldoc idea. At least in my
experience (after first converting to asciidoc and now to sphinx) this
is a non-issue. Yes, there's the oddball misrendering, but that's no
worse than the oddball typo. And a really good way for
drive-by-contributors and newbies to send in a few useful patches.
Personally I'm totally happy to have that bit of ugliness, same way I
don't reject patches just because they trip over checkpatch.pl in some
minor detail. The downside otoh means we'll have to forever maintain 2
versions of kernel-doc, and I don't want that. Jani have already
ripped out some of the more bonghits features of kernel-doc (like
anything with a \w: is a heading and other nonsense like that). The
only thing we really need to keep as special is the overall kerneldoc
layout, and the special referencing using &, %, @ and friends.

Reimplementing kernel-doc in python is also something Jani
discussed. But I think it only makes sense if we go ahead and also
modernize the parser, i.e. use something like python-clang to parse C,
and build a modern parser-combinators thing for the kernel-doc itself.
The regex state machinery is a horror show, whether it's written in
cargo-culted perl or python.

I agree that we need to update the kernel-doc howto (and Jani's
working on that already too), but I think there's no need to split up
that finely. Maybe that's a bit against sphinx best practices, but
right now we have rather large documents (both plain text and
docbook). Splitting definitely makes sense in some cases (e.g.
gpu.tmpl is only this large because we couldn't do cross-template
referencing), but by-and-large I think the current splitting is mostly
reasonable. A better demonstration vehicle is imo converting all
existing .tmpl to .rst. And also making sure the new kernel-doc is
still able to parse all existing kernel-doc comments, even when
they're not pulled into .tmpl files. Jani has done that (using
hacked-up versions of the sphinx-lint script) for his tree, making
sure there's no unexpected changes anywhere.

I think next steps would be:
- rebase flat-table onto Jani's work and relicense under gplv2
- look into rewriting kernel-doc in python more as a long-term project
- start converting docs instead - I really want to start reaping
benefits of all this work as soon as possible.

Thanks, Daniel


>
> With regards
>
>   -- Markus --
>
> [1] https://return42.github.io/sphkerneldoc
> [2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
>
>
>
> Heiser, Markus (7):
>   python: add scripts/site-packages
>   sphinx-doc: add basic