Re: [PATCH] kernel-doc: better handle '::' sequences

[Markus Heiser]

Am 30.03.21 um 13:35 schrieb Jani Nikula:

If the introduction were "/*rST" instead of "/**", would we have
consensus?  It gives us a path to let people intermix kernel-doc and
hawkmoth comments in the same file, which would be amazing.



If you want to allow two syntaxes for documentation comments (current
kernel-doc and pure reStructuredText with just the comment markers and
indentation removed) I think the natural first step would be to modify
kernel-doc the perl script to support that. It would probably even be
trivial.


My 2cent: to tag the markup of the documentation, in python they
use a variable named __docformat__ [PEP-258] / e.g.:

__docformat__ = "restructuredtext en"

[PEP-258] https://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format

> Perhaps the bare minimum is running rustdoc first, and generating the
> results into Sphinx static pages [1], to make them part of the
> whole. Even if the HTML style might be different.

Cross referencing will be problematic, I think.

  -- Markus --

Re: [RFC] scripts: kernel-doc: avoid warnings due to initial commented lines in file

[Markus Heiser]

Am 09.03.21 um 13:53 schrieb Aditya Srivastava:

Starting commented lines in a file mostly contains comments describing
license, copyright or general information about the file.

E.g., in sound/pci/ctxfi/ctresource.c, initial comment lines describe
its copyright and other related file informations.


The opening comment mark /** is used for kernel-doc comments [1]

[1] 
https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#how-to-format-kernel-doc-comments


  -- Markus --



But as kernel-doc reads these lines, it results in ineffective warnings by
kernel-doc, related to these.

Provide a simple fix by skipping first three lines in a file for checking
kernel-doc comments.

Suggested-by: Lukas Bulwahn 
Signed-off-by: Aditya Srivastava 
---
  scripts/kernel-doc | 6 +-
  1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index e1e562b2e2e7..431add05248e 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2375,6 +2375,7 @@ sub process_file($) {
  my $file;
  my $initial_section_counter = $section_counter;
  my ($orig_file) = @_;
+my $lineno = 0;# to maintain the count of line number in a file
  
  $file = map_filename($orig_file);
  
@@ -2388,13 +2389,16 @@ sub process_file($) {
  
  $section_counter = 0;

  while () {
+   $lineno++;
while (s/\\\s*$//) {
$_ .= ;
+   $lineno++;
}
# Replace tabs by spaces
  while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
# Hand this line to the appropriate state handler
-   if ($state == STATE_NORMAL) {
+   if ($state == STATE_NORMAL
+   && $lineno > 3) {# to avoid starting comment lines 
describing the file
process_normal();
} elsif ($state == STATE_NAME) {
process_name($file, $_);

Re: linux-next: build failure after upgrading sphinx

[Markus Heiser]

Am 31.08.20 um 21:03 schrieb Jonathan Corbet:

On Thu, 27 Aug 2020 14:50:17 +1000
Stephen Rothwell  wrote:


Today I upgraded ot sphinx v3.2.1 and got the following error from
"make htmldocs":

Running Sphinx v3.2.1
enabling CJK for LaTeX builder

Extension error:
Could not import extension cdomain (exception: cannot import name 
'c_funcptr_sig_re' from 'sphinx.domains.c' 
(/usr/lib/python3/dist-packages/sphinx/domains/c.py))

I have downgraded to version 2.4.3 and await suggestions/patches :-)


[Adding Markus]

Markus, this looks like an issue with the "handle function-like macros"
code that has your name on it.  The Sphinx folks think that functionality
can just be removed:

https://github.com/sphinx-doc/sphinx/issues/7421

Do you agree?  We need to look at what they're saying about the :name:
directive as well; somehow I missed that when it first went in.

Thanks,

jon


Hi Jon, thanks for taking me into CC.

I guess you refer this post:

  https://github.com/sphinx-doc/sphinx/issues/7421#issuecomment-609830660

What I know is: The Sphinx >= v3.0 includes a "C, initial
rewrite" which is not downward compatible.

 https://github.com/sphinx-doc/sphinx/commit/0f49e30c#diff-59e33b0

---

To give an short answer to opener's question:

Our Sphinx-build is not Sphinx >= v3.0 ready.  We recommend
to follow our installation instructions [1] and install
requirements by::

  (virtualenv) $ pip install -r Documentation/sphinx/requirements.txt

In the requirements.txt we stick Sphinx at 'Sphinx==2.4.4'.
In my personal opinion there are more problems than just the
C-domain when using other Sphinx Versions (e.g. PDF is most
often problematic).

[1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install

---

I can't say if this *rewrite* has a better "handle
function-like macros" or not.

Since the changes are not downward compatible,
Documentation/sphinx/cdomain.py needs a rewrite
(Unfortunately, I hadn't had the time to do this).

TL;DR;

In my linuxdoc [2] project I use the same cdomain.py
implementation and split cdomain into v2 and v3 [3].  ATM
linuxdoc/cdomainv3.py is just a skeleton which is used in
Sphinx >= v3.0 installations.  My experience is, that it
spits out more noise, but I haven't had time to look closer
right now.

[2] https://return42.github.io/linuxdoc
[3] https://github.com/return42/linuxdoc/commit/70673dc


  -- Markus --

Re: Documentation: build failure with sphinx >= 3.0.0: exception: cannot import name 'c_funcptr_sig_re' from 'sphinx.domains.c'

[Markus Heiser]

Am 12.08.20 um 10:21 schrieb Markus Heiser:


Am 12.08.20 um 09:30 schrieb Salvatore Bonaccorso:
[..]

The problem is actually related to changes happening in Sphinx 3.0.0.
There is the followign issue filled upstream:

https://github.com/sphinx-doc/sphinx/issues/7421

'c_funcptr_sig_re' was removed upstream in sphinx v3.0.0b1 and so the
kernel documentation build fails. This is reproducible with a recent
sphinx version (in attached case it is 3.2.0):

$ make PYTHON=python3 xmldocs
   SPHINX  xmldocs --> file:///home/build/linux/Documentation/output/xml
make[2]: Nothing to be done for 'xml'.
Running Sphinx v3.2.0

Extension error:
Could not import extension cdomain (exception: cannot import name 
'c_funcptr_sig_re' from 'sphinx.domains.c' 
(/usr/lib/python3/dist-packages/sphinx/domains/c.py))

make[1]: *** [Documentation/Makefile:115: xmldocs] Error 2
make: *** [Makefile:1655: xmldocs] Error 2

Distribution reports related to this issue:
https://bugs.debian.org/963636
https://bugs.archlinux.org/task/66178
https://bugs.archlinux.org/task/66156


As a workaround to make the documentation build again (but known that
parts of the documentation will be broken), we could drop the cdomain
extension.

Regards,
Salvatore


@jon, do you have time to implement a patch?
.. sorry, I'am in a hurry :o

In the linked github issue you find also a patch that fixes a
cdomain.py.  I tested the patch (shee below) in the linux kernel.

   For me it works.


Sorry, I have to correct:

  a Patch needs more work

sphinx-doc has changed the way how domains are implemented [1].

-- Markus --

[1] https://github.com/sphinx-doc/sphinx/commit/0f49e30c

Re: Documentation: build failure with sphinx >= 3.0.0: exception: cannot import name 'c_funcptr_sig_re' from 'sphinx.domains.c'

[Markus Heiser]

Am 12.08.20 um 09:30 schrieb Salvatore Bonaccorso:
[..]

The problem is actually related to changes happening in Sphinx 3.0.0.
There is the followign issue filled upstream:

https://github.com/sphinx-doc/sphinx/issues/7421

'c_funcptr_sig_re' was removed upstream in sphinx v3.0.0b1 and so the
kernel documentation build fails. This is reproducible with a recent
sphinx version (in attached case it is 3.2.0):

$ make PYTHON=python3 xmldocs
   SPHINX  xmldocs --> file:///home/build/linux/Documentation/output/xml
make[2]: Nothing to be done for 'xml'.
Running Sphinx v3.2.0

Extension error:
Could not import extension cdomain (exception: cannot import name 
'c_funcptr_sig_re' from 'sphinx.domains.c' 
(/usr/lib/python3/dist-packages/sphinx/domains/c.py))
make[1]: *** [Documentation/Makefile:115: xmldocs] Error 2
make: *** [Makefile:1655: xmldocs] Error 2

Distribution reports related to this issue:
https://bugs.debian.org/963636
https://bugs.archlinux.org/task/66178
https://bugs.archlinux.org/task/66156


As a workaround to make the documentation build again (but known that
parts of the documentation will be broken), we could drop the cdomain
extension.

Regards,
Salvatore


@jon, do you have time to implement a patch?
.. sorry, I'am in a hurry :o

In the linked github issue you find also a patch that fixes a
cdomain.py.  I tested the patch (shee below) in the linux kernel.

  For me it works.

BTW: The patch of Documentation/sphinx/requirements.txt
was only nedded for a test.

  -- Markus --


---


diff --git a/Documentation/sphinx/cdomain.py b/Documentation/sphinx/cdomain.py
index cbac8e608dc4..65e15d48891e 100644
--- a/Documentation/sphinx/cdomain.py
+++ b/Documentation/sphinx/cdomain.py
@@ -31,16 +31,35 @@ u"""
arguments types of function-like macros.

 """
+import re

 from docutils import nodes
 from docutils.parsers.rst import directives

 import sphinx
 from sphinx import addnodes
-from sphinx.domains.c import c_funcptr_sig_re, c_sig_re
 from sphinx.domains.c import CObject as Base_CObject
 from sphinx.domains.c import CDomain as Base_CDomain

+# C from commit https://github.com/return42/linuxdoc/commit/48f09de2
+# fixes 
https://github.com/sphinx-doc/sphinx/commit/0f49e30c51b5cc5055cda5b4b294c2dd9d1df573#r38750737

+
+# pylint: disable=invalid-name
+c_sig_re = re.compile(
+r'''^([^(]*?)  # return type
+([\w:.]+)  \s* # thing name (colon allowed for C++)
+(?: \((.*)\) )?# optionally arguments
+(\s+const)? $  # const specifier
+''', re.VERBOSE)
+
+c_funcptr_sig_re = re.compile(
+r'''^([^(]+?)  # return type
+(\( [^()]+ \)) \s* # name in parentheses
+\( (.*) \) # arguments
+(\s+const)? $  # const specifier
+''', re.VERBOSE)
+# pylint: enable=invalid-name
+
 __version__  = '1.0'

 # Get Sphinx version
diff --git a/Documentation/sphinx/requirements.txt 
b/Documentation/sphinx/requirements.txt

index 489f6626de67..f7486fd7ec85 100644
--- a/Documentation/sphinx/requirements.txt
+++ b/Documentation/sphinx/requirements.txt
@@ -1,3 +1,3 @@
 docutils
-Sphinx==2.4.4
+Sphinx==3.2.0
 sphinx_rtd_theme

Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book

[Markus Heiser]

Am 14.06.19 um 15:42 schrieb Jani Nikula:

On Thu, 13 Jun 2019, Mauro Carvalho Chehab  wrote:

From: Mauro Carvalho Chehab

As we don't want a generic Sphinx extension to execute commands,
change the one proposed to Markus to call the abi_book.pl
script.

Use a script to parse the Documentation/ABI directory and output
it at the admin-guide.

We had a legacy kernel-doc perl script so we used that instead of
rewriting it in python. Just to keep it bug-for-bug compatible with the
past. That was the only reason.

I see absolutely zero reason to add a new perl monstrosity with a python
extension to call it. All of this could be better done in python,
directly.

Please don't complicate the documentation build. I know you know we all
worked hard to take apart the old DocBook Rube Goldberg machine to
replace it with Sphinx. Please don't turn the Sphinx build to another
complicated mess.

My strong preferences are, in this order:

1) Convert the ABI documentation to reStructuredText

2) Have the python extension read the ABI files directly, without an
extra pipeline.



I agree with Jani. No matter how the decision ends, since I can't help here, I'd 
rather not show up in the copyright.


  -- Markus --

Re: [PATCH 0/9] docs: Fix various build warnings/errors

[Markus Heiser]

Am 08.03.19 um 21:16 schrieb Tobin C. Harding:

Hi Tobin,

the problem was a missing empty line (see my comment on this patch).

The reST primer from sphinx-doc is always a good reference for the daily work

   
http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#directives

Thanks Marcus, that's actually how I came to the conclusion that they
should by asterisks.


http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks

Anyways, as usual LKML sorted me out :)


Its only a 'primer', a reduced introduction.  But it gives also some "(ref)"s
to the docutils origin where the refernce of reST is located.

E.g. the first "(ref)" of the chapter "Lists and Quote-like blocks" (you linked
above) points to the bullet list reference at:

  
http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#bullet-lists

"""A text block which begins with a "*", "+", "-", "•", "‣", or "⁃", followed by 
whitespace, is a bullet list item"""



-- Markus --

Re: [PATCH 0/9] docs: Fix various build warnings/errors

[Markus Heiser]

Am 07.03.19 um 22:11 schrieb Tobin C. Harding:

Hi,

I had a few hours to spare so I thought I'd clear some Sphinx build
warnings/errors.  There isn't anything too controversial here.  The only
interesting thing I hit was in patch 7 (docs: Remove unknown 'hint'
directive), I couldn't work out if this was a serious directive, a joke,
or a typo.  


Hi Tobin,

the problem was a missing empty line (see my comment on this patch).

The reST primer from sphinx-doc is always a good reference for the daily work

  
http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#directives

-- Markus --

Re: [PATCH 7/9] docs: Remove unknown 'hint' directive

[Markus Heiser]

Am 08.03.19 um 04:51 schrieb Randy Dunlap:

On 3/7/19 1:11 PM, Tobin C. Harding wrote:

Current RST file contains an unknown directive causing Sphinx to emit

ERROR: Unexpected indentation.

Use normal language construct instead.

Signed-off-by: Tobin C. Harding 


This is a good idea.  My previous patch eliminated the warning but the
..hint is not presented very well in the generated output.  :)

Thanks.


---
  Documentation/driver-api/dmaengine/dmatest.rst | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/driver-api/dmaengine/dmatest.rst 
b/Documentation/driver-api/dmaengine/dmatest.rst
index 8d81f1a7169b..25eecd2769b0 100644
--- a/Documentation/driver-api/dmaengine/dmatest.rst
+++ b/Documentation/driver-api/dmaengine/dmatest.rst
@@ -59,8 +59,8 @@ parameter, that specific channel is requested using the 
dmaengine and a thread
  is created with the existing parameters. This thread is set as pending
  and will be executed once run is set to 1. Any parameters set after the thread
  is created are not applied.



Here a blank line is missed. Thats while '.. hint:' directive is not detected
well.  Without the blank line the '.. hint:` string is a part of the section
above.


-.. hint::
-  available channel list could be extracted by running the following command::
+
+Hint: available channel list could be extracted by running the following 
command::
  
  % ls -1 /sys/class/dma/


  -- Markus --

Re: [PATCH 10/17] prmem: documentation

[Markus Heiser]

Am 26.10.18 um 11:26 schrieb Peter Zijlstra:
>> Jon,
>>
>> So the below document is a prime example for why I think RST sucks. As a
>> text document readability is greatly diminished by all the markup
>> nonsense.
>>
>> This stuff should not become write-only content like html and other
>> gunk. The actual text file is still the primary means of reading this.
>
> I think Igor neglected to read doc-guide/sphinx.rst:
>
> Specific guidelines for the kernel documentation
> 
>
> Here are some specific guidelines for the kernel documentation:
>
> * Please don't go overboard with reStructuredText markup. Keep it
>simple. For the most part the documentation should be plain text with
>just enough consistency in formatting that it can be converted to
>other formats.
>
> I agree that it's overly marked up.

To add my two cent ..

> WTH is with all those ** ?

I guess Igor was looking for a definition list ...

>> +Available protections for kernel data
>> +-
>> +
>> +- **constant**
>> +   Labelled as **const**, the data is never supposed to be altered.
>> +   It is statically allocated - if it has any memory footprint at all.
>> +   The compiler can even optimize it away, where possible, by replacing
>> +   references to a **const** with its actual value.


+Available protections for kernel data
+-
+
+constant
+   Labelled as const, the data is never supposed to be altered.
+   It is statically allocated - if it has any memory footprint at all.
+   The compiler can even optimize it away, where possible, by replacing
+   references to a const with its actual value.

see "Lists and Quote-like blocks" [1]

[1] 
http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks


-- Markus --

Re: [PATCH 10/17] prmem: documentation

[Markus Heiser]

Am 26.10.18 um 11:26 schrieb Peter Zijlstra:
>> Jon,
>>
>> So the below document is a prime example for why I think RST sucks. As a
>> text document readability is greatly diminished by all the markup
>> nonsense.
>>
>> This stuff should not become write-only content like html and other
>> gunk. The actual text file is still the primary means of reading this.
>
> I think Igor neglected to read doc-guide/sphinx.rst:
>
> Specific guidelines for the kernel documentation
> 
>
> Here are some specific guidelines for the kernel documentation:
>
> * Please don't go overboard with reStructuredText markup. Keep it
>simple. For the most part the documentation should be plain text with
>just enough consistency in formatting that it can be converted to
>other formats.
>
> I agree that it's overly marked up.

To add my two cent ..

> WTH is with all those ** ?

I guess Igor was looking for a definition list ...

>> +Available protections for kernel data
>> +-
>> +
>> +- **constant**
>> +   Labelled as **const**, the data is never supposed to be altered.
>> +   It is statically allocated - if it has any memory footprint at all.
>> +   The compiler can even optimize it away, where possible, by replacing
>> +   references to a **const** with its actual value.


+Available protections for kernel data
+-
+
+constant
+   Labelled as const, the data is never supposed to be altered.
+   It is statically allocated - if it has any memory footprint at all.
+   The compiler can even optimize it away, where possible, by replacing
+   references to a const with its actual value.

see "Lists and Quote-like blocks" [1]

[1] 
http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#lists-and-quote-like-blocks


-- Markus --

Re: [PATCH 13/18] wait: wait.h: Get rid of a kernel-doc/Sphinx warnings

[Markus Heiser]

> Am 10.05.2018 um 18:42 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Thu, 10 May 2018 09:38:46 -0600
> Jonathan Corbet  escreveu:
> 
>> On Thu, 10 May 2018 11:21:13 -0300
>> Mauro Carvalho Chehab  wrote:
>> 
>>> The problem with a hint-based mechanism is that it will generate
>>> false hints. If added, we may end by needing to add extra tags to
>>> disable the hints mechanism where it gets wrong, or to periodically
>>> do code changes at kernel-doc comments in order to make the hints
>>> logic happy.
>>> 
>>> So, IMO, we should provide non-hints based mechanism, like forcing the
>>> string that prepends the colon to have a keyword that will make it to
>>> parse the block as literal, where expressions like:
>>> 
>>> See the code-block foo:
>>> See the following code example:
>>> See the following flow diagram:
>>> See the following artwork:
>>> 
>>> Is the best alternative to avoid "::", as on the enclosed patch.  
>> 
>> But this, too, is a hint-based mechanism.  Thanks for the patches, I'll
>> keep them around, but I would like an opportunity to try to do better
>> before applying them.  I fear that using magic words in this way will
>> lead to a constant stream of surprises, and I'd like to avoid that if
>> possible...
> 
> Yes, it is still hint-based. A careful selection of the "magic spell
> words/phrases" would minimize the risks of false positives, but it
> could still lead into some unwanted surprises.
> 
> IMO, "::" (or some other character combination that is not used
> elsewhere) is still the safest option.

Right, let's just stay with reST:

 http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

We already have some special kernel-doc additions e.g. for highlighting,
cross referencing and "DOC:":

 
https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#how-to-format-kernel-doc-comments

But we should not break with reST fundamentals.

There are other plain text markups on the market, I would remember Markdown as 
one.
They all come with markup rules (syntax), to make text parseable. Mauro brought 
the
example with lists and colons. In ReST, the "::" introduce an indented literal 
block,
which can be used for code block examples or a small ASCII art.

FWIW: at docutils there is also a (slow ongoing) discussion about reST syntax
alternatives  

 http://docutils.sourceforge.net/docs/dev/rst/alternatives.html

may the syntax discussion is better placed there?

-- Markus --

Re: [PATCH 13/18] wait: wait.h: Get rid of a kernel-doc/Sphinx warnings

[Markus Heiser]

> Am 10.05.2018 um 18:42 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Thu, 10 May 2018 09:38:46 -0600
> Jonathan Corbet  escreveu:
> 
>> On Thu, 10 May 2018 11:21:13 -0300
>> Mauro Carvalho Chehab  wrote:
>> 
>>> The problem with a hint-based mechanism is that it will generate
>>> false hints. If added, we may end by needing to add extra tags to
>>> disable the hints mechanism where it gets wrong, or to periodically
>>> do code changes at kernel-doc comments in order to make the hints
>>> logic happy.
>>> 
>>> So, IMO, we should provide non-hints based mechanism, like forcing the
>>> string that prepends the colon to have a keyword that will make it to
>>> parse the block as literal, where expressions like:
>>> 
>>> See the code-block foo:
>>> See the following code example:
>>> See the following flow diagram:
>>> See the following artwork:
>>> 
>>> Is the best alternative to avoid "::", as on the enclosed patch.  
>> 
>> But this, too, is a hint-based mechanism.  Thanks for the patches, I'll
>> keep them around, but I would like an opportunity to try to do better
>> before applying them.  I fear that using magic words in this way will
>> lead to a constant stream of surprises, and I'd like to avoid that if
>> possible...
> 
> Yes, it is still hint-based. A careful selection of the "magic spell
> words/phrases" would minimize the risks of false positives, but it
> could still lead into some unwanted surprises.
> 
> IMO, "::" (or some other character combination that is not used
> elsewhere) is still the safest option.

Right, let's just stay with reST:

 http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

We already have some special kernel-doc additions e.g. for highlighting,
cross referencing and "DOC:":

 
https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#how-to-format-kernel-doc-comments

But we should not break with reST fundamentals.

There are other plain text markups on the market, I would remember Markdown as 
one.
They all come with markup rules (syntax), to make text parseable. Mauro brought 
the
example with lists and colons. In ReST, the "::" introduce an indented literal 
block,
which can be used for code block examples or a small ASCII art.

FWIW: at docutils there is also a (slow ongoing) discussion about reST syntax
alternatives  

 http://docutils.sourceforge.net/docs/dev/rst/alternatives.html

may the syntax discussion is better placed there?

-- Markus --

Re: [PATCH 13/18] wait: wait.h: Get rid of a kernel-doc/Sphinx warnings

[Markus Heiser]

> Am 09.05.2018 um 20:35 schrieb Jonathan Corbet :
> 
>> Why should I care for people not using text editors to write code?

Eh? .. why must readers write code?

We have rules how to write kernel-doc markup since the beginning of the century.
If you do not want to contribute kernel-doc markups, then just leave it, change
your comment start-tag to

 /*

and feel free to type what ever you want. Sorry, but this discussion is 
superfluous.

-- Markus -

Re: [PATCH 13/18] wait: wait.h: Get rid of a kernel-doc/Sphinx warnings

[Markus Heiser]

> Am 09.05.2018 um 20:35 schrieb Jonathan Corbet :
> 
>> Why should I care for people not using text editors to write code?

Eh? .. why must readers write code?

We have rules how to write kernel-doc markup since the beginning of the century.
If you do not want to contribute kernel-doc markups, then just leave it, change
your comment start-tag to

 /*

and feel free to type what ever you want. Sorry, but this discussion is 
superfluous.

-- Markus -

Re: make xmldocs failed with error after 4.17 merge period

[Markus Heiser]

> Am 06.04.2018 um 12:51 schrieb Heikki Krogerus 
> <heikki.kroge...@linux.intel.com>:
> 
> Hi Markus,
> 
> On Fri, Apr 06, 2018 at 12:03:55PM +0200, Markus Heiser wrote:
>>>> There are ways to do this, look at how the v4l2 and I think the drm
>>>> subsystems handle ascii art such that "real" drawings end up being
>>>> produced.
>>> 
>>> Thanks. I did not actually find anything else except use of tables and
>>> code-blocks in v4l documentation. Is that what you were referring?
>> 
>> If it is about *figures*: we have a directive named 'kernel-figure',
>> which is a full replacement of the 'figure' directive from Sphinx-Doc.
>> In addition it supports *inline* SVG and DOT markups. Read:
>> 
>> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#figures-images
> 
> Thanks for the info.
> 
> I don't want to use for example dot language in kernel documentation.
> I want to be able to clearly see the figure also from the plain text
> file. That's why I prefer ascii art.
> 
> Isn't there a way we could render ascii art as svg diagram?

Sorry, not yet. Johannes started a discussion about this. Its a long
time ago and I do not have any new ideas yet :/ see:

  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07035.html

-- Markus --

Re: make xmldocs failed with error after 4.17 merge period

[Markus Heiser]

> Am 06.04.2018 um 12:51 schrieb Heikki Krogerus 
> :
> 
> Hi Markus,
> 
> On Fri, Apr 06, 2018 at 12:03:55PM +0200, Markus Heiser wrote:
>>>> There are ways to do this, look at how the v4l2 and I think the drm
>>>> subsystems handle ascii art such that "real" drawings end up being
>>>> produced.
>>> 
>>> Thanks. I did not actually find anything else except use of tables and
>>> code-blocks in v4l documentation. Is that what you were referring?
>> 
>> If it is about *figures*: we have a directive named 'kernel-figure',
>> which is a full replacement of the 'figure' directive from Sphinx-Doc.
>> In addition it supports *inline* SVG and DOT markups. Read:
>> 
>> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#figures-images
> 
> Thanks for the info.
> 
> I don't want to use for example dot language in kernel documentation.
> I want to be able to clearly see the figure also from the plain text
> file. That's why I prefer ascii art.
> 
> Isn't there a way we could render ascii art as svg diagram?

Sorry, not yet. Johannes started a discussion about this. Its a long
time ago and I do not have any new ideas yet :/ see:

  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07035.html

-- Markus --

Re: make xmldocs failed with error after 4.17 merge period

[Markus Heiser]

> Am 06.04.2018 um 11:11 schrieb Heikki Krogerus 
> :
[...]

>> An ascii graphic in typec.rst cause the error.
> 
> Thanks for the report. I'm going to propose that we fix this by
> marking the ascii art as comment:
> 
> diff --git a/Documentation/driver-api/usb/typec.rst 
> b/Documentation/driver-api/usb/typec.rst
> index feb31946490b..972c11bf4141 100644
> --- a/Documentation/driver-api/usb/typec.rst
> +++ b/Documentation/driver-api/usb/typec.rst
> @@ -212,7 +212,7 @@ port drivers can use USB Role Class API with those.
> 
> Illustration of the muxes behind a connector that supports an alternate 
> mode:
> 
> - 
> +..   
> |   Connector  |
> 
>| |
> 
> I hope that works.
 
 Try it and see!  :)
>>> 
>>> It will fix this issue. I was just wondering if use of ascii art is
>>> acceptable in general with the .rst files? But then again, why
>>> wouldn't it be.

[...]

> I was propsed to use something called "Literal Block" with ascii art.
> http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#literal-blocks

about *ASCII-art*: see fix from Jani ...

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg19302.html

where the '::' is a short-markup for a literal-block.


>> There are ways to do this, look at how the v4l2 and I think the drm
>> subsystems handle ascii art such that "real" drawings end up being
>> produced.
> 
> Thanks. I did not actually find anything else except use of tables and
> code-blocks in v4l documentation. Is that what you were referring?

If it is about *figures*: we have a directive named 'kernel-figure',
which is a full replacement of the 'figure' directive from Sphinx-Doc.
In addition it supports *inline* SVG and DOT markups. Read:

 https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#figures-images

-- Markus --

Re: make xmldocs failed with error after 4.17 merge period

[Markus Heiser]

> Am 06.04.2018 um 11:11 schrieb Heikki Krogerus 
> :
[...]

>> An ascii graphic in typec.rst cause the error.
> 
> Thanks for the report. I'm going to propose that we fix this by
> marking the ascii art as comment:
> 
> diff --git a/Documentation/driver-api/usb/typec.rst 
> b/Documentation/driver-api/usb/typec.rst
> index feb31946490b..972c11bf4141 100644
> --- a/Documentation/driver-api/usb/typec.rst
> +++ b/Documentation/driver-api/usb/typec.rst
> @@ -212,7 +212,7 @@ port drivers can use USB Role Class API with those.
> 
> Illustration of the muxes behind a connector that supports an alternate 
> mode:
> 
> - 
> +..   
> |   Connector  |
> 
>| |
> 
> I hope that works.
 
 Try it and see!  :)
>>> 
>>> It will fix this issue. I was just wondering if use of ascii art is
>>> acceptable in general with the .rst files? But then again, why
>>> wouldn't it be.

[...]

> I was propsed to use something called "Literal Block" with ascii art.
> http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#literal-blocks

about *ASCII-art*: see fix from Jani ...

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg19302.html

where the '::' is a short-markup for a literal-block.


>> There are ways to do this, look at how the v4l2 and I think the drm
>> subsystems handle ascii art such that "real" drawings end up being
>> produced.
> 
> Thanks. I did not actually find anything else except use of tables and
> code-blocks in v4l documentation. Is that what you were referring?

If it is about *figures*: we have a directive named 'kernel-figure',
which is a full replacement of the 'figure' directive from Sphinx-Doc.
In addition it supports *inline* SVG and DOT markups. Read:

 https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#figures-images

-- Markus --

Re: [PATCH] scripts: kernel_doc: fixup reporting of function identifiers

[Markus Heiser]

> Am 16.02.2018 um 15:56 schrieb Jonathan Corbet :
> 
> On Tue, 13 Feb 2018 13:31:46 +0200
> Mike Rapoport  wrote:
> 
>> When function description includes brackets after the function name as
>> suggested by Documentation/doc-guide/kernel-doc, the kernel-doc script
>> omits the function name from "Scanning doc for" report.
>> Extending match for identifier name with optional brackets fixes this
>> issue.
> 
> So let me channel akpm here and ask: what are the user-visible effects of
> this problem?  I ask because applying it doesn't make any difference in
> the "make htmldocs" output here.  So I don't understand why you're
> wanting to make this change.

Use kernel-doc -v and take a look on the info-messages.

In Documentation/doc-guide/kernel-doc we recommend to use

/**
 * foo() - lorem ipsum

to tag functions, but if you do so, the info message is broken,
the function name is missed at the end of the message:

 ../test123.c:2: info: Scanning doc for  

Here is the similar patch to kernel-doc python version:

 
https://github.com/return42/linuxdoc/commit/84d665df34cc3c9908a4b8ce0fdf193165a1ffa4

@Mike: thanks!

--Markus--

Re: [PATCH] scripts: kernel_doc: fixup reporting of function identifiers

[Markus Heiser]

> Am 16.02.2018 um 15:56 schrieb Jonathan Corbet :
> 
> On Tue, 13 Feb 2018 13:31:46 +0200
> Mike Rapoport  wrote:
> 
>> When function description includes brackets after the function name as
>> suggested by Documentation/doc-guide/kernel-doc, the kernel-doc script
>> omits the function name from "Scanning doc for" report.
>> Extending match for identifier name with optional brackets fixes this
>> issue.
> 
> So let me channel akpm here and ask: what are the user-visible effects of
> this problem?  I ask because applying it doesn't make any difference in
> the "make htmldocs" output here.  So I don't understand why you're
> wanting to make this change.

Use kernel-doc -v and take a look on the info-messages.

In Documentation/doc-guide/kernel-doc we recommend to use

/**
 * foo() - lorem ipsum

to tag functions, but if you do so, the info message is broken,
the function name is missed at the end of the message:

 ../test123.c:2: info: Scanning doc for  

Here is the similar patch to kernel-doc python version:

 
https://github.com/return42/linuxdoc/commit/84d665df34cc3c9908a4b8ce0fdf193165a1ffa4

@Mike: thanks!

--Markus--

Re: [PATCH 4/6] scripts: kernel-doc: support in-line comments on nested structs/unions

[Markus Heiser]

> Am 16.02.2018 um 15:56 schrieb Mauro Carvalho Chehab 
> <mche...@s-opensource.com>:
> 
> Em Fri, 16 Feb 2018 15:52:33 +0100
> Markus Heiser <markus.hei...@darmarit.de> escreveu:
> 
>>> Am 16.02.2018 um 14:48 schrieb Mauro Carvalho Chehab 
>>> <mche...@s-opensource.com>:
>>> 
>>> The parser at kernel-doc rejects names with dots in the middle.
>>> Fix it, in order to support nested structs/unions.
>>> 
>>> Tested-by: Jani Nikula <jani.nik...@intel.com>
>>> Signed-off-by: Mauro Carvalho Chehab <mche...@s-opensource.com>
>>> ---
>>> scripts/kernel-doc | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>> 
>>> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
>>> index fee8952037b1..06d7f3f2c094 100755
>>> --- a/scripts/kernel-doc
>>> +++ b/scripts/kernel-doc
>>> @@ -363,7 +363,7 @@ my $doc_sect = $doc_com .
>>> my $doc_content = $doc_com_body . '(.*)';
>>> my $doc_block = $doc_com . 'DOC:\s*(.*)?';
>>> my $doc_inline_start = '^\s*/\*\*\s*$';
>>> -my $doc_inline_sect = '\s*\*\s*(@[\w\s]+):(.*)';
>>> +my $doc_inline_sect = '\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)';  
>> 
>> Thanks!
>> 
>> FWIW: added similar patch to python variant of kernel-doc:
>> 
>> https://github.com/return42/linuxdoc/commit/5c5da9a
>> 
>> rendered example:
>> 
>>  
>> https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct
> 
> On a quick look, on your example, bar2.barbar description looks different
> than what we get from the perl version.
> 
> There, it generates it as:
> 
> ``bar2.barbar``
>  Description for **barbar** inside **foo.bar2**

very attentive, thanks a lot!

When I implemented support for nested data types, I missed
to fix the highlighting pattern of those.

  https://github.com/return42/linuxdoc/commit/4b43f419

rendered: 
https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct

-- Markus --

Re: [PATCH 4/6] scripts: kernel-doc: support in-line comments on nested structs/unions

[Markus Heiser]

> Am 16.02.2018 um 15:56 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Fri, 16 Feb 2018 15:52:33 +0100
> Markus Heiser  escreveu:
> 
>>> Am 16.02.2018 um 14:48 schrieb Mauro Carvalho Chehab 
>>> :
>>> 
>>> The parser at kernel-doc rejects names with dots in the middle.
>>> Fix it, in order to support nested structs/unions.
>>> 
>>> Tested-by: Jani Nikula 
>>> Signed-off-by: Mauro Carvalho Chehab 
>>> ---
>>> scripts/kernel-doc | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>> 
>>> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
>>> index fee8952037b1..06d7f3f2c094 100755
>>> --- a/scripts/kernel-doc
>>> +++ b/scripts/kernel-doc
>>> @@ -363,7 +363,7 @@ my $doc_sect = $doc_com .
>>> my $doc_content = $doc_com_body . '(.*)';
>>> my $doc_block = $doc_com . 'DOC:\s*(.*)?';
>>> my $doc_inline_start = '^\s*/\*\*\s*$';
>>> -my $doc_inline_sect = '\s*\*\s*(@[\w\s]+):(.*)';
>>> +my $doc_inline_sect = '\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)';  
>> 
>> Thanks!
>> 
>> FWIW: added similar patch to python variant of kernel-doc:
>> 
>> https://github.com/return42/linuxdoc/commit/5c5da9a
>> 
>> rendered example:
>> 
>>  
>> https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct
> 
> On a quick look, on your example, bar2.barbar description looks different
> than what we get from the perl version.
> 
> There, it generates it as:
> 
> ``bar2.barbar``
>  Description for **barbar** inside **foo.bar2**

very attentive, thanks a lot!

When I implemented support for nested data types, I missed
to fix the highlighting pattern of those.

  https://github.com/return42/linuxdoc/commit/4b43f419

rendered: 
https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct

-- Markus --

Re: [PATCH 4/6] scripts: kernel-doc: support in-line comments on nested structs/unions

[Markus Heiser]

> Am 16.02.2018 um 14:48 schrieb Mauro Carvalho Chehab 
> :
> 
> The parser at kernel-doc rejects names with dots in the middle.
> Fix it, in order to support nested structs/unions.
> 
> Tested-by: Jani Nikula 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> scripts/kernel-doc | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index fee8952037b1..06d7f3f2c094 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -363,7 +363,7 @@ my $doc_sect = $doc_com .
> my $doc_content = $doc_com_body . '(.*)';
> my $doc_block = $doc_com . 'DOC:\s*(.*)?';
> my $doc_inline_start = '^\s*/\*\*\s*$';
> -my $doc_inline_sect = '\s*\*\s*(@[\w\s]+):(.*)';
> +my $doc_inline_sect = '\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)';

Thanks!

FWIW: added similar patch to python variant of kernel-doc:

https://github.com/return42/linuxdoc/commit/5c5da9a

rendered example:

  
https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct

-- Markus --

Re: [PATCH 4/6] scripts: kernel-doc: support in-line comments on nested structs/unions

[Markus Heiser]

> Am 16.02.2018 um 14:48 schrieb Mauro Carvalho Chehab 
> :
> 
> The parser at kernel-doc rejects names with dots in the middle.
> Fix it, in order to support nested structs/unions.
> 
> Tested-by: Jani Nikula 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> scripts/kernel-doc | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index fee8952037b1..06d7f3f2c094 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -363,7 +363,7 @@ my $doc_sect = $doc_com .
> my $doc_content = $doc_com_body . '(.*)';
> my $doc_block = $doc_com . 'DOC:\s*(.*)?';
> my $doc_inline_start = '^\s*/\*\*\s*$';
> -my $doc_inline_sect = '\s*\*\s*(@[\w\s]+):(.*)';
> +my $doc_inline_sect = '\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)';

Thanks!

FWIW: added similar patch to python variant of kernel-doc:

https://github.com/return42/linuxdoc/commit/5c5da9a

rendered example:

  
https://return42.github.io/linuxdoc/linuxdoc-howto/all-in-a-tumble.html#struct-my-long-struct

-- Markus --

Re: [PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Markus Heiser]

> Am 07.02.2018 um 18:26 schrieb Jonathan Corbet :
> 
> It can be useful to put code snippets into kerneldoc comments; that can be
> done with the "::" operator at the end of a line like this::
> 
>  if (desperate)
>  run_in_circles();
> 
> kernel-doc currently fails to understand these literal blocks and applies
> its normal markup to them, which is then treated as literal by sphinx.  The
> result is unsightly markup instead of a useful code snippet.
> 
> Apply a hack to the output code to recognize literal blocks and avoid
> performing any special markup on them.  It's ugly, but that means it fits
> in well with the rest of the script.
> 
> Signed-off-by: Jonathan Corbet 


[...]

FYI; added similar patch to python version of kernel-doc:

 https://github.com/return42/linuxdoc/commit/3205b8a68

may you like to add regexpr for code-block directive to your patch (jani 
mentioned already).

-- Markus --

Re: [PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Markus Heiser]

> Am 07.02.2018 um 18:26 schrieb Jonathan Corbet :
> 
> It can be useful to put code snippets into kerneldoc comments; that can be
> done with the "::" operator at the end of a line like this::
> 
>  if (desperate)
>  run_in_circles();
> 
> kernel-doc currently fails to understand these literal blocks and applies
> its normal markup to them, which is then treated as literal by sphinx.  The
> result is unsightly markup instead of a useful code snippet.
> 
> Apply a hack to the output code to recognize literal blocks and avoid
> performing any special markup on them.  It's ugly, but that means it fits
> in well with the rest of the script.
> 
> Signed-off-by: Jonathan Corbet 


[...]

FYI; added similar patch to python version of kernel-doc:

 https://github.com/return42/linuxdoc/commit/3205b8a68

may you like to add regexpr for code-block directive to your patch (jani 
mentioned already).

-- Markus --

Re: [RFC] doc: fix code snippet build warnings

[Markus Heiser]

> Am 16.01.2018 um 11:22 schrieb Jani Nikula :
> 
> On Wed, 10 Jan 2018, Jonathan Corbet  wrote:
>> On Wed, 10 Jan 2018 15:04:53 +1100
>> "Tobin C. Harding"  wrote:
>> 
>>> Posting as RFC in the hope that someone knows how to massage sphinx
>>> correctly to fix this patch.
>>> 
>>> Currently function kernel-doc contains a multi-line code snippet. This
>>> is causing sphinx to emit 5 build warnings
>>> 
>>> WARNING: Unexpected indentation.
>>> WARNING: Unexpected indentation.
>>> WARNING: Block quote ends without a blank line; unexpected unindent.
>>> WARNING: Block quote ends without a blank line; unexpected unindent.
>>> WARNING: Inline literal start-string without end-string.
>>> 
>>> And the snippet is not rendering correctly in HTML.
>>> 
>>> We can stop shpinx complaining by using '::' instead of the currently
>>> used '``' however this still does not render correctly in HTML. The
>>> rendering is [arguably] better but still incorrect. Sphinx renders two
>>> function calls thus:
>>> 
>>> :c:func:`rcu_read_lock()`;
>>> 
>>> The rest of the snippet does however have correct spacing.
>> 
>> The behavior when `` was used is not surprising, that really just does a
>> font change.  Once you went with a literal block (with "::") though, the
>> situation changes a bit.  That really should work.
>> 
>> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
>> in kernel-doc.  Once we go into the literal mode, it shouldn't be
>> screwing around with the text anymore.  Of course, kernel-doc doesn't
>> understand enough RST to know that.  I'm a little nervous about trying to
>> teach it more, but maybe we have to do that; we should certainly be able
>> to put code snippets into the docs and have them come through unmolested.
> 
> I think eventually the Right Thing would be to move most of kernel-doc's
> mucking of the comments (i.e. highlights) into kernel-doc the Sphinx
> extension. I've toyed with the idea, but it's not as trivial as I would
> like it to be.
> 
> Basically it involves flagging all the kernel-doc nodes during the read
> phase, and registering handlers to post process the doctrees. At that
> stage, it's no longer simple regexp replaces, it's replacing doctree
> nodes with ones that have reference nodes within them. It's more
> complicated,

Hi Jani, my thoughts about:  

1.) The reST syntax [1] (the parser part) is not *extendable*, we
can only add new roles [2] or directives.

2.) the kernel-doc syntax is weak and ambiguous. This
remains mainly in tagging only with a start-tag or only with a end-tag 
e.g:

* sectioning:   "Return:" --> end-tag just ":"
* fields:   "@arg1:"  --> better
* highlight/ref: start tag [@%$&] / no end tag

even if I don't know how (1.), if the highlight-syntax becomes a part
of the reST syntax we have to quote [@%$&] elsewhere. This will break
with valid reST documents which is unacceptable.

Thats why I think we need to pre-parse highlighting in the kernel-doc
Perl script, even if it is a bit hackish (how should it be otherwise,
if the syntax is already hackish). I haven't had the time to implement
such a highlight parser right now, but this would be the first shot of
mine. 

An alternative might be to use roles [2] but this means we have to
break with the the kernel-doc (highlight) syntax which is IMO also
unacceptable ... tricky situation :o

[1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
[2] http://docutils.sourceforge.net/docs/howto/rst-roles.html


> but at that stage we can ignore stuff that should stay
> verbatim. I think it's also possible to check that the reference targets
> actually exist. In short, at that phase we have all the knowledge about
> the rst structure, parsed by Sphinx, and we don't have to duplicate that
> knowledge in kernel-doc the perl script.
> 
> Note that this has nothing to do with swithing to Python based
> kernel-doc suggested by Markus previously.

;)

-- Markus --

Re: [RFC] doc: fix code snippet build warnings

[Markus Heiser]

> Am 16.01.2018 um 11:22 schrieb Jani Nikula :
> 
> On Wed, 10 Jan 2018, Jonathan Corbet  wrote:
>> On Wed, 10 Jan 2018 15:04:53 +1100
>> "Tobin C. Harding"  wrote:
>> 
>>> Posting as RFC in the hope that someone knows how to massage sphinx
>>> correctly to fix this patch.
>>> 
>>> Currently function kernel-doc contains a multi-line code snippet. This
>>> is causing sphinx to emit 5 build warnings
>>> 
>>> WARNING: Unexpected indentation.
>>> WARNING: Unexpected indentation.
>>> WARNING: Block quote ends without a blank line; unexpected unindent.
>>> WARNING: Block quote ends without a blank line; unexpected unindent.
>>> WARNING: Inline literal start-string without end-string.
>>> 
>>> And the snippet is not rendering correctly in HTML.
>>> 
>>> We can stop shpinx complaining by using '::' instead of the currently
>>> used '``' however this still does not render correctly in HTML. The
>>> rendering is [arguably] better but still incorrect. Sphinx renders two
>>> function calls thus:
>>> 
>>> :c:func:`rcu_read_lock()`;
>>> 
>>> The rest of the snippet does however have correct spacing.
>> 
>> The behavior when `` was used is not surprising, that really just does a
>> font change.  Once you went with a literal block (with "::") though, the
>> situation changes a bit.  That really should work.
>> 
>> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
>> in kernel-doc.  Once we go into the literal mode, it shouldn't be
>> screwing around with the text anymore.  Of course, kernel-doc doesn't
>> understand enough RST to know that.  I'm a little nervous about trying to
>> teach it more, but maybe we have to do that; we should certainly be able
>> to put code snippets into the docs and have them come through unmolested.
> 
> I think eventually the Right Thing would be to move most of kernel-doc's
> mucking of the comments (i.e. highlights) into kernel-doc the Sphinx
> extension. I've toyed with the idea, but it's not as trivial as I would
> like it to be.
> 
> Basically it involves flagging all the kernel-doc nodes during the read
> phase, and registering handlers to post process the doctrees. At that
> stage, it's no longer simple regexp replaces, it's replacing doctree
> nodes with ones that have reference nodes within them. It's more
> complicated,

Hi Jani, my thoughts about:  

1.) The reST syntax [1] (the parser part) is not *extendable*, we
can only add new roles [2] or directives.

2.) the kernel-doc syntax is weak and ambiguous. This
remains mainly in tagging only with a start-tag or only with a end-tag 
e.g:

* sectioning:   "Return:" --> end-tag just ":"
* fields:   "@arg1:"  --> better
* highlight/ref: start tag [@%$&] / no end tag

even if I don't know how (1.), if the highlight-syntax becomes a part
of the reST syntax we have to quote [@%$&] elsewhere. This will break
with valid reST documents which is unacceptable.

Thats why I think we need to pre-parse highlighting in the kernel-doc
Perl script, even if it is a bit hackish (how should it be otherwise,
if the syntax is already hackish). I haven't had the time to implement
such a highlight parser right now, but this would be the first shot of
mine. 

An alternative might be to use roles [2] but this means we have to
break with the the kernel-doc (highlight) syntax which is IMO also
unacceptable ... tricky situation :o

[1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
[2] http://docutils.sourceforge.net/docs/howto/rst-roles.html


> but at that stage we can ignore stuff that should stay
> verbatim. I think it's also possible to check that the reference targets
> actually exist. In short, at that phase we have all the knowledge about
> the rst structure, parsed by Sphinx, and we don't have to duplicate that
> knowledge in kernel-doc the perl script.
> 
> Note that this has nothing to do with swithing to Python based
> kernel-doc suggested by Markus previously.

;)

-- Markus --

Re: [PATCH] linux-next: docs-rst: Fix typos in kfigure.py

[Markus Heiser]

> Am 11.01.2018 um 12:00 schrieb Masanari Iida :
> 
> This patch fixes some spelling typos found in kfigure.py

FYI: I copied the patch to the linuxdoc project [1] / Thanks!

[1] https://github.com/return42/linuxdoc/commit/41d228f

-- Markus --

Re: [PATCH] linux-next: docs-rst: Fix typos in kfigure.py

[Markus Heiser]

> Am 11.01.2018 um 12:00 schrieb Masanari Iida :
> 
> This patch fixes some spelling typos found in kfigure.py

FYI: I copied the patch to the linuxdoc project [1] / Thanks!

[1] https://github.com/return42/linuxdoc/commit/41d228f

-- Markus --

Re: [PATCH v3 1/1] runchecks: Generalize make C={1,2} to support multiple checkers

[Markus Heiser]

> Am 05.01.2018 um 15:30 schrieb Jani Nikula :
> 
> On Thu, 04 Jan 2018, Knut Omang  wrote:
>> On Thu, 2018-01-04 at 17:50 +0200, Jani Nikula wrote:

[...]

>> Hmm - I have been burnt by the use of unstable interfaces in Python before,
>> when I needed it to work on a (Linux) system with Python v.2.6.x only
>> - argparse was introduced in v.2.7. and alternative choices are not 
>> at all clear to me, see for instance:
>> 
>>  https://dmerej.info/blog/post/docopt-v-argparse/
>> 
>> If this program was part of a "standalone" python project with a well 
>> defined python
>> environment, I would probably have used argparse, which I have used in other 
>> projects.
>> 
>> In fact I hesitated even to use python for this, because of fear of 
>> versioning issues..
>> When I was tempted anyway, and after looking at the existing examples in 
>> scripts/ 
>> ruling out python v.3.x, it felt safer to stay with the bare minimum of 
>> module 
>> features for this simple logic.
>> 
>> I do feel confident that the benefits of python for this outweighs the 
>> drawbacks
>> compared to my initial shell script implementation, or using perl or even C.
>> 
>> Further advice on this appreciated,
> 
> Again, I can only offer my opinion of requiring Python v2.7 and using
> argparse, but it doesn't carry much weight. Up to the kbuild
> maintainers.

FYI: Py2.6 support has ended and Py2 expected to be end in 2020, the
countdown is running ;)

  https://pythonclock.org/

If you wan't a command-line parser which is "builtin" use argparse,
this is what I do mostly. If you are able to use requirements from
outside use click [1]. For more infos read e.g. [2]. Docopt [3] has
its charm but I would prefer the decorators from click.

[1] http://click.pocoo.org/5/
[2] 
https://realpython.com/blog/python/comparing-python-command-line-parsing-libraries-argparse-docopt-click/
[3] https://github.com/docopt

-- Markus --

Re: [PATCH v3 1/1] runchecks: Generalize make C={1,2} to support multiple checkers

[Markus Heiser]

> Am 05.01.2018 um 15:30 schrieb Jani Nikula :
> 
> On Thu, 04 Jan 2018, Knut Omang  wrote:
>> On Thu, 2018-01-04 at 17:50 +0200, Jani Nikula wrote:

[...]

>> Hmm - I have been burnt by the use of unstable interfaces in Python before,
>> when I needed it to work on a (Linux) system with Python v.2.6.x only
>> - argparse was introduced in v.2.7. and alternative choices are not 
>> at all clear to me, see for instance:
>> 
>>  https://dmerej.info/blog/post/docopt-v-argparse/
>> 
>> If this program was part of a "standalone" python project with a well 
>> defined python
>> environment, I would probably have used argparse, which I have used in other 
>> projects.
>> 
>> In fact I hesitated even to use python for this, because of fear of 
>> versioning issues..
>> When I was tempted anyway, and after looking at the existing examples in 
>> scripts/ 
>> ruling out python v.3.x, it felt safer to stay with the bare minimum of 
>> module 
>> features for this simple logic.
>> 
>> I do feel confident that the benefits of python for this outweighs the 
>> drawbacks
>> compared to my initial shell script implementation, or using perl or even C.
>> 
>> Further advice on this appreciated,
> 
> Again, I can only offer my opinion of requiring Python v2.7 and using
> argparse, but it doesn't carry much weight. Up to the kbuild
> maintainers.

FYI: Py2.6 support has ended and Py2 expected to be end in 2020, the
countdown is running ;)

  https://pythonclock.org/

If you wan't a command-line parser which is "builtin" use argparse,
this is what I do mostly. If you are able to use requirements from
outside use click [1]. For more infos read e.g. [2]. Docopt [3] has
its charm but I would prefer the decorators from click.

[1] http://click.pocoo.org/5/
[2] 
https://realpython.com/blog/python/comparing-python-command-line-parsing-libraries-argparse-docopt-click/
[3] https://github.com/docopt

-- Markus --

Re: [PATCH] doc: memory-barriers: reStructure Text

[Markus Heiser]

> Am 05.01.2018 um 04:52 schrieb afzal mohammed :

[...]

> Initially i was sceptical of rst & once instead of hitting the fly,
> hit "make htmldocs" on the keyboard :), and the opinion about it was
> changed.

Yes, I understand that some of us have a (reasonable) doubt about
the reST markup.  It is not perfect in any matter, e.g. I don't like
the ``monospace`` markup.  But this is my home opinion.

My hope is, that those of us who have a doubt give reST
a chance ... it is a compromise, not as bad as you might
think first ... your cooperation and your criticism is
needed and welcome. Please let me invite you / read on:

There are other plain-text markups e.g. AsciiDoc or Markdown. The
reST markup and the Sphinx-builder is a compromise from a evaluation
in 2016 (see linux-doc ML subject "muddying the waters" [1]).

Jani wrote an article about the evaluation and it results [2]. And
there are other articles documenting all the various aspects.

- A report from the documentation maintainer [3]
- Kernel documentation with Sphinx, part 2: how it works [4]
- Kernel documentation update [5]

To summarize it with my words:

The old DocBook-based toolchain was hard to maintain and of course
who want's to write XML? A consistent plain-text markup for
articles in /Documentation/* and source-code comments (kernel-doc)
was needed.

The markup: IMO reST wins that race, because it has a extendable
markup specification while others plain-text markups like Markdown
have various (HTML) builders with various markup dialects[7] (which
is more a mess than a definition).

The builder: IMO Sphinx-Doc wins that race, since it is (well)
maintained, widely used and has a interface for extensions.
I.e. the one extension we wrote: 'kernel-doc' to parse kernel-doc
comments from source code and include them in the articles.
Perspective: Sphinx-Doc also offers solutions we might use in the
future (e.g. building man-pages). Not to end in a mess, extensions
should be implemented cautiously and deliberately (be patient).

But that should not fool you; yes we have known problems with our
toolchain and it is not yet ;) perfect in any matter (e.g. the
highlighting in kernel-doc comments or the PDF generation or the
sphinx-doc versions shipped with various distributions or ..) 

Anyway, today we have more than before: The reST learning curve is
(compared to DocBook) not hard for newbees and our toolchain is
flexible for all the requirements wich might come up in the future.

IMO the actual challenge is the content and the organization
of the doc-tree and for this 

[1] 
https://www.mail-archive.com/search?q=muddying+the+waters=linux-doc%40vger.kernel.org
[2] https://lwn.net/Articles/692704/
[3] https://lwn.net/Articles/704613/
[4] https://lwn.net/Articles/692705/
[5] https://lwn.net/Articles/705224/
[6] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
[7] http://pandoc.org/MANUAL.html#markdown-variants

> It was easy to navigate through various docs & the realized
> that various topics (& many) were present (yes, it was there earlier
> also, but had to dive inside Documentation & search, while viewing the
> toplevel index.html made them standout). It was like earlier you had
> to go after docs, but now it was docs coming after you, that is my
> opinion.
> 
> Later while fighting with memory-barriers.txt, felt that it might be
> good for it as well as to be in that company.
> 
> And the readability as a text is not hurt as well.
> 
> It was thought that rst conversion could be done quickly, but since
> this was my first attempt with rst, had to put some effort to get a
> not so bad output, even if this patch dies, i am happy to have learnt
> rst conversion to some extent.

Thats exactly what I mean: give reST a chance :)

-- Markus --

Re: [PATCH] doc: memory-barriers: reStructure Text

[Markus Heiser]

> Am 05.01.2018 um 04:52 schrieb afzal mohammed :

[...]

> Initially i was sceptical of rst & once instead of hitting the fly,
> hit "make htmldocs" on the keyboard :), and the opinion about it was
> changed.

Yes, I understand that some of us have a (reasonable) doubt about
the reST markup.  It is not perfect in any matter, e.g. I don't like
the ``monospace`` markup.  But this is my home opinion.

My hope is, that those of us who have a doubt give reST
a chance ... it is a compromise, not as bad as you might
think first ... your cooperation and your criticism is
needed and welcome. Please let me invite you / read on:

There are other plain-text markups e.g. AsciiDoc or Markdown. The
reST markup and the Sphinx-builder is a compromise from a evaluation
in 2016 (see linux-doc ML subject "muddying the waters" [1]).

Jani wrote an article about the evaluation and it results [2]. And
there are other articles documenting all the various aspects.

- A report from the documentation maintainer [3]
- Kernel documentation with Sphinx, part 2: how it works [4]
- Kernel documentation update [5]

To summarize it with my words:

The old DocBook-based toolchain was hard to maintain and of course
who want's to write XML? A consistent plain-text markup for
articles in /Documentation/* and source-code comments (kernel-doc)
was needed.

The markup: IMO reST wins that race, because it has a extendable
markup specification while others plain-text markups like Markdown
have various (HTML) builders with various markup dialects[7] (which
is more a mess than a definition).

The builder: IMO Sphinx-Doc wins that race, since it is (well)
maintained, widely used and has a interface for extensions.
I.e. the one extension we wrote: 'kernel-doc' to parse kernel-doc
comments from source code and include them in the articles.
Perspective: Sphinx-Doc also offers solutions we might use in the
future (e.g. building man-pages). Not to end in a mess, extensions
should be implemented cautiously and deliberately (be patient).

But that should not fool you; yes we have known problems with our
toolchain and it is not yet ;) perfect in any matter (e.g. the
highlighting in kernel-doc comments or the PDF generation or the
sphinx-doc versions shipped with various distributions or ..) 

Anyway, today we have more than before: The reST learning curve is
(compared to DocBook) not hard for newbees and our toolchain is
flexible for all the requirements wich might come up in the future.

IMO the actual challenge is the content and the organization
of the doc-tree and for this 

[1] 
https://www.mail-archive.com/search?q=muddying+the+waters=linux-doc%40vger.kernel.org
[2] https://lwn.net/Articles/692704/
[3] https://lwn.net/Articles/704613/
[4] https://lwn.net/Articles/692705/
[5] https://lwn.net/Articles/705224/
[6] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
[7] http://pandoc.org/MANUAL.html#markdown-variants

> It was easy to navigate through various docs & the realized
> that various topics (& many) were present (yes, it was there earlier
> also, but had to dive inside Documentation & search, while viewing the
> toplevel index.html made them standout). It was like earlier you had
> to go after docs, but now it was docs coming after you, that is my
> opinion.
> 
> Later while fighting with memory-barriers.txt, felt that it might be
> good for it as well as to be in that company.
> 
> And the readability as a text is not hurt as well.
> 
> It was thought that rst conversion could be done quickly, but since
> this was my first attempt with rst, had to put some effort to get a
> not so bad output, even if this patch dies, i am happy to have learnt
> rst conversion to some extent.

Thats exactly what I mean: give reST a chance :)

-- Markus --

Re: [PATCH] doc: memory-barriers: reStructure Text

[Markus Heiser]

> Am 04.01.2018 um 04:59 schrieb afzal mohammed :
> 
> Hi,
> 
> On Thu, Jan 04, 2018 at 09:48:50AM +0800, Boqun Feng wrote:
> 
>>> The location chosen is "Documentation/kernel-hacking", i was unsure
>>> where this should reside & there was no .rst file in top-level directory
>>> "Documentation", so put it into one of the existing folder that seemed
>>> to me as not that unsuitable.
>>> 
>>> Other files refer to memory-barrier.txt, those also needs to be
>>> adjusted based on where .rst can reside.
> 
>> How do you plan to handle the external references? For example, the
>> following LWN articles has a link this file:
>> 
>>  https://lwn.net/Articles/718628/
>> 
>> And changing the name and/or location will break that link, AFAIK.
> 
> If necessary to handle these, symlink might help here i believe.

IMO symlinks are mostly ending in a mess, URLs are never stable.
There is a 

 https://www.kernel.org/doc/html/latest/objects.inv

to handle such requirements. Take a look at *intersphinx* :

 http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

to see how it works:  Each Sphinx HTML build creates a file named objects.inv 
that
contains a mapping from object names to URIs relative to the HTML set’s root.

This means articles from external (like lwn articles) has to be recompiled.
Not perfect, but a first solution. 

> Am 04.01.2018 um 00:48 schrieb Peter Zijlstra :
> 
> So I hate this rst crap with a passion, so NAK from me.

I really like them, factually valuable comments .. please
express your concern so that we have a chance to move on.

> Upon trying to understand memory-barriers.txt, i felt that it might be
> better to have it in PDF/HTML format, thus attempted to convert it to
> rst. And i see it not being welcomed, hence shelving the conversion.

I think that's a pity.

-- Markus --

Re: [PATCH] doc: memory-barriers: reStructure Text

[Markus Heiser]

> Am 04.01.2018 um 04:59 schrieb afzal mohammed :
> 
> Hi,
> 
> On Thu, Jan 04, 2018 at 09:48:50AM +0800, Boqun Feng wrote:
> 
>>> The location chosen is "Documentation/kernel-hacking", i was unsure
>>> where this should reside & there was no .rst file in top-level directory
>>> "Documentation", so put it into one of the existing folder that seemed
>>> to me as not that unsuitable.
>>> 
>>> Other files refer to memory-barrier.txt, those also needs to be
>>> adjusted based on where .rst can reside.
> 
>> How do you plan to handle the external references? For example, the
>> following LWN articles has a link this file:
>> 
>>  https://lwn.net/Articles/718628/
>> 
>> And changing the name and/or location will break that link, AFAIK.
> 
> If necessary to handle these, symlink might help here i believe.

IMO symlinks are mostly ending in a mess, URLs are never stable.
There is a 

 https://www.kernel.org/doc/html/latest/objects.inv

to handle such requirements. Take a look at *intersphinx* :

 http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

to see how it works:  Each Sphinx HTML build creates a file named objects.inv 
that
contains a mapping from object names to URIs relative to the HTML set’s root.

This means articles from external (like lwn articles) has to be recompiled.
Not perfect, but a first solution. 

> Am 04.01.2018 um 00:48 schrieb Peter Zijlstra :
> 
> So I hate this rst crap with a passion, so NAK from me.

I really like them, factually valuable comments .. please
express your concern so that we have a chance to move on.

> Upon trying to understand memory-barriers.txt, i felt that it might be
> better to have it in PDF/HTML format, thus attempted to convert it to
> rst. And i see it not being welcomed, hence shelving the conversion.

I think that's a pity.

-- Markus --

Re: [PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

> Am 12.12.2017 um 16:23 schrieb Daniel Vetter :
> 
>>> Thanks for your patch, but similar fix is already merged here [1]
>>> 
>>> Michal
>>> 
>>> [1] 
>>> https://cgit.freedesktop.org/drm-tip/commit/?id=006c23327f8de8575508c458131b304188d426f7
>> 
>> 
>> Thanks for pointing out. I miss the ":doc: GuC-specific firmware loader"
>> fix in that patch (doc section was removed in d9e2e0143c).  Can you or
>> someone else from @intel fix this also?
> 
> I've rebased the patch from you to only take that remaining part.
> 
> Thanks for fixing this.
> -Daniel

Thanks for picking this up [1].

[1] https://cgit.freedesktop.org/drm/drm-intel/commit/?id=0132a1a5

-- Markus --

Re: [PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

> Am 12.12.2017 um 16:23 schrieb Daniel Vetter :
> 
>>> Thanks for your patch, but similar fix is already merged here [1]
>>> 
>>> Michal
>>> 
>>> [1] 
>>> https://cgit.freedesktop.org/drm-tip/commit/?id=006c23327f8de8575508c458131b304188d426f7
>> 
>> 
>> Thanks for pointing out. I miss the ":doc: GuC-specific firmware loader"
>> fix in that patch (doc section was removed in d9e2e0143c).  Can you or
>> someone else from @intel fix this also?
> 
> I've rebased the patch from you to only take that remaining part.
> 
> Thanks for fixing this.
> -Daniel

Thanks for picking this up [1].

[1] https://cgit.freedesktop.org/drm/drm-intel/commit/?id=0132a1a5

-- Markus --

Re: [PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

> Am 12.12.2017 um 13:05 schrieb Michal Wajdeczko <michal.wajdec...@intel.com>:
> 
> On Tue, 12 Dec 2017 12:38:37 +0100, Markus Heiser <markus.hei...@darmarit.de> 
> wrote:
> 
>> With commit d9e2e0143c the 'GuC-specific firmware loader' doc
>> section was removed from intel_guc_loader.c without a
>> replacement.  So lets remove it from the Kernel-doc::
>> 
>>  .. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
>> :doc: GuC-specific firmware loader
>> 
>> With commit e8668bbcb0 intel_guc_loader.c was renamed to to
>> intel_guc_fw.c and to name just one, intel_guc_init_hw() was
>> renamed to intel_guc_fw_upload(). Since we get errors in the
>> Sphinx build like:
>> 
>> - Error: Cannot open file ./drivers/gpu/drm/i915/intel_guc_loader.c
>> 
>> Change the kernel-doc directive from intel_guc_loader.c to
>> intel_guc_fw.c
>> 
>> Signed-off-by: Markus Heiser <markus.hei...@darmarit.de>
>> ---
> 
> Thanks for your patch, but similar fix is already merged here [1]
> 
> Michal
> 
> [1] 
> https://cgit.freedesktop.org/drm-tip/commit/?id=006c23327f8de8575508c458131b304188d426f7


Thanks for pointing out. I miss the ":doc: GuC-specific firmware loader"
fix in that patch (doc section was removed in d9e2e0143c).  Can you or
someone else from @intel fix this also?

Thanks!

 -- Markus --



> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

> Am 12.12.2017 um 13:05 schrieb Michal Wajdeczko :
> 
> On Tue, 12 Dec 2017 12:38:37 +0100, Markus Heiser  
> wrote:
> 
>> With commit d9e2e0143c the 'GuC-specific firmware loader' doc
>> section was removed from intel_guc_loader.c without a
>> replacement.  So lets remove it from the Kernel-doc::
>> 
>>  .. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
>> :doc: GuC-specific firmware loader
>> 
>> With commit e8668bbcb0 intel_guc_loader.c was renamed to to
>> intel_guc_fw.c and to name just one, intel_guc_init_hw() was
>> renamed to intel_guc_fw_upload(). Since we get errors in the
>> Sphinx build like:
>> 
>> - Error: Cannot open file ./drivers/gpu/drm/i915/intel_guc_loader.c
>> 
>> Change the kernel-doc directive from intel_guc_loader.c to
>> intel_guc_fw.c
>> 
>> Signed-off-by: Markus Heiser 
>> ---
> 
> Thanks for your patch, but similar fix is already merged here [1]
> 
> Michal
> 
> [1] 
> https://cgit.freedesktop.org/drm-tip/commit/?id=006c23327f8de8575508c458131b304188d426f7


Thanks for pointing out. I miss the ":doc: GuC-specific firmware loader"
fix in that patch (doc section was removed in d9e2e0143c).  Can you or
someone else from @intel fix this also?

Thanks!

 -- Markus --



> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

With commit d9e2e0143c the 'GuC-specific firmware loader' doc
section was removed from intel_guc_loader.c without a
replacement.  So lets remove it from the Kernel-doc::

  .. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
 :doc: GuC-specific firmware loader

With commit e8668bbcb0 intel_guc_loader.c was renamed to to
intel_guc_fw.c and to name just one, intel_guc_init_hw() was
renamed to intel_guc_fw_upload(). Since we get errors in the
Sphinx build like:

- Error: Cannot open file ./drivers/gpu/drm/i915/intel_guc_loader.c

Change the kernel-doc directive from intel_guc_loader.c to
intel_guc_fw.c

Signed-off-by: Markus Heiser <markus.hei...@darmarit.de>
---
 Documentation/gpu/i915.rst | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/Documentation/gpu/i915.rst b/Documentation/gpu/i915.rst
index 2e7ee03..e94d3ac 100644
--- a/Documentation/gpu/i915.rst
+++ b/Documentation/gpu/i915.rst
@@ -341,10 +341,7 @@ GuC
 GuC-specific firmware loader
 
 
-.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
-   :doc: GuC-specific firmware loader
-
-.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
+.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_fw.c
:internal:
 
 GuC-based command submission
-- 
2.7.4

[PATCH] docs: fix, intel_guc_loader.c has been moved to intel_guc_fw.c

[Markus Heiser]

With commit d9e2e0143c the 'GuC-specific firmware loader' doc
section was removed from intel_guc_loader.c without a
replacement.  So lets remove it from the Kernel-doc::

  .. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
 :doc: GuC-specific firmware loader

With commit e8668bbcb0 intel_guc_loader.c was renamed to to
intel_guc_fw.c and to name just one, intel_guc_init_hw() was
renamed to intel_guc_fw_upload(). Since we get errors in the
Sphinx build like:

- Error: Cannot open file ./drivers/gpu/drm/i915/intel_guc_loader.c

Change the kernel-doc directive from intel_guc_loader.c to
intel_guc_fw.c

Signed-off-by: Markus Heiser 
---
 Documentation/gpu/i915.rst | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/Documentation/gpu/i915.rst b/Documentation/gpu/i915.rst
index 2e7ee03..e94d3ac 100644
--- a/Documentation/gpu/i915.rst
+++ b/Documentation/gpu/i915.rst
@@ -341,10 +341,7 @@ GuC
 GuC-specific firmware loader
 
 
-.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
-   :doc: GuC-specific firmware loader
-
-.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_loader.c
+.. kernel-doc:: drivers/gpu/drm/i915/intel_guc_fw.c
:internal:
 
 GuC-based command submission
-- 
2.7.4

Re: [PATCH v2] doc: convert printk-formats.txt to rst

[Markus Heiser]

> Am 07.12.2017 um 06:49 schrieb Tobin C. Harding :
> 
> Documentation/printk-formats.txt is a candidate for conversion to
> ReStructuredText format. Some effort has already been made to do this
> conversion even thought the suffix is currently .txt
> 
[...]
> 
> Signed-off-by: Tobin C. Harding 
> ---
> 
> v2:
> - Revert to use ASCII table.
> - Implement (or revert) changes as suggested by Randy Dunlap.
> - Change file location to core-api/ (inc required index changes) 
> - Remove some of the double back ticks.
> 
> Last two suggested by Jonathan Corbet.
> 

Hm, can't apply v2 on Jon's docs-next .. is the v2 patch on top of your v1?

If so, please take a look at our fabulously ;) documentation ..

 https://www.kernel.org/doc/html/latest/process/submitting-patches.html

"""
When you submit or resubmit a patch or patch series, include the complete patch 
description and justification for it. Don’t just say that this is version N of 
the patch (series). Don’t expect the subsystem maintainer to refer back to 
earlier patch versions or referenced URLs to find the patch description and put 
that into the patch. I.e., the patch (series) and its description should be 
self-contained. This benefits both the maintainers and reviewers. Some 
reviewers probably didn’t even receive earlier versions of the patch.

"""

-- Markus --

Re: [PATCH v2] doc: convert printk-formats.txt to rst

[Markus Heiser]

> Am 07.12.2017 um 06:49 schrieb Tobin C. Harding :
> 
> Documentation/printk-formats.txt is a candidate for conversion to
> ReStructuredText format. Some effort has already been made to do this
> conversion even thought the suffix is currently .txt
> 
[...]
> 
> Signed-off-by: Tobin C. Harding 
> ---
> 
> v2:
> - Revert to use ASCII table.
> - Implement (or revert) changes as suggested by Randy Dunlap.
> - Change file location to core-api/ (inc required index changes) 
> - Remove some of the double back ticks.
> 
> Last two suggested by Jonathan Corbet.
> 

Hm, can't apply v2 on Jon's docs-next .. is the v2 patch on top of your v1?

If so, please take a look at our fabulously ;) documentation ..

 https://www.kernel.org/doc/html/latest/process/submitting-patches.html

"""
When you submit or resubmit a patch or patch series, include the complete patch 
description and justification for it. Don’t just say that this is version N of 
the patch (series). Don’t expect the subsystem maintainer to refer back to 
earlier patch versions or referenced URLs to find the patch description and put 
that into the patch. I.e., the patch (series) and its description should be 
self-contained. This benefits both the maintainers and reviewers. Some 
reviewers probably didn’t even receive earlier versions of the patch.

"""

-- Markus --

Re: [PATCH] doc: convert printk-formats.txt to rst

[Markus Heiser]

> Am 06.12.2017 um 02:45 schrieb Tobin C. Harding :
> 
> Documentation/printk-formats.txt is a candidate for conversion to
> ReStructuredText format. Some effort has already been made to do this
> conversion even thought the suffix is currently .txt
> 
> Changes required to complete conversion
> 
> - Add double backticks where needed.
> - Add entry to Documentation/index.rst
> - Use flat-table instead of ASCII table.

[...]

> +=
> +How to Get ``printk`` Format Specifiers Right
> +=
> 
> :Author: Randy Dunlap 
> :Author: Andrew Murray 
> @@ -8,56 +8,91 @@ How to get printk format specifiers right
> Integer types
> =
> 
> -::
> +For printing integer types, we have the following format specifiers:
> + 
> +   .. flat-table:: 
> +  :widths: 2 2
> +
> +  * - **Type**
> + - **Specifier**
> +
> +  * - ``int``
> +- ``%d`` or ``%x``
> +
> +  * - ``unsigned int``
> + - ``%u`` or ``%x``
> +
> +  * - ``long``
> + - ``%ld`` or ``%lx``
> +
> +  * - ``unsigned long``
> + - ``%lu`` or ``%lx``
> +
> +  * - ``long long``
> + - ``%lld`` or ``%llx``
> 
> - If variable is of Type, use printk format specifier:
> - 
> - int %d or %x
> - unsigned int%u or %x
> - long%ld or %lx
> - unsigned long   %lu or %lx
> - long long   %lld or %llx
> - unsigned long long  %llu or %llx
> - size_t  %zu or %zx
> - ssize_t %zd or %zx
> - s32 %d or %x
> - u32 %u or %x
> - s64 %lld or %llx
> - u64 %llu or %llx
> -
> -If  is dependent on a config option for its size (e.g., ``sector_t``,
> +  * - ``unsigned long long``
> + - ``%llu`` or ``%llx``
> +
> +  * - ``size_t``
> + - ``%zu`` or ``%zx``
> +
> +  * - ``ssize_t``
> + - ``%zd`` or ``%zx``
> +
> +  * - ``s32``
> + - ``%d`` or ``%x``
> +
> +  * - ``u32``
> + - ``%u`` or ``%x``
> +
> +  * - ``s64``
> + - ``%lld`` or ``%llx``
> +
> +  * - ``u64``
> + - ``%llu`` or ``%llx``
> +
> +

Thanks!

just a question .. might it be better we stay with ASCII table
in cases like this. I guess this table won't changed often.
The flat-table directive is good for big and therefore frequently 
changed tables where a small precise diff reduce the patch.
But flat-table is also hard to read in plain text. Its a balancing
and thats my opinion, lets hear what other say ...

-- Markus --

Re: [PATCH] doc: convert printk-formats.txt to rst

[Markus Heiser]

> Am 06.12.2017 um 02:45 schrieb Tobin C. Harding :
> 
> Documentation/printk-formats.txt is a candidate for conversion to
> ReStructuredText format. Some effort has already been made to do this
> conversion even thought the suffix is currently .txt
> 
> Changes required to complete conversion
> 
> - Add double backticks where needed.
> - Add entry to Documentation/index.rst
> - Use flat-table instead of ASCII table.

[...]

> +=
> +How to Get ``printk`` Format Specifiers Right
> +=
> 
> :Author: Randy Dunlap 
> :Author: Andrew Murray 
> @@ -8,56 +8,91 @@ How to get printk format specifiers right
> Integer types
> =
> 
> -::
> +For printing integer types, we have the following format specifiers:
> + 
> +   .. flat-table:: 
> +  :widths: 2 2
> +
> +  * - **Type**
> + - **Specifier**
> +
> +  * - ``int``
> +- ``%d`` or ``%x``
> +
> +  * - ``unsigned int``
> + - ``%u`` or ``%x``
> +
> +  * - ``long``
> + - ``%ld`` or ``%lx``
> +
> +  * - ``unsigned long``
> + - ``%lu`` or ``%lx``
> +
> +  * - ``long long``
> + - ``%lld`` or ``%llx``
> 
> - If variable is of Type, use printk format specifier:
> - 
> - int %d or %x
> - unsigned int%u or %x
> - long%ld or %lx
> - unsigned long   %lu or %lx
> - long long   %lld or %llx
> - unsigned long long  %llu or %llx
> - size_t  %zu or %zx
> - ssize_t %zd or %zx
> - s32 %d or %x
> - u32 %u or %x
> - s64 %lld or %llx
> - u64 %llu or %llx
> -
> -If  is dependent on a config option for its size (e.g., ``sector_t``,
> +  * - ``unsigned long long``
> + - ``%llu`` or ``%llx``
> +
> +  * - ``size_t``
> + - ``%zu`` or ``%zx``
> +
> +  * - ``ssize_t``
> + - ``%zd`` or ``%zx``
> +
> +  * - ``s32``
> + - ``%d`` or ``%x``
> +
> +  * - ``u32``
> + - ``%u`` or ``%x``
> +
> +  * - ``s64``
> + - ``%lld`` or ``%llx``
> +
> +  * - ``u64``
> + - ``%llu`` or ``%llx``
> +
> +

Thanks!

just a question .. might it be better we stay with ASCII table
in cases like this. I guess this table won't changed often.
The flat-table directive is good for big and therefore frequently 
changed tables where a small precise diff reduce the patch.
But flat-table is also hard to read in plain text. Its a balancing
and thats my opinion, lets hear what other say ...

-- Markus --

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

[Markus Heiser]

> Am 07.10.2017 um 18:34 schrieb Jonathan Corbet :
> 
> On Wed,  4 Oct 2017 08:48:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
>> Right now, it is not possible to document nested struct and nested unions.
>> kernel-doc simply ignore them.
>> 
>> Add support to document them.
> 
> So I've finally found some time to actually look at these; sorry for being
> so absent from the discussion.  I plead $EXCUSES...
> 
> Some overall impressions:
> 
> - Seems like something we want.
> - I love hacking all the cruft out of kernel-doc; I've been meaning to
>  do that for a bit.
> - It would sure be nice to restore proper man-page generation rather than
>  documenting a hack with a perl script.  Someday.
> 
> I have one real complaint, though: with these patches applied, a "make
> htmldocs" generates about 5500 (!) more warnings than it did before.  Over
> the last couple of months, I put a bit of focused time into reducing
> warnings, and managed to get rid of 20-30 of them.  Now I feel despondent.
> 
> I really don't want to add that much noise to the docs build; I think it
> will defeat any hope of cleaning up the warnings we already have.  I
> wonder if we could suppress warnings about nested structs by default, and
> add a flag or envar to turn them back on for those who want them?

This is what I vote for: +1

> In truth, now that I look, the real problem is that the warnings are
> repeated many times - as in, like 100 times:
> 
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> <107 instances deleted...>
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> 
> That's not something that used to happen, and is certainly not helpful.
> Figuring that out would help a lot.  Can I get you to take a look at
> what's going on here?

Hi Jon, if you grep for 

 .. kernel-doc:: include/net/cfg80211.h

e.g. by:  grep cfg80211.h Documentation/driver-api/80211/cfg80211.rst | wc -l
you will see, that this directive is used 110 times in one reST file. If you
have 5 warnings about the kernel-doc markup in include/net/cfg80211.h you will
get 550 warnings about kernel-doc markup just for just one source code file.

This is because the kernel-doc parser is an external process which is called
(in this example) 110 times for one reST file and one source code file. If 
include/net/cfg80211.h is referred in other reST files, we got accordingly
more and more warnings .. thats really noise. 

So what I see is, that a major part of such noise is self made, it was one major
goal when I started with the python implementation: run kernel-doc parser in 
the 
Sphinx process, cache the results and use it in all reST files where it is 
required
from a kernel-doc directive.

Since there is also a man page solution in the py- version I vote to merge the
py-version into the kernel (ATM man is similar to what we had in DocBook and
it is expandable). If you want to have a preview of the result, try this patch:

 https://return42.github.io/linuxdoc/linux.html

The only drawback of the py version I see is, that Mauro prefers perl and I do
not want to lose him as an contributer to the kernel-doc parser. IMO he is an
experienced C programmer with an excellent regexpr knowledge. This is, what is
needed for this job. May be we can motivate him to give python one more try,
this is what I hope.

The py version includes all patches we had to the perl version, but I also
know, that you are not 100% compliant with the coding style, this could
all be fixed in a RFC round. Excuse me if I annoying with this solution;
IMO it contains what we need, if we really want to make a step forward.

Thanks,

-- Markus --

> 
> Thanks,
> 
> jon
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

[Markus Heiser]

> Am 07.10.2017 um 18:34 schrieb Jonathan Corbet :
> 
> On Wed,  4 Oct 2017 08:48:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
>> Right now, it is not possible to document nested struct and nested unions.
>> kernel-doc simply ignore them.
>> 
>> Add support to document them.
> 
> So I've finally found some time to actually look at these; sorry for being
> so absent from the discussion.  I plead $EXCUSES...
> 
> Some overall impressions:
> 
> - Seems like something we want.
> - I love hacking all the cruft out of kernel-doc; I've been meaning to
>  do that for a bit.
> - It would sure be nice to restore proper man-page generation rather than
>  documenting a hack with a perl script.  Someday.
> 
> I have one real complaint, though: with these patches applied, a "make
> htmldocs" generates about 5500 (!) more warnings than it did before.  Over
> the last couple of months, I put a bit of focused time into reducing
> warnings, and managed to get rid of 20-30 of them.  Now I feel despondent.
> 
> I really don't want to add that much noise to the docs build; I think it
> will defeat any hope of cleaning up the warnings we already have.  I
> wonder if we could suppress warnings about nested structs by default, and
> add a flag or envar to turn them back on for those who want them?

This is what I vote for: +1

> In truth, now that I look, the real problem is that the warnings are
> repeated many times - as in, like 100 times:
> 
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> <107 instances deleted...>
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> 
> That's not something that used to happen, and is certainly not helpful.
> Figuring that out would help a lot.  Can I get you to take a look at
> what's going on here?

Hi Jon, if you grep for 

 .. kernel-doc:: include/net/cfg80211.h

e.g. by:  grep cfg80211.h Documentation/driver-api/80211/cfg80211.rst | wc -l
you will see, that this directive is used 110 times in one reST file. If you
have 5 warnings about the kernel-doc markup in include/net/cfg80211.h you will
get 550 warnings about kernel-doc markup just for just one source code file.

This is because the kernel-doc parser is an external process which is called
(in this example) 110 times for one reST file and one source code file. If 
include/net/cfg80211.h is referred in other reST files, we got accordingly
more and more warnings .. thats really noise. 

So what I see is, that a major part of such noise is self made, it was one major
goal when I started with the python implementation: run kernel-doc parser in 
the 
Sphinx process, cache the results and use it in all reST files where it is 
required
from a kernel-doc directive.

Since there is also a man page solution in the py- version I vote to merge the
py-version into the kernel (ATM man is similar to what we had in DocBook and
it is expandable). If you want to have a preview of the result, try this patch:

 https://return42.github.io/linuxdoc/linux.html

The only drawback of the py version I see is, that Mauro prefers perl and I do
not want to lose him as an contributer to the kernel-doc parser. IMO he is an
experienced C programmer with an excellent regexpr knowledge. This is, what is
needed for this job. May be we can motivate him to give python one more try,
this is what I hope.

The py version includes all patches we had to the perl version, but I also
know, that you are not 100% compliant with the coding style, this could
all be fixed in a RFC round. Excuse me if I annoying with this solution;
IMO it contains what we need, if we really want to make a step forward.

Thanks,

-- Markus --

> 
> Thanks,
> 
> jon
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rcu kernel-doc issues (4.14-rc1)

[Markus Heiser]

> Am 18.09.2017 um 04:40 schrieb Paul E. McKenney :
[...]
> And after some playing around, I did get rid of the error messages.
> The code-block output looks a bit strange though, hints?  I preceded
> the code block with "::", again at Akira's suggestion.  It looks fine
> except for the :c:func: and backquotes on the first and last lines.
> 
>   Thanx, Paul
> 
> 
> 
> :c:func:`rcu_read_lock()`;
> p = rcu_dereference(gp);
> long_lived = is_long_lived(p);
> if (long_lived) {
>   if (!atomic_inc_not_zero(p->refcnt))
>   long_lived = false;
>   else
>   p = rcu_pointer_handoff(p);
>   }
> :c:func:`rcu_read_unlock()`;

FYI: 

such replacements in code-blocks are comming from the "Highlights
and cross-references" see:

 
https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#highlights-and-cross-references

and this is still a bug in the kernel-doc parser:

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg14409.html

-- Markus --

Re: rcu kernel-doc issues (4.14-rc1)

[Markus Heiser]

> Am 18.09.2017 um 04:40 schrieb Paul E. McKenney :
[...]
> And after some playing around, I did get rid of the error messages.
> The code-block output looks a bit strange though, hints?  I preceded
> the code block with "::", again at Akira's suggestion.  It looks fine
> except for the :c:func: and backquotes on the first and last lines.
> 
>   Thanx, Paul
> 
> 
> 
> :c:func:`rcu_read_lock()`;
> p = rcu_dereference(gp);
> long_lived = is_long_lived(p);
> if (long_lived) {
>   if (!atomic_inc_not_zero(p->refcnt))
>   long_lived = false;
>   else
>   p = rcu_pointer_handoff(p);
>   }
> :c:func:`rcu_read_unlock()`;

FYI: 

such replacements in code-blocks are comming from the "Highlights
and cross-references" see:

 
https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#highlights-and-cross-references

and this is still a bug in the kernel-doc parser:

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg14409.html

-- Markus --

Re: rcu kernel-doc issues (4.14-rc1)

[Markus Heiser]

> Am 17.09.2017 um 03:26 schrieb Randy Dunlap :
> 
> On 4.14-rc1, I am seeing lots of warnings on rcu kernel-doc:
> 
> .. kernel-doc:: include/linux/rcupdate.h
>  :external:
> ./Documentation/core-api/kernel-api.rst:357: ERROR: Error in "kernel-doc" 
> directive:
> unknown option: "external".

FYI: about kernel-doc option (export) take a look at

 https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html

-- Markus --

Re: rcu kernel-doc issues (4.14-rc1)

[Markus Heiser]

> Am 17.09.2017 um 03:26 schrieb Randy Dunlap :
> 
> On 4.14-rc1, I am seeing lots of warnings on rcu kernel-doc:
> 
> .. kernel-doc:: include/linux/rcupdate.h
>  :external:
> ./Documentation/core-api/kernel-api.rst:357: ERROR: Error in "kernel-doc" 
> directive:
> unknown option: "external".

FYI: about kernel-doc option (export) take a look at

 https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html

-- Markus --

Re: [PATCH 1/2] PM: docs: Describe high-level PM strategies and sleep states

[Markus Heiser]

Hi Rafael,

great work, helps me and the world saving fossil fuels ;)

> Am 20.08.2017 um 18:05 schrieb Rafael J. Wysocki :
> 
> From: Rafael J. Wysocki 
> 
> Reorganize the power management part of admin-guide by adding a
> description of major power management strategies supported by the
> kernel (system-wide and working-state power management) to it and
> dividing the rest of the material into the system-wide PM and
> working-state PM chapters.
> 
> On top of that, add a description of system sleep states to the
> system-wide PM chapter.
> 
> Signed-off-by: Rafael J. Wysocki 
> ---
> Documentation/admin-guide/pm/index.rst |5 
> Documentation/admin-guide/pm/sleep-states.rst  |  234 
> +
> Documentation/admin-guide/pm/strategies.rst|   52 +
> Documentation/admin-guide/pm/system-wide.rst   |   15 +
> Documentation/admin-guide/pm/working-state.rst |   16 +
> 5 files changed, 320 insertions(+), 2 deletions(-)
> 
> Index: linux-pm/Documentation/admin-guide/pm/index.rst
> ===
> --- linux-pm.orig/Documentation/admin-guide/pm/index.rst
> +++ linux-pm/Documentation/admin-guide/pm/index.rst
> @@ -5,8 +5,9 @@ Power Management
> .. toctree::
>:maxdepth: 2
> 
> -   cpufreq
> -   intel_pstate
> +   strategies
> +   system-wide
> +   working-state
> 
> .. only::  subproject and html

BTW: lets drop this C 'only' block here. Subprojects are only where a
Documentation/*/conf.py exists. The build does not have sub-sub-projects
at Documentation/*/*/conf.py level.


> Index: linux-pm/Documentation/admin-guide/pm/sleep-states.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/sleep-states.rst
> @@ -0,0 +1,234 @@
> +===
> +System Sleep States
> +===
> +
> +::
> +
> + Copyright (c) 2017 Intel Corp., Rafael J. Wysocki 
> 
> +
> +Sleep states are global low-power states of the entire system in which user
> +space code cannot be executed and the overall system activity is 
> significantly
> +reduced.
> +
> +
> +

[...]
> Index: linux-pm/Documentation/admin-guide/pm/system-wide.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/system-wide.rst
> @@ -0,0 +1,15 @@
> +
> +System-Wide Power Management
> +
> +
> +.. toctree::
> +   :maxdepth: 2
> +
> +   sleep-states
> +
> +.. only::  subproject and html
> +
> +   Indices
> +   ===
> +
> +   * :ref:`genindex`

Same here, lets drop the 'only' block.


> Index: linux-pm/Documentation/admin-guide/pm/working-state.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/working-state.rst
> @@ -0,0 +1,16 @@
> +==
> +Working-State Power Management
> +==
> +
> +.. toctree::
> +   :maxdepth: 2
> +
> +   cpufreq
> +   intel_pstate
> +
> +.. only::  subproject and html
> +
> +   Indices
> +   ===
> +
> +   * :ref:`genindex`

Same here, lets drop the 'only' block.

> Index: linux-pm/Documentation/admin-guide/pm/strategies.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/strategies.rst
> @@ -0,0 +1,52 @@
> +===
> +Power Management Strategies
> +===
> +
> +::
> +
> + Copyright (c) 2017 Intel Corp., Rafael J. Wysocki 
> 
> +
> +The Linux kernel supports two major high-level power management strategies.
> +
> +One of them is based on using global low-power states of the whole system in
> +which user space code cannot be executed and the overall system activity is
> +significantly reduced, referred to as :doc:`sleep states `.  
> The
> +kernel puts the system into one of these states when requested by user space
> +and the system stays in it until a special signal is received from one of
> +designated devices, triggering a transition to the ``working state`` in which
> +user space code can run.  Because sleep states are global and the whole 
> system
> +is affected by the state changes, this strategy is referred to as the
> +:doc:`system-wide power management `.

I really appreciate your hyper-ref markup using :doc:`Foo ` and `Bar`_
it is very clear when reading the raw text files in the source tree. I think
this has its value, but it also has its drawbacks when chapters are renamed
or moved from one file to another. Thats, why I recommend using explicitly
anchors / may be the maintainer Jon can give us his assessment. What I mean::

.. _foo:

Foo
===

lorem see :ref:`bar`

.. _bar:

Bar
===

ipsum see :ref:`foo`

With explicitly anchors:

- 

Re: [PATCH 1/2] PM: docs: Describe high-level PM strategies and sleep states

[Markus Heiser]

Hi Rafael,

great work, helps me and the world saving fossil fuels ;)

> Am 20.08.2017 um 18:05 schrieb Rafael J. Wysocki :
> 
> From: Rafael J. Wysocki 
> 
> Reorganize the power management part of admin-guide by adding a
> description of major power management strategies supported by the
> kernel (system-wide and working-state power management) to it and
> dividing the rest of the material into the system-wide PM and
> working-state PM chapters.
> 
> On top of that, add a description of system sleep states to the
> system-wide PM chapter.
> 
> Signed-off-by: Rafael J. Wysocki 
> ---
> Documentation/admin-guide/pm/index.rst |5 
> Documentation/admin-guide/pm/sleep-states.rst  |  234 
> +
> Documentation/admin-guide/pm/strategies.rst|   52 +
> Documentation/admin-guide/pm/system-wide.rst   |   15 +
> Documentation/admin-guide/pm/working-state.rst |   16 +
> 5 files changed, 320 insertions(+), 2 deletions(-)
> 
> Index: linux-pm/Documentation/admin-guide/pm/index.rst
> ===
> --- linux-pm.orig/Documentation/admin-guide/pm/index.rst
> +++ linux-pm/Documentation/admin-guide/pm/index.rst
> @@ -5,8 +5,9 @@ Power Management
> .. toctree::
>:maxdepth: 2
> 
> -   cpufreq
> -   intel_pstate
> +   strategies
> +   system-wide
> +   working-state
> 
> .. only::  subproject and html

BTW: lets drop this C 'only' block here. Subprojects are only where a
Documentation/*/conf.py exists. The build does not have sub-sub-projects
at Documentation/*/*/conf.py level.


> Index: linux-pm/Documentation/admin-guide/pm/sleep-states.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/sleep-states.rst
> @@ -0,0 +1,234 @@
> +===
> +System Sleep States
> +===
> +
> +::
> +
> + Copyright (c) 2017 Intel Corp., Rafael J. Wysocki 
> 
> +
> +Sleep states are global low-power states of the entire system in which user
> +space code cannot be executed and the overall system activity is 
> significantly
> +reduced.
> +
> +
> +

[...]
> Index: linux-pm/Documentation/admin-guide/pm/system-wide.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/system-wide.rst
> @@ -0,0 +1,15 @@
> +
> +System-Wide Power Management
> +
> +
> +.. toctree::
> +   :maxdepth: 2
> +
> +   sleep-states
> +
> +.. only::  subproject and html
> +
> +   Indices
> +   ===
> +
> +   * :ref:`genindex`

Same here, lets drop the 'only' block.


> Index: linux-pm/Documentation/admin-guide/pm/working-state.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/working-state.rst
> @@ -0,0 +1,16 @@
> +==
> +Working-State Power Management
> +==
> +
> +.. toctree::
> +   :maxdepth: 2
> +
> +   cpufreq
> +   intel_pstate
> +
> +.. only::  subproject and html
> +
> +   Indices
> +   ===
> +
> +   * :ref:`genindex`

Same here, lets drop the 'only' block.

> Index: linux-pm/Documentation/admin-guide/pm/strategies.rst
> ===
> --- /dev/null
> +++ linux-pm/Documentation/admin-guide/pm/strategies.rst
> @@ -0,0 +1,52 @@
> +===
> +Power Management Strategies
> +===
> +
> +::
> +
> + Copyright (c) 2017 Intel Corp., Rafael J. Wysocki 
> 
> +
> +The Linux kernel supports two major high-level power management strategies.
> +
> +One of them is based on using global low-power states of the whole system in
> +which user space code cannot be executed and the overall system activity is
> +significantly reduced, referred to as :doc:`sleep states `.  
> The
> +kernel puts the system into one of these states when requested by user space
> +and the system stays in it until a special signal is received from one of
> +designated devices, triggering a transition to the ``working state`` in which
> +user space code can run.  Because sleep states are global and the whole 
> system
> +is affected by the state changes, this strategy is referred to as the
> +:doc:`system-wide power management `.

I really appreciate your hyper-ref markup using :doc:`Foo ` and `Bar`_
it is very clear when reading the raw text files in the source tree. I think
this has its value, but it also has its drawbacks when chapters are renamed
or moved from one file to another. Thats, why I recommend using explicitly
anchors / may be the maintainer Jon can give us his assessment. What I mean::

.. _foo:

Foo
===

lorem see :ref:`bar`

.. _bar:

Bar
===

ipsum see :ref:`foo`

With explicitly anchors:

- there is no need to track when a chapter is renamed
- or moved to another file,
- same to renaming files
- with intersphinx [1] other 

Re: [PATCH 2/2] PM: docs: Delete the obsolete states.txt document

[Markus Heiser]

> Am 20.08.2017 um 18:06 schrieb Rafael J. Wysocki :
> 
> From: Rafael J. Wysocki 
> 
> The Documentation/power/states.txt document is now redundant and
> sonewhat outdated, so delete it.
> 
> Signed-off-by: Rafael J. Wysocki 
> ---
> Documentation/power/states.txt |  127 
> -
> 1 file changed, 127 deletions(-)

strange, I tried to apply the series, 1/2 applied but with 2/2 I get a 
"sha1 information is lacking or useless", did I miss something?

-- Markus --

Re: [PATCH 2/2] PM: docs: Delete the obsolete states.txt document

[Markus Heiser]

> Am 20.08.2017 um 18:06 schrieb Rafael J. Wysocki :
> 
> From: Rafael J. Wysocki 
> 
> The Documentation/power/states.txt document is now redundant and
> sonewhat outdated, so delete it.
> 
> Signed-off-by: Rafael J. Wysocki 
> ---
> Documentation/power/states.txt |  127 
> -
> 1 file changed, 127 deletions(-)

strange, I tried to apply the series, 1/2 applied but with 2/2 I get a 
"sha1 information is lacking or useless", did I miss something?

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 12:57 schrieb Mauro Carvalho Chehab 
> :
> 
> So, we need a way for kfigure to fallback when distros don't
> have such feature.

Hm .. I'am not very happy to implement where distros packaging fail.
But .. if there is really a need for, I will do so.

Lets see how this series goes upstream and feel free to remind
on my words if I completely suppressed the idea after upstream. :) 

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 12:57 schrieb Mauro Carvalho Chehab 
> :
> 
> So, we need a way for kfigure to fallback when distros don't
> have such feature.

Hm .. I'am not very happy to implement where distros packaging fail.
But .. if there is really a need for, I will do so.

Lets see how this series goes upstream and feel free to remind
on my words if I completely suppressed the idea after upstream. :) 

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 12:09 schrieb Mauro Carvalho Chehab 
> :
> 
>> 
>> Anyway, I guess we should modify kfigure.py to check if PDF is
>> available, falling back to SVG, using ImageMagick to convert
>> from SVG to PDF.
> 
> Ok, I discovered that, on Fedora, support for pdf, png and some other
> formats are packaged on a separate package: graphviz-gd.x86_64
> 
> With it installed, it now shows "pdf":
> 

Ah, OK .. thanks a lot for clarifying.

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 12:09 schrieb Mauro Carvalho Chehab 
> :
> 
>> 
>> Anyway, I guess we should modify kfigure.py to check if PDF is
>> available, falling back to SVG, using ImageMagick to convert
>> from SVG to PDF.
> 
> Ok, I discovered that, on Fedora, support for pdf, png and some other
> formats are packaged on a separate package: graphviz-gd.x86_64
> 
> With it installed, it now shows "pdf":
> 

Ah, OK .. thanks a lot for clarifying.

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 00:08 schrieb Mauro Carvalho Chehab 
> :
> 
> [1] There's an unrelated bug with the Kernel's sphinx extension
> kimage: when parsing GraphViz graphs, it uses "-Tpdf" argument,
> in order to generate a PDF image. That doesn't work on some
> distros, as GraphViz doesn't support PDF images. 
> 

Hmm .. http://www.graphviz.org/content/output-formats#dpdf

Is it a problem of the GraphViz version?

See http://www.graphviz.org/News.php ..

  """New Release 2.32 (1 August 2013)

 ... In addition, if the poppler library is available, Graphviz
 can now use PDF files as images."""

I suppose that 2.32 is in your distro .. it's from 2013

It seems, that the graphiz homepage is not up to date. I found 
the repo at github. There is a issue about PDF

  https://github.com/ellson/graphviz/issues/1180

So I guess graphviz is compiled without HAVE_PANGOCAIRO in your
distros?

-- Markus --

Re: [PATCH 0/5] Add a script to check for Sphinx install requirements

[Markus Heiser]

> Am 17.07.2017 um 00:08 schrieb Mauro Carvalho Chehab 
> :
> 
> [1] There's an unrelated bug with the Kernel's sphinx extension
> kimage: when parsing GraphViz graphs, it uses "-Tpdf" argument,
> in order to generate a PDF image. That doesn't work on some
> distros, as GraphViz doesn't support PDF images. 
> 

Hmm .. http://www.graphviz.org/content/output-formats#dpdf

Is it a problem of the GraphViz version?

See http://www.graphviz.org/News.php ..

  """New Release 2.32 (1 August 2013)

 ... In addition, if the poppler library is available, Graphviz
 can now use PDF files as images."""

I suppose that 2.32 is in your distro .. it's from 2013

It seems, that the graphiz homepage is not up to date. I found 
the repo at github. There is a issue about PDF

  https://github.com/ellson/graphviz/issues/1180

So I guess graphviz is compiled without HAVE_PANGOCAIRO in your
distros?

-- Markus --

Re: [PATCH 3/5] sphinx-pre-install: use a requirements file

[Markus Heiser]

> Am 17.07.2017 um 00:08 schrieb Mauro Carvalho Chehab 
> :
> 
> Instead of using 3 commands to install a virtualenv, use
> a single one, reading the requirements from this file:
> 
>   Documentation/sphinx/requirements.txt
> 
> Signed-off-by: Mauro Carvalho Chehab 

Hi Mauro,

I get a ..

 fatal: sha1 information is lacking or useless 
(Documentation/doc-guide/sphinx.rst)

and  I miss the Documentation/doc-guide/sphinx.rst file in this patch.

Did it only happened to me?

-- Markus --

Re: [PATCH 3/5] sphinx-pre-install: use a requirements file

[Markus Heiser]

> Am 17.07.2017 um 00:08 schrieb Mauro Carvalho Chehab 
> :
> 
> Instead of using 3 commands to install a virtualenv, use
> a single one, reading the requirements from this file:
> 
>   Documentation/sphinx/requirements.txt
> 
> Signed-off-by: Mauro Carvalho Chehab 

Hi Mauro,

I get a ..

 fatal: sha1 information is lacking or useless 
(Documentation/doc-guide/sphinx.rst)

and  I miss the Documentation/doc-guide/sphinx.rst file in this patch.

Did it only happened to me?

-- Markus --

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

Hy Mauro,

thanks a lot for your RFC, your patch consolidate a lot of
knowledge around Sphinx build requirements and brings a huge
value I will no longer miss.

I tested v6 of this patch on ubuntu and there is only some
conceptual bikeshedding I can do. 


> Am 15.07.2017 um 14:49 schrieb Mauro Carvalho Chehab 
> :
> 
> As I said, the idea is to let the user to decide what it wants.
> 
> I focused on the packaging approach first because such logic
> is required for other packages. Now that it is working, just
> sent a version 5 that will use virtualenv for Sphinx by default.

Thanks! .. I don't know how I can make it better (I'am not a perl
programmer) but it seems, that global

my @missing;
my @opt_missing;

and the "sub add_package" is dominant, while the 'virtaulenv'
is glued in .. may we can find a better structure (later).


> Yet, before spending more time on such script, I'd like to have
> more feedback if:
> 
>   - is this approach acceptable?

Truly Yes!

I see, there is a value in the the "OS-packaging approach" even
if I prefer the  "native-packaging approach". Last means I like
to use the native packaging tools from python and LiveTeX.

For python, instead of :

printf "\t$virtualenv sphinx_1.4\n";
printf "\tpip install 'docutils==0.12'\n";
printf "\tpip install 'Sphinx==1.4.9'\n";
printf "\tpip install sphinx_rtd_theme\n";

add Documentation/sphinx/requirements.txt:

 ---
docutils==0.12
Sphinx==1.4.9
sphinx_rtd_theme
 -

And the print ...

printf "\t$virtualenv sphinx_1.4\n";
printf "\tpip install -r 
Documentation/sphinx/requirements.txt\n";

For TexLive:

ATM I have no idea how to set up a *requierements* file and
install everything without sudo. But I have seen your 'kpsewhich'
approach  which is very interesting for me. I suppose a solution
for this will end in a combination of 'kpsewhich' and  'tlmgr'.
But for this I have to do more investigations / sorry that
I can't spend more time on this task right now.

>   - should it have an optional argument that will make the
> script to run the needed commands;

No. We can do this later upstream.

>   - should it be integrated at the Documentation/Makefile?

No.

>   - what's the best name/location for such script?

I like to see the script under Documentation/sphinx

> I guess it could also use kpsewhich to check if the needed
> texlive packages are installed. However, the problem with such
> approach is that texlive-kpathsea-bin package should be installed
> first, in order to provide such command.

I see you have solved it in v6 .. Thanks!

> 
> So, installing PDF and math dependencies would require two steps.
> 
>> I tested sphinx-pre-install and it works fine for me, thats not the
>> point. The point is: what do we recommend? E.g. for me it advices me
>> to run:
>> 
>>  sudo apt-get install python3-sphinx python3-sphinx-rtd-theme
>> 
>> We should not assume that the developer (better: the build-user) owns the
>> privilege to install fine grained OS packages. There is a admin-part and
>> a user-part:
> 
> That's not relevant. Typically, anyone that is building a Kernel has
> admin privileges, otherwise it can't actually test the Kernel that was
> built.

Hmm .. buildbots and Continuous Integration (CI)?

> Ok, there are exceptions to that, but, on such case, the user should
> be able to request the admin to install whatever packages are needed
> to build the Kernel.
> 
> Thanks,
> Mauro

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

Hy Mauro,

thanks a lot for your RFC, your patch consolidate a lot of
knowledge around Sphinx build requirements and brings a huge
value I will no longer miss.

I tested v6 of this patch on ubuntu and there is only some
conceptual bikeshedding I can do. 


> Am 15.07.2017 um 14:49 schrieb Mauro Carvalho Chehab 
> :
> 
> As I said, the idea is to let the user to decide what it wants.
> 
> I focused on the packaging approach first because such logic
> is required for other packages. Now that it is working, just
> sent a version 5 that will use virtualenv for Sphinx by default.

Thanks! .. I don't know how I can make it better (I'am not a perl
programmer) but it seems, that global

my @missing;
my @opt_missing;

and the "sub add_package" is dominant, while the 'virtaulenv'
is glued in .. may we can find a better structure (later).


> Yet, before spending more time on such script, I'd like to have
> more feedback if:
> 
>   - is this approach acceptable?

Truly Yes!

I see, there is a value in the the "OS-packaging approach" even
if I prefer the  "native-packaging approach". Last means I like
to use the native packaging tools from python and LiveTeX.

For python, instead of :

printf "\t$virtualenv sphinx_1.4\n";
printf "\tpip install 'docutils==0.12'\n";
printf "\tpip install 'Sphinx==1.4.9'\n";
printf "\tpip install sphinx_rtd_theme\n";

add Documentation/sphinx/requirements.txt:

 ---
docutils==0.12
Sphinx==1.4.9
sphinx_rtd_theme
 -

And the print ...

printf "\t$virtualenv sphinx_1.4\n";
printf "\tpip install -r 
Documentation/sphinx/requirements.txt\n";

For TexLive:

ATM I have no idea how to set up a *requierements* file and
install everything without sudo. But I have seen your 'kpsewhich'
approach  which is very interesting for me. I suppose a solution
for this will end in a combination of 'kpsewhich' and  'tlmgr'.
But for this I have to do more investigations / sorry that
I can't spend more time on this task right now.

>   - should it have an optional argument that will make the
> script to run the needed commands;

No. We can do this later upstream.

>   - should it be integrated at the Documentation/Makefile?

No.

>   - what's the best name/location for such script?

I like to see the script under Documentation/sphinx

> I guess it could also use kpsewhich to check if the needed
> texlive packages are installed. However, the problem with such
> approach is that texlive-kpathsea-bin package should be installed
> first, in order to provide such command.

I see you have solved it in v6 .. Thanks!

> 
> So, installing PDF and math dependencies would require two steps.
> 
>> I tested sphinx-pre-install and it works fine for me, thats not the
>> point. The point is: what do we recommend? E.g. for me it advices me
>> to run:
>> 
>>  sudo apt-get install python3-sphinx python3-sphinx-rtd-theme
>> 
>> We should not assume that the developer (better: the build-user) owns the
>> privilege to install fine grained OS packages. There is a admin-part and
>> a user-part:
> 
> That's not relevant. Typically, anyone that is building a Kernel has
> admin privileges, otherwise it can't actually test the Kernel that was
> built.

Hmm .. buildbots and Continuous Integration (CI)?

> Ok, there are exceptions to that, but, on such case, the user should
> be able to request the admin to install whatever packages are needed
> to build the Kernel.
> 
> Thanks,
> Mauro

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

> Am 15.07.2017 um 04:21 schrieb Mauro Carvalho Chehab 
> <mche...@s-opensource.com>:
> 
> Em Fri, 14 Jul 2017 19:35:59 +0200
> Markus Heiser <markus.hei...@darmarit.de> escreveu:
> 
>>> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab 
>>> <mche...@s-opensource.com>:
>>> 
>>> Solving Sphinx dependencies can be painful. Add a script to
>>> check if everything is ok.  
>> 
>> just my 5cent:
>> 
>> What we need is a "requirements.txt" file to define a
>> **reference environment**. E.g. to stick Sphinx 1.4.9 in
>> such a reference environment::
>> 
>>  ---
>> Sphinx==1.4.9
>> sphinx_rtd_theme
>>  -
>> 
>> The rest is similarly to what you wrote in doc-guide/sphinx.rst ...
>> 
>> The ref-environment can be build with virtualenv & pip::
>> 
>> $ virtualenv --python=python3 docenv
>> (doc-env) $ source ./docenv/bin/activate
>> (doc-env) $ pip install -r requirements.txt
>> 
>> From now we can start our build as usual. If not already done,
>> first activate the environment::
>> 
>> $ . ./docenv/bin/activate
>> (doc-env) $ make htmldocs
>> 
>> This (requirements.txt) is the way python packaging goes.
> 
> 
> The above assumes that the user wants to use virtenv and
> have python3, virtualenv3 and pip3 already installed.
> 
> I agree that a virtual environment works better than using
> distro-specific packaging, as Sphinx toolchain is really
> fragile. But we should give an option for the developer to
> use whatever he wants.

The developer is free to choose the way he like. But we are talking
about what is "best practice".

I tested sphinx-pre-install and it works fine for me, thats not the
point. The point is: what do we recommend? E.g. for me it advices me
to run:

sudo apt-get install python3-sphinx python3-sphinx-rtd-theme

We should not assume that the developer (better: the build-user) owns the
privilege to install fine grained OS packages. There is a admin-part and
a user-part:

The admin (sudoer) installs binaries into the OS:

* gcc
* make
* python3, virtualenv
* LiveTex
* ImageMagick
* perl
* graphviz

The user (developer) installs additional requirements anywhere
under $HOME:

* Python 

  best practice here is virtualenv and pip .. and the *recipe*
  to get a *reference environment* is the requirements.txt
 
* LaTeX

  what is best practice here? .. I know there is some package
  managing with "TeX Live Manager" (tlmgr) but I have no experience
  with. Is there a way to define a *requirements* file and
  to install such requirements anywhere under $HOME?

My english is less eloquent, but I hope its clear what I mean.

To have a script is nice, but first lets explore what best practice
is.


-- Markus --

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

> Am 15.07.2017 um 04:21 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Fri, 14 Jul 2017 19:35:59 +0200
> Markus Heiser  escreveu:
> 
>>> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab 
>>> :
>>> 
>>> Solving Sphinx dependencies can be painful. Add a script to
>>> check if everything is ok.  
>> 
>> just my 5cent:
>> 
>> What we need is a "requirements.txt" file to define a
>> **reference environment**. E.g. to stick Sphinx 1.4.9 in
>> such a reference environment::
>> 
>>  ---
>> Sphinx==1.4.9
>> sphinx_rtd_theme
>>  -
>> 
>> The rest is similarly to what you wrote in doc-guide/sphinx.rst ...
>> 
>> The ref-environment can be build with virtualenv & pip::
>> 
>> $ virtualenv --python=python3 docenv
>> (doc-env) $ source ./docenv/bin/activate
>> (doc-env) $ pip install -r requirements.txt
>> 
>> From now we can start our build as usual. If not already done,
>> first activate the environment::
>> 
>> $ . ./docenv/bin/activate
>> (doc-env) $ make htmldocs
>> 
>> This (requirements.txt) is the way python packaging goes.
> 
> 
> The above assumes that the user wants to use virtenv and
> have python3, virtualenv3 and pip3 already installed.
> 
> I agree that a virtual environment works better than using
> distro-specific packaging, as Sphinx toolchain is really
> fragile. But we should give an option for the developer to
> use whatever he wants.

The developer is free to choose the way he like. But we are talking
about what is "best practice".

I tested sphinx-pre-install and it works fine for me, thats not the
point. The point is: what do we recommend? E.g. for me it advices me
to run:

sudo apt-get install python3-sphinx python3-sphinx-rtd-theme

We should not assume that the developer (better: the build-user) owns the
privilege to install fine grained OS packages. There is a admin-part and
a user-part:

The admin (sudoer) installs binaries into the OS:

* gcc
* make
* python3, virtualenv
* LiveTex
* ImageMagick
* perl
* graphviz

The user (developer) installs additional requirements anywhere
under $HOME:

* Python 

  best practice here is virtualenv and pip .. and the *recipe*
  to get a *reference environment* is the requirements.txt
 
* LaTeX

  what is best practice here? .. I know there is some package
  managing with "TeX Live Manager" (tlmgr) but I have no experience
  with. Is there a way to define a *requirements* file and
  to install such requirements anywhere under $HOME?

My english is less eloquent, but I hope its clear what I mean.

To have a script is nice, but first lets explore what best practice
is.


-- Markus --

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab 
> :
> 
> Solving Sphinx dependencies can be painful. Add a script to
> check if everything is ok.

just my 5cent:

What we need is a "requirements.txt" file to define a
**reference environment**. E.g. to stick Sphinx 1.4.9 in
such a reference environment::

 ---
Sphinx==1.4.9
sphinx_rtd_theme
 -

The rest is similarly to what you wrote in doc-guide/sphinx.rst ...

The ref-environment can be build with virtualenv & pip::

$ virtualenv --python=python3 docenv
(doc-env) $ source ./docenv/bin/activate
(doc-env) $ pip install -r requirements.txt

>From now we can start our build as usual. If not already done,
first activate the environment::

$ . ./docenv/bin/activate
(doc-env) $ make htmldocs

This (requirements.txt) is the way python packaging goes.

Ok, this won't solve TeX installation problems of Linux distros,
for this a script like the one here in your RFC is helpful.

-- Markus --

Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

[Markus Heiser]

> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab 
> :
> 
> Solving Sphinx dependencies can be painful. Add a script to
> check if everything is ok.

just my 5cent:

What we need is a "requirements.txt" file to define a
**reference environment**. E.g. to stick Sphinx 1.4.9 in
such a reference environment::

 ---
Sphinx==1.4.9
sphinx_rtd_theme
 -

The rest is similarly to what you wrote in doc-guide/sphinx.rst ...

The ref-environment can be build with virtualenv & pip::

$ virtualenv --python=python3 docenv
(doc-env) $ source ./docenv/bin/activate
(doc-env) $ pip install -r requirements.txt

>From now we can start our build as usual. If not already done,
first activate the environment::

$ . ./docenv/bin/activate
(doc-env) $ make htmldocs

This (requirements.txt) is the way python packaging goes.

Ok, this won't solve TeX installation problems of Linux distros,
for this a script like the one here in your RFC is helpful.

-- Markus --

Re: [PATCH 2/6] docs-rst: update Sphinx install instructions

[Markus Heiser]

> Am 14.07.2017 um 17:44 schrieb Jonathan Corbet :
> 
> On Fri, 14 Jul 2017 08:08:19 -0300
> Mauro Carvalho Chehab  wrote:
> 
>> +Please see :ref:`sphinx_install` at the doc-guide for details about
>> +Sphinx requirements.
> 
> One small comment here: formatting things this way assumes that people are
> reading the documentation in a web browser.  Can we replace "at the
> doc-guide" with "in Documentation/doc-guide/sphinx.rst" to improve the
> usability of the plain-text files?

Hm .. I suppose there is a plain text builder for sphinx .. but building
plain-text before reading sources seems not an acceptable option ;)

-- Markus --

Re: [PATCH 2/6] docs-rst: update Sphinx install instructions

[Markus Heiser]

> Am 14.07.2017 um 17:44 schrieb Jonathan Corbet :
> 
> On Fri, 14 Jul 2017 08:08:19 -0300
> Mauro Carvalho Chehab  wrote:
> 
>> +Please see :ref:`sphinx_install` at the doc-guide for details about
>> +Sphinx requirements.
> 
> One small comment here: formatting things this way assumes that people are
> reading the documentation in a web browser.  Can we replace "at the
> doc-guide" with "in Documentation/doc-guide/sphinx.rst" to improve the
> usability of the plain-text files?

Hm .. I suppose there is a plain text builder for sphinx .. but building
plain-text before reading sources seems not an acceptable option ;)

-- Markus --

Re: [PATCH v2 00/53] Get rid of Docbook

[Markus Heiser]

> Am 05.07.2017 um 23:45 schrieb Jim Davis <jim.ep...@gmail.com>:
> 
> On Fri, Jun 16, 2017 at 7:03 AM, Markus Heiser
> <markus.hei...@darmarit.de> wrote:
>> 
> 
>> docproc and some lines in the Makefile & .gitignore
>> 
>> ./scripts/docproc.c
>> ./scripts/.docproc.cmd
>> ./scripts/Makefile
>> ./scripts/.gitignore
> 
> With 4.12, running make xmldocs (or any other working target) and then
> "make cleandocs; git clean -fdx" turns up
> 
> Removing scripts/.check-lc_ctype.cmd
> Removing scripts/.docproc.cmd
> Removing scripts/basic/.fixdep.cmd
> Removing scripts/basic/fixdep
> Removing scripts/check-lc_ctype
> Removing scripts/docproc

Tried with Jon's docs-next .. only 'fixdep' remains (not part
of Documentation). I guess the docproc & Co. was already fixed
by Jon [1]:

 52b3f23 Docs: clean up some DocBook loose ends

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg13238.html

-- Markus --

Re: [PATCH v2 00/53] Get rid of Docbook

[Markus Heiser]

> Am 05.07.2017 um 23:45 schrieb Jim Davis :
> 
> On Fri, Jun 16, 2017 at 7:03 AM, Markus Heiser
>  wrote:
>> 
> 
>> docproc and some lines in the Makefile & .gitignore
>> 
>> ./scripts/docproc.c
>> ./scripts/.docproc.cmd
>> ./scripts/Makefile
>> ./scripts/.gitignore
> 
> With 4.12, running make xmldocs (or any other working target) and then
> "make cleandocs; git clean -fdx" turns up
> 
> Removing scripts/.check-lc_ctype.cmd
> Removing scripts/.docproc.cmd
> Removing scripts/basic/.fixdep.cmd
> Removing scripts/basic/fixdep
> Removing scripts/check-lc_ctype
> Removing scripts/docproc

Tried with Jon's docs-next .. only 'fixdep' remains (not part
of Documentation). I guess the docproc & Co. was already fixed
by Jon [1]:

 52b3f23 Docs: clean up some DocBook loose ends

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg13238.html

-- Markus --

Re: [PATCH 0/5] Make PDF builds work again

[Markus Heiser]

> Am 05.07.2017 um 23:22 schrieb Jim Davis :
> 
> On Mon, Jul 3, 2017 at 5:44 AM, Jonathan Corbet  wrote:
>> On Mon, 3 Jul 2017 10:25:38 +0200
>> Daniel Vetter  wrote:
>> 
>>> Only now stumbled over the full thread, but the drm patch is already
>>> queued up for at least 4.13 (Dave was out and all that). I guess we could
>>> try to cherry-pick through stable.
>> 
>> I kind of gave up on the 4.12 goal, at least for now.  The number of
>> complaints has not been huge - I suspect you're far from the only one who
>> is not too worried about building PDFs...:)
> 
> If fixing pdf (and ps) builds isn't worth the bother -- which I
> wouldn't debate -- then it's best to just drop those build targets.
> The only worrisome thing I see here is having build targets carried
> from release to release that don't work.

my 5cent:

we have to communicate that PDF build is in a beta stage (for a long
time).

Sphinx-doc's PDF chain was not well maintained for a long time. With
newer versions (started with 1.5 and continued in 1.6) it becomes
better and better. This gives me some hope that there comes a day
where building PDFs is robust enough to use in automatic builds.

As long as we try to support various version of Sphinx shipped by
various distros and at the same time make/need deep (LaTeX)
adjustments, we will find those discussions on the ML.

If you are doubtful about my assessment, compare Sphinx's 
TeX stuff from master 

https://github.com/sphinx-doc/sphinx/tree/master/sphinx/texinputs

with e.g. 1.4.9 tag

https://github.com/sphinx-doc/sphinx/tree/1.4.9/sphinx/texinputs

-- Markus --

Re: [PATCH 0/5] Make PDF builds work again

[Markus Heiser]

> Am 05.07.2017 um 23:22 schrieb Jim Davis :
> 
> On Mon, Jul 3, 2017 at 5:44 AM, Jonathan Corbet  wrote:
>> On Mon, 3 Jul 2017 10:25:38 +0200
>> Daniel Vetter  wrote:
>> 
>>> Only now stumbled over the full thread, but the drm patch is already
>>> queued up for at least 4.13 (Dave was out and all that). I guess we could
>>> try to cherry-pick through stable.
>> 
>> I kind of gave up on the 4.12 goal, at least for now.  The number of
>> complaints has not been huge - I suspect you're far from the only one who
>> is not too worried about building PDFs...:)
> 
> If fixing pdf (and ps) builds isn't worth the bother -- which I
> wouldn't debate -- then it's best to just drop those build targets.
> The only worrisome thing I see here is having build targets carried
> from release to release that don't work.

my 5cent:

we have to communicate that PDF build is in a beta stage (for a long
time).

Sphinx-doc's PDF chain was not well maintained for a long time. With
newer versions (started with 1.5 and continued in 1.6) it becomes
better and better. This gives me some hope that there comes a day
where building PDFs is robust enough to use in automatic builds.

As long as we try to support various version of Sphinx shipped by
various distros and at the same time make/need deep (LaTeX)
adjustments, we will find those discussions on the ML.

If you are doubtful about my assessment, compare Sphinx's 
TeX stuff from master 

https://github.com/sphinx-doc/sphinx/tree/master/sphinx/texinputs

with e.g. 1.4.9 tag

https://github.com/sphinx-doc/sphinx/tree/1.4.9/sphinx/texinputs

-- Markus --

Re: [PULL] Docs for 4.13

[Markus Heiser]

Hi Jon,

> Am 04.07.2017 um 06:32 schrieb Linus Torvalds :

[...]

> At the same time, lots of people run a lot of builds, and while I'd
> love to see warnings about docs failures, I am *not* willing to slow
> down my usual build enormously. I run "male allmodconfig" builds
> between every single pull during the merge window, and while it's
> often parallel with me looking at the problems, I don't really want to
> slow the build down too much. And the doc building is still *slow*.
> 
> Is there some fast "just basic sanity checks" that would be more reasonable?
> 
> Because one thing that the switch to sphinx has done is that the doc
> build environment seems saner (tool-wise). So now that kind of thing
> would at least be _possible_ to do in ways I don't think was
> reasonable with docbook.

Sphinx has 'dummy' builder since v1.4. I send a patch with a lintdocs
target [1]. Having a lintdocs target under which we can optimizes
linting might be useful at all and the 'dummy' builder is IMO a
good starting point ..

> And now docbook is finally gone. But sphinx isn't exactly a speed demon 
> either.

The time-saving of the 'dummy' builder against the 'html' builder is about 40%.

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg13248.html

-- Markus --

Re: [PULL] Docs for 4.13

[Markus Heiser]

Hi Jon,

> Am 04.07.2017 um 06:32 schrieb Linus Torvalds :

[...]

> At the same time, lots of people run a lot of builds, and while I'd
> love to see warnings about docs failures, I am *not* willing to slow
> down my usual build enormously. I run "male allmodconfig" builds
> between every single pull during the merge window, and while it's
> often parallel with me looking at the problems, I don't really want to
> slow the build down too much. And the doc building is still *slow*.
> 
> Is there some fast "just basic sanity checks" that would be more reasonable?
> 
> Because one thing that the switch to sphinx has done is that the doc
> build environment seems saner (tool-wise). So now that kind of thing
> would at least be _possible_ to do in ways I don't think was
> reasonable with docbook.

Sphinx has 'dummy' builder since v1.4. I send a patch with a lintdocs
target [1]. Having a lintdocs target under which we can optimizes
linting might be useful at all and the 'dummy' builder is IMO a
good starting point ..

> And now docbook is finally gone. But sphinx isn't exactly a speed demon 
> either.

The time-saving of the 'dummy' builder against the 'html' builder is about 40%.

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg13248.html

-- Markus --

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

[...]

> Am 19.06.2017 um 17:13 schrieb Markus Heiser <markus.hei...@darmarit.de>:
> 
>>> Typically I have a PY_ENV target in my projects, building a virtualenv
>>> in a folder named ./local.

[...]

>> Yeah, IMHO, it makes sense to have something like that at the main build,
>> as an optional feature,

> OK, I will prepare a RFC patch ...

For those who are interested in / lets discuss further in context
of my patch:

  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg12844.html

Thanks!

-- Markus --

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

[...]

> Am 19.06.2017 um 17:13 schrieb Markus Heiser :
> 
>>> Typically I have a PY_ENV target in my projects, building a virtualenv
>>> in a folder named ./local.

[...]

>> Yeah, IMHO, it makes sense to have something like that at the main build,
>> as an optional feature,

> OK, I will prepare a RFC patch ...

For those who are interested in / lets discuss further in context
of my patch:

  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg12844.html

Thanks!

-- Markus --

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

> Am 19.06.2017 um 16:38 schrieb Mauro Carvalho Chehab 
> :
> 
> HI Markus,
> 

Hi Mauro :)

[...]

>> Typically I have a PY_ENV target in my projects, building a virtualenv
>> in a folder named ./local. E.g. in LinuxDoc [1] I use something like this:
>> 
>> PY ?=3
>> PYTHON ?= python$(PY)
>> ..
>> VIRTUALENV = virtualenv --python=$(PYTHON)
>> VTENV_OPTS = "--no-site-packages"
>> PY_ENV = ./local/py$(PY)
> 
> I would split the PATH name on a separate var. This way, if one would
> like to have multiple Sphinx versions, all it would need would be to
> change the directory.
> I would prefer to call it as "./sphinx" (or something similar).

I will take this in mind.

> So, I would do:
> 
>  PY_DIR = ./sphinx
>  PY_ENV = $(PY_DIR)/py$(PY)
> 
> Don't forget to add $PY_DIR directory to .gitignore on your patch.

good point ;)

>> ..
>> quiet_cmd_virtualenv = PYENV $@
>> cmd_virtualenv = \
>>  if [ ! -d "./$(PY_ENV)" ];then \
>>  $(VIRTUALENV) $(VIRTUALENV_VERBOSE) $(VTENV_OPTS) $2; \
>>  else \
>>  echo "using virtualenv from $2"; \
>>  fi
>> ...
>> # to build *local* environment, python and virtualenv from the OS is needed!
>> $(PY_ENV): virtualenv-exe python-exe
>>  $(call cmd,virtualenv,$(PY_ENV))
>>  @$(PY_ENV_BIN)/pip install $(PIP_VERBOSE) -r requirements.txt
> 
> Shouldn't it be using "pip$(PY)" instead?

No, with @$(PY_ENV_BIN)/pip you always use the pip from the environment
and this is always named pip. E.g::

 ./local/py3/bin/pip
 ./local/py2/bin/pip

Or: same as "sphinx-build" command .. there is no sohinx-build3 ;)

> Also, better to seek for requirements on a file under Documentation/sphinx/
> directory.

right, the above was only a sloppily C from what I have in my project.

>> ..
>> 
>> And the sphinxbuild coammand is used from there::
>> 
>> SPHINXBUILD ?= $(PY_ENV_BIN)/sphinx-build
>> 
>> By this I can stick versions packages. E.g. to select last version of
>> RTD theme and Sphinx version 1.5 or upper, I add the following lines to
>> my reqierements.txt::
>> 
>> Sphinx>=1.5  
>> sphinx_rtd_theme
>> 
>> If you are interested, I can prepare a patch, to add such functionality 
>> (as option) to Documents/Makefile (which will be documented in the 
>> doc-guide).
> 
> Yeah, IMHO, it makes sense to have something like that at the main build,
> as an optional feature, e. g. perhaps adding a new make target, like:
> 
>   $ make sphinx_virtenv
> 
> That would create and populate PY_DIR. If $PY_DIR/bin/sphinx-build exists
> and it is executable file, run it. Otherwise, use the system's sphinx, 
> if available.

OK, I will prepare a RFC patch ... I will have time for this only on the
weekend, so have some patience with me.

-- Markus --

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

> Am 19.06.2017 um 16:38 schrieb Mauro Carvalho Chehab 
> :
> 
> HI Markus,
> 

Hi Mauro :)

[...]

>> Typically I have a PY_ENV target in my projects, building a virtualenv
>> in a folder named ./local. E.g. in LinuxDoc [1] I use something like this:
>> 
>> PY ?=3
>> PYTHON ?= python$(PY)
>> ..
>> VIRTUALENV = virtualenv --python=$(PYTHON)
>> VTENV_OPTS = "--no-site-packages"
>> PY_ENV = ./local/py$(PY)
> 
> I would split the PATH name on a separate var. This way, if one would
> like to have multiple Sphinx versions, all it would need would be to
> change the directory.
> I would prefer to call it as "./sphinx" (or something similar).

I will take this in mind.

> So, I would do:
> 
>  PY_DIR = ./sphinx
>  PY_ENV = $(PY_DIR)/py$(PY)
> 
> Don't forget to add $PY_DIR directory to .gitignore on your patch.

good point ;)

>> ..
>> quiet_cmd_virtualenv = PYENV $@
>> cmd_virtualenv = \
>>  if [ ! -d "./$(PY_ENV)" ];then \
>>  $(VIRTUALENV) $(VIRTUALENV_VERBOSE) $(VTENV_OPTS) $2; \
>>  else \
>>  echo "using virtualenv from $2"; \
>>  fi
>> ...
>> # to build *local* environment, python and virtualenv from the OS is needed!
>> $(PY_ENV): virtualenv-exe python-exe
>>  $(call cmd,virtualenv,$(PY_ENV))
>>  @$(PY_ENV_BIN)/pip install $(PIP_VERBOSE) -r requirements.txt
> 
> Shouldn't it be using "pip$(PY)" instead?

No, with @$(PY_ENV_BIN)/pip you always use the pip from the environment
and this is always named pip. E.g::

 ./local/py3/bin/pip
 ./local/py2/bin/pip

Or: same as "sphinx-build" command .. there is no sohinx-build3 ;)

> Also, better to seek for requirements on a file under Documentation/sphinx/
> directory.

right, the above was only a sloppily C from what I have in my project.

>> ..
>> 
>> And the sphinxbuild coammand is used from there::
>> 
>> SPHINXBUILD ?= $(PY_ENV_BIN)/sphinx-build
>> 
>> By this I can stick versions packages. E.g. to select last version of
>> RTD theme and Sphinx version 1.5 or upper, I add the following lines to
>> my reqierements.txt::
>> 
>> Sphinx>=1.5  
>> sphinx_rtd_theme
>> 
>> If you are interested, I can prepare a patch, to add such functionality 
>> (as option) to Documents/Makefile (which will be documented in the 
>> doc-guide).
> 
> Yeah, IMHO, it makes sense to have something like that at the main build,
> as an optional feature, e. g. perhaps adding a new make target, like:
> 
>   $ make sphinx_virtenv
> 
> That would create and populate PY_DIR. If $PY_DIR/bin/sphinx-build exists
> and it is executable file, run it. Otherwise, use the system's sphinx, 
> if available.

OK, I will prepare a RFC patch ... I will have time for this only on the
weekend, so have some patience with me.

-- Markus --

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

> Am 19.06.2017 um 15:46 schrieb Jonathan Corbet :
> 
> On Mon, 19 Jun 2017 06:08:37 -0700
> Christoph Hellwig  wrote:
> 
>> On Mon, Jun 19, 2017 at 08:24:10AM -0300, Mauro Carvalho Chehab wrote:
>>> As the Sphinx build seems very fragile, specially for
>>> PDF output, add a notice about how to use it on a virtual
>>> environment.  
>> 
>> Please don't.  python venv are good at one thing only, and that is
>> making a mess of your python module path.  Don't ever use them on a
>> system that has proper package management.
> 
> Well, they are also good for running specific versions of a given package
> without disrupting the setup of your system as a whole, and when even a
> system with proper package management may not offer the version you need.
> 
> I guess my question with the patch is whether this text really belongs in
> changes.rst, or whether it should be part of the doc-guide.

Hi, here are my 5cents:

Typically I have a PY_ENV target in my projects, building a virtualenv
in a folder named ./local. E.g. in LinuxDoc [1] I use something like this:


PY ?=3
PYTHON ?= python$(PY)
..
VIRTUALENV = virtualenv --python=$(PYTHON)
VTENV_OPTS = "--no-site-packages"
PY_ENV = ./local/py$(PY)
..
quiet_cmd_virtualenv = PYENV $@
cmd_virtualenv = \
if [ ! -d "./$(PY_ENV)" ];then \
$(VIRTUALENV) $(VIRTUALENV_VERBOSE) $(VTENV_OPTS) $2; \
else \
echo "using virtualenv from $2"; \
fi
...
# to build *local* environment, python and virtualenv from the OS is needed!
$(PY_ENV): virtualenv-exe python-exe
$(call cmd,virtualenv,$(PY_ENV))
@$(PY_ENV_BIN)/pip install $(PIP_VERBOSE) -r requirements.txt
..

And the sphinxbuild coammand is used from there::

SPHINXBUILD ?= $(PY_ENV_BIN)/sphinx-build

By this I can stick versions packages. E.g. to select last version of
RTD theme and Sphinx version 1.5 or upper, I add the following lines to
my reqierements.txt::

Sphinx>=1.5
sphinx_rtd_theme

If you are interested, I can prepare a patch, to add such functionality 
(as option) to Documents/Makefile (which will be documented in the doc-guide).

[1] https://github.com/return42/linuxdoc/blob/master/utils/makefile.python

-- Markus --






> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] changes.rst: explain the usage of virtual environment

[Markus Heiser]

> Am 19.06.2017 um 15:46 schrieb Jonathan Corbet :
> 
> On Mon, 19 Jun 2017 06:08:37 -0700
> Christoph Hellwig  wrote:
> 
>> On Mon, Jun 19, 2017 at 08:24:10AM -0300, Mauro Carvalho Chehab wrote:
>>> As the Sphinx build seems very fragile, specially for
>>> PDF output, add a notice about how to use it on a virtual
>>> environment.  
>> 
>> Please don't.  python venv are good at one thing only, and that is
>> making a mess of your python module path.  Don't ever use them on a
>> system that has proper package management.
> 
> Well, they are also good for running specific versions of a given package
> without disrupting the setup of your system as a whole, and when even a
> system with proper package management may not offer the version you need.
> 
> I guess my question with the patch is whether this text really belongs in
> changes.rst, or whether it should be part of the doc-guide.

Hi, here are my 5cents:

Typically I have a PY_ENV target in my projects, building a virtualenv
in a folder named ./local. E.g. in LinuxDoc [1] I use something like this:


PY ?=3
PYTHON ?= python$(PY)
..
VIRTUALENV = virtualenv --python=$(PYTHON)
VTENV_OPTS = "--no-site-packages"
PY_ENV = ./local/py$(PY)
..
quiet_cmd_virtualenv = PYENV $@
cmd_virtualenv = \
if [ ! -d "./$(PY_ENV)" ];then \
$(VIRTUALENV) $(VIRTUALENV_VERBOSE) $(VTENV_OPTS) $2; \
else \
echo "using virtualenv from $2"; \
fi
...
# to build *local* environment, python and virtualenv from the OS is needed!
$(PY_ENV): virtualenv-exe python-exe
$(call cmd,virtualenv,$(PY_ENV))
@$(PY_ENV_BIN)/pip install $(PIP_VERBOSE) -r requirements.txt
..

And the sphinxbuild coammand is used from there::

SPHINXBUILD ?= $(PY_ENV_BIN)/sphinx-build

By this I can stick versions packages. E.g. to select last version of
RTD theme and Sphinx version 1.5 or upper, I add the following lines to
my reqierements.txt::

Sphinx>=1.5
sphinx_rtd_theme

If you are interested, I can prepare a patch, to add such functionality 
(as option) to Documents/Makefile (which will be documented in the doc-guide).

[1] https://github.com/return42/linuxdoc/blob/master/utils/makefile.python

-- Markus --






> 
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/53] Get rid of Docbook

[Markus Heiser]

> Am 19.05.2017 um 01:01 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Thu, 18 May 2017 11:26:08 -0600
> Jonathan Corbet  escreveu:
> 
>> On Tue, 16 May 2017 09:15:52 -0300
>> Mauro Carvalho Chehab  wrote:
>> 
>>> This patch series convert the remaining DocBooks to ReST.
>> 
>> Gotta love this:
>> 
>>> 124 files changed, 7469 insertions(+), 10511 deletions(-)
> 
> Heh, yeah, lots of stuff changed :-)

Hi Mauro,

I guess there are little remains lying around ;) what I have found:

DocBook stuff

./scripts/kernel-doc-xml-ref

docproc and some lines in the Makefile & .gitignore

./scripts/docproc.c
./scripts/.docproc.cmd
./scripts/Makefile
./scripts/.gitignore

Do we need the following any longer? 

./Documentation/sphinx/convert_template.sed
./Documentation/sphinx/post_convert.sed

If you have time for, can you take a look at this?

Thanks!

— Markus —

Re: [PATCH v2 00/53] Get rid of Docbook

[Markus Heiser]

> Am 19.05.2017 um 01:01 schrieb Mauro Carvalho Chehab 
> :
> 
> Em Thu, 18 May 2017 11:26:08 -0600
> Jonathan Corbet  escreveu:
> 
>> On Tue, 16 May 2017 09:15:52 -0300
>> Mauro Carvalho Chehab  wrote:
>> 
>>> This patch series convert the remaining DocBooks to ReST.
>> 
>> Gotta love this:
>> 
>>> 124 files changed, 7469 insertions(+), 10511 deletions(-)
> 
> Heh, yeah, lots of stuff changed :-)

Hi Mauro,

I guess there are little remains lying around ;) what I have found:

DocBook stuff

./scripts/kernel-doc-xml-ref

docproc and some lines in the Makefile & .gitignore

./scripts/docproc.c
./scripts/.docproc.cmd
./scripts/Makefile
./scripts/.gitignore

Do we need the following any longer? 

./Documentation/sphinx/convert_template.sed
./Documentation/sphinx/post_convert.sed

If you have time for, can you take a look at this?

Thanks!

— Markus —

Re: [PATCH 03/36] docs-rst: convert kernel-locking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjust tables with got broken by conversion
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|1 -
> Documentation/DocBook/kernel-locking.tmpl | 2151 -
> Documentation/kernel-hacking/hacking.rst  |  811 +++
> Documentation/kernel-hacking/index.rst|  814 +--
> Documentation/kernel-hacking/locking.rst  | 1453 +++
> 5 files changed, 2268 insertions(+), 2962 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-locking.tmpl
> create mode 100644 Documentation/kernel-hacking/hacking.rst
> create mode 100644 Documentation/kernel-hacking/locking.rst
> 
> ...

> diff --git a/Documentation/kernel-hacking/index.rst 
> b/Documentation/kernel-hacking/index.rst
> index 1a456b60a7cf..b3d8fe56d310 100644
> --- a/Documentation/kernel-hacking/index.rst
> +++ b/Documentation/kernel-hacking/index.rst
> @@ -1,811 +1,5 @@
> -
> -Unreliable Guide To Hacking The Linux Kernel
> -
> +.. toctree::
> +   :maxdepth: 2

Every .rst file needs a title, otherwise  is inserted,
see breadcrumb at the top :

 https://mchehab.fedorapeople.org/kernel_docs/kernel-hacking/index.html

-- Markus --

Re: [PATCH 03/36] docs-rst: convert kernel-locking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjust tables with got broken by conversion
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|1 -
> Documentation/DocBook/kernel-locking.tmpl | 2151 -
> Documentation/kernel-hacking/hacking.rst  |  811 +++
> Documentation/kernel-hacking/index.rst|  814 +--
> Documentation/kernel-hacking/locking.rst  | 1453 +++
> 5 files changed, 2268 insertions(+), 2962 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-locking.tmpl
> create mode 100644 Documentation/kernel-hacking/hacking.rst
> create mode 100644 Documentation/kernel-hacking/locking.rst
> 
> ...

> diff --git a/Documentation/kernel-hacking/index.rst 
> b/Documentation/kernel-hacking/index.rst
> index 1a456b60a7cf..b3d8fe56d310 100644
> --- a/Documentation/kernel-hacking/index.rst
> +++ b/Documentation/kernel-hacking/index.rst
> @@ -1,811 +1,5 @@
> -
> -Unreliable Guide To Hacking The Linux Kernel
> -
> +.. toctree::
> +   :maxdepth: 2

Every .rst file needs a title, otherwise  is inserted,
see breadcrumb at the top :

 https://mchehab.fedorapeople.org/kernel_docs/kernel-hacking/index.html

-- Markus --

Re: [PATCH 03/36] docs-rst: convert kernel-locking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjust tables with got broken by conversion
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|1 -
> Documentation/DocBook/kernel-locking.tmpl | 2151 -
> Documentation/kernel-hacking/hacking.rst  |  811 +++
> Documentation/kernel-hacking/index.rst|  814 +--
> Documentation/kernel-hacking/locking.rst  | 1453 +++
> 5 files changed, 2268 insertions(+), 2962 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-locking.tmpl
> create mode 100644 Documentation/kernel-hacking/hacking.rst
> create mode 100644 Documentation/kernel-hacking/locking.rst
...

There are some misplaced colons at the end of the filenames ...

> +``arch/x86/include/asm/uaccess_32.h:``::
> +
> +#define copy_to_user(to,from,n) \
> +(__builtin_constant_p(n) ?  \
> + __constant_copy_to_user((to),(from),(n)) : \
> + __generic_copy_to_user((to),(from),(n)))
> +

...

> +
> +``arch/sparc/kernel/head.S:``::
> +
> +/*
> + * Sun people can't spell worth damn. "compatability" indeed.
> + * At least we *know* we can't spell, and use a spell-checker.
> + */
> +

...

> +
> +``arch/sparc/lib/checksum.S:``::
> +
> +/* Sun, you just can't beat me, you just can't.  Stop trying,
> + * give up.  I'm serious, I am going to kick the living shit
> + * out of you, game over, lights out.
> + */

-- Markus--

Re: [PATCH 03/36] docs-rst: convert kernel-locking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjust tables with got broken by conversion
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|1 -
> Documentation/DocBook/kernel-locking.tmpl | 2151 -
> Documentation/kernel-hacking/hacking.rst  |  811 +++
> Documentation/kernel-hacking/index.rst|  814 +--
> Documentation/kernel-hacking/locking.rst  | 1453 +++
> 5 files changed, 2268 insertions(+), 2962 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-locking.tmpl
> create mode 100644 Documentation/kernel-hacking/hacking.rst
> create mode 100644 Documentation/kernel-hacking/locking.rst
...

There are some misplaced colons at the end of the filenames ...

> +``arch/x86/include/asm/uaccess_32.h:``::
> +
> +#define copy_to_user(to,from,n) \
> +(__builtin_constant_p(n) ?  \
> + __constant_copy_to_user((to),(from),(n)) : \
> + __generic_copy_to_user((to),(from),(n)))
> +

...

> +
> +``arch/sparc/kernel/head.S:``::
> +
> +/*
> + * Sun people can't spell worth damn. "compatability" indeed.
> + * At least we *know* we can't spell, and use a spell-checker.
> + */
> +

...

> +
> +``arch/sparc/lib/checksum.S:``::
> +
> +/* Sun, you just can't beat me, you just can't.  Stop trying,
> + * give up.  I'm serious, I am going to kick the living shit
> + * out of you, game over, lights out.
> + */

-- Markus--

Re: [PATCH 01/36] docs-rst: convert kernel-hacking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjusted to use ..note and ..warning
> - Minor fixes for it to be parsed without errors
> - Use **bold** for emphasis.
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|2 +-
> Documentation/DocBook/kernel-hacking.tmpl | 1312 -
> Documentation/conf.py |2 +
> Documentation/index.rst   |1 +
> Documentation/kernel-hacking/conf.py  |   10 +
> Documentation/kernel-hacking/index.rst|  794 +
> 6 files changed, 808 insertions(+), 1313 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-hacking.tmpl
> create mode 100644 Documentation/kernel-hacking/conf.py
> create mode 100644 Documentation/kernel-hacking/index.rst
> 
> 


> +:c:func:`cpu_to_be32()`/:c:func:`be32_to_cpu()`/:c:func:`cpu_to_le32()`/:c:func:`le32_to_cpu()`
>  ``include/asm/byteorder.h``
> +---
> +

Hi Mauro, just my bikeshedding:

what do you think, do we really need to refer functions in titles?
As far as I know, there is no use-case where we can get any benefit
from. So I recommend to write titles more simple, e.g.:

  cpu_to_be32()/be32_to_cpu()/cpu_to_le32()/le32_to_cpu() 
include/asm/byteorder.h

.. which is long enough ;)

-- Markus --

Re: [PATCH 01/36] docs-rst: convert kernel-hacking to ReST

[Markus Heiser]

Am 12.05.2017 um 15:59 schrieb Mauro Carvalho Chehab :

> Use pandoc to convert documentation to ReST by calling
> Documentation/sphinx/tmplcvt script.
> 
> - Manually adjusted to use ..note and ..warning
> - Minor fixes for it to be parsed without errors
> - Use **bold** for emphasis.
> 
> Signed-off-by: Mauro Carvalho Chehab 
> ---
> Documentation/DocBook/Makefile|2 +-
> Documentation/DocBook/kernel-hacking.tmpl | 1312 -
> Documentation/conf.py |2 +
> Documentation/index.rst   |1 +
> Documentation/kernel-hacking/conf.py  |   10 +
> Documentation/kernel-hacking/index.rst|  794 +
> 6 files changed, 808 insertions(+), 1313 deletions(-)
> delete mode 100644 Documentation/DocBook/kernel-hacking.tmpl
> create mode 100644 Documentation/kernel-hacking/conf.py
> create mode 100644 Documentation/kernel-hacking/index.rst
> 
> 


> +:c:func:`cpu_to_be32()`/:c:func:`be32_to_cpu()`/:c:func:`cpu_to_le32()`/:c:func:`le32_to_cpu()`
>  ``include/asm/byteorder.h``
> +---
> +

Hi Mauro, just my bikeshedding:

what do you think, do we really need to refer functions in titles?
As far as I know, there is no use-case where we can get any benefit
from. So I recommend to write titles more simple, e.g.:

  cpu_to_be32()/be32_to_cpu()/cpu_to_le32()/le32_to_cpu() 
include/asm/byteorder.h

.. which is long enough ;)

-- Markus --

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Markus Heiser]

Am 21.04.2017 um 01:21 schrieb Mauro Carvalho Chehab :

> - I'm not a python programmer ;-) I just took Markus "generic" kernel-cmd
>  code, hardcoding there a call to the script.
> 
>  With (a lot of) time, I would likely be able to find a solution to add
>  the entire ABI logic there, but, in this case, we would lose the
>  capability of calling the script without Sphinx.

Hi Mauro,

I have no problem with calling the perl script, but your last sentence
is not correct: We can implement a python module, which is used by sphinx
and we can add a command line as well.

--Markus--

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Markus Heiser]

Am 21.04.2017 um 01:21 schrieb Mauro Carvalho Chehab :

> - I'm not a python programmer ;-) I just took Markus "generic" kernel-cmd
>  code, hardcoding there a call to the script.
> 
>  With (a lot of) time, I would likely be able to find a solution to add
>  the entire ABI logic there, but, in this case, we would lose the
>  capability of calling the script without Sphinx.

Hi Mauro,

I have no problem with calling the perl script, but your last sentence
is not correct: We can implement a python module, which is used by sphinx
and we can add a command line as well.

--Markus--

Re: making documentation targets on v4.10 with Fedora 25

[Markus Heiser]

On 23.02.2017 20:44, Jim Davis wrote:

On Thu, Feb 23, 2017 at 2:59 AM, Jani Nikula
 wrote:

On Mon, 20 Feb 2017, Jim Davis  wrote:

For the Sphinx targets, htmldocs, pdfdocs, epubdocs, and cleandocs
failed.  cleandocs works without the O= argument, and arguably the O=
thing isn't very useful with any of these targets, but it is supported
by the top-level Makefile.


Why do you say O= isn't useful with the targets?


You're right, they are.  Silly me.

Something like

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 707c65337ebf..eb4294322150 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -98,7 +98,7 @@ installmandocs:

 cleandocs:
$(Q)rm -rf $(BUILDDIR)
-   $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -C Documentation/media clean
+   $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -C
$(srctree)/Documentation/media clean

 endif # HAVE_SPHINX

seems to get cleandocs working again.


I guess the O= error was already patched:

  http://www.spinics.net/lists/linux-doc/msg42989.html


-- Markus --

Re: making documentation targets on v4.10 with Fedora 25

[Markus Heiser]

On 23.02.2017 20:44, Jim Davis wrote:

On Thu, Feb 23, 2017 at 2:59 AM, Jani Nikula
 wrote:

On Mon, 20 Feb 2017, Jim Davis  wrote:

For the Sphinx targets, htmldocs, pdfdocs, epubdocs, and cleandocs
failed.  cleandocs works without the O= argument, and arguably the O=
thing isn't very useful with any of these targets, but it is supported
by the top-level Makefile.


Why do you say O= isn't useful with the targets?


You're right, they are.  Silly me.

Something like

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 707c65337ebf..eb4294322150 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -98,7 +98,7 @@ installmandocs:

 cleandocs:
$(Q)rm -rf $(BUILDDIR)
-   $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -C Documentation/media clean
+   $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -C
$(srctree)/Documentation/media clean

 endif # HAVE_SPHINX

seems to get cleandocs working again.


I guess the O= error was already patched:

  http://www.spinics.net/lists/linux-doc/msg42989.html


-- Markus --

Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

[Markus Heiser]

Am 26.01.2017 um 20:26 schrieb Jani Nikula :

> On Thu, 26 Jan 2017, Jonathan Corbet  wrote:
>> Give me a new kerneldoc that passes those tests, and I'll happily
>> merge it.  (I have some sympathy with the idea that we should look
>> into other parsers, but I would not hold up a new kerneldoc that
>> passed those tests on this basis alone.)
> 
> I'll just note in passing that having another parser that actually works
> for our needs might be a pink unicorn pony. It might exist, it might
> not, and someone would have to put in the hours to try to find it, tame
> it, and bring it to the kernel. But it would be awesome to
> have. Switching to a homebrew Python parser first does not preclude a
> unicorn hunt later.

Here are my experience about parsing C code and kernel-doc comments.

The reg-expressions divide into two parts:

a.) those parsing "C sources", catching up function prototypes, structs etc. and

b.) those parsing "kernel-doc comments", catching up attribute descriptions,
   cross references etc.

When I developed the py-version in my POC I realized that the reg-expressions
parsing C sources (a.) aren't so bad. They have a long history and are well
tested against kernel' sources (As far as I remember, I added only one regexp
more to match function prototypes).

 This was the time where I looked at some other parsing tool and
 after a day I throw away the idea of using a external parser
 tool, first.

Most problems I have had, was parsing the kernel-doc markup itself. E.g. the
ambiguous attribute markup "* @foo: lorem" and its cross-ref "@foo". The latter
syntax is ambiguous, it fails mostly on new-lines and with strings like
"m...@foo.bar".

When I looked at the whole sources, I also realized that we have two flavors of
kernel-doc markups.

b.1) Those from traditional DocBook where whitespaces aren't markups and

b.2) those which has been rewritten with reST markup in, where whitespaces are a
part of the reST.

But this was only the half truth of b.2) : the 'new' markup did not only
consists of pure reST markup. For convince it is a mix of kernel-doc markup and
reST markup (e.g. remember the cross-ref mentioned above).

I suppose that we will never completely get rid off traditional (b.1), since
this means; changing the whole kernel source ;)

At that time I wanted to implement a parser which has the ability to handle both
flavors. A (undocumented) 'vintage' mode an the user-documented 'reST' mode.
But what is the criteria to switch from one mode to the other?  For this I made
a primitive assumption: every C source file which is used in a ".. kernel-doc::"
directive has to be marked up with the modern reST flavor.

ATM, the py-version of kernel-doc implements the same state machine as the perl
one and the modes are implemented in the same state machine (not perfect but it
worked for me first, suppose we can make it better).

I remember about a very early discussion we had about those modes and I know
that it doesn't find friends in the community (at that time).  May be today we
have more experience and new ideas.

  I really like to see (to work on) a parser with we can parse the
  whole kernel source and generate reST from.

What do you think, is it a bloody idea?

Thanks!

-- Markus --

Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

[Markus Heiser]

Am 26.01.2017 um 20:26 schrieb Jani Nikula :

> On Thu, 26 Jan 2017, Jonathan Corbet  wrote:
>> Give me a new kerneldoc that passes those tests, and I'll happily
>> merge it.  (I have some sympathy with the idea that we should look
>> into other parsers, but I would not hold up a new kerneldoc that
>> passed those tests on this basis alone.)
> 
> I'll just note in passing that having another parser that actually works
> for our needs might be a pink unicorn pony. It might exist, it might
> not, and someone would have to put in the hours to try to find it, tame
> it, and bring it to the kernel. But it would be awesome to
> have. Switching to a homebrew Python parser first does not preclude a
> unicorn hunt later.

Here are my experience about parsing C code and kernel-doc comments.

The reg-expressions divide into two parts:

a.) those parsing "C sources", catching up function prototypes, structs etc. and

b.) those parsing "kernel-doc comments", catching up attribute descriptions,
   cross references etc.

When I developed the py-version in my POC I realized that the reg-expressions
parsing C sources (a.) aren't so bad. They have a long history and are well
tested against kernel' sources (As far as I remember, I added only one regexp
more to match function prototypes).

 This was the time where I looked at some other parsing tool and
 after a day I throw away the idea of using a external parser
 tool, first.

Most problems I have had, was parsing the kernel-doc markup itself. E.g. the
ambiguous attribute markup "* @foo: lorem" and its cross-ref "@foo". The latter
syntax is ambiguous, it fails mostly on new-lines and with strings like
"m...@foo.bar".

When I looked at the whole sources, I also realized that we have two flavors of
kernel-doc markups.

b.1) Those from traditional DocBook where whitespaces aren't markups and

b.2) those which has been rewritten with reST markup in, where whitespaces are a
part of the reST.

But this was only the half truth of b.2) : the 'new' markup did not only
consists of pure reST markup. For convince it is a mix of kernel-doc markup and
reST markup (e.g. remember the cross-ref mentioned above).

I suppose that we will never completely get rid off traditional (b.1), since
this means; changing the whole kernel source ;)

At that time I wanted to implement a parser which has the ability to handle both
flavors. A (undocumented) 'vintage' mode an the user-documented 'reST' mode.
But what is the criteria to switch from one mode to the other?  For this I made
a primitive assumption: every C source file which is used in a ".. kernel-doc::"
directive has to be marked up with the modern reST flavor.

ATM, the py-version of kernel-doc implements the same state machine as the perl
one and the modes are implemented in the same state machine (not perfect but it
worked for me first, suppose we can make it better).

I remember about a very early discussion we had about those modes and I know
that it doesn't find friends in the community (at that time).  May be today we
have more experience and new ideas.

  I really like to see (to work on) a parser with we can parse the
  whole kernel source and generate reST from.

What do you think, is it a bloody idea?

Thanks!

-- Markus --

Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

[Markus Heiser]

Am 25.01.2017 um 21:59 schrieb Jani Nikula :

>> But the problem I see here is, that the perl script generates a
>> reST output which I can't use. As an example we can take a look at
>> the man-page builder I shipped in the series.
> 
> Sorry, I still don't understand *why* you can't use the same rst. Your
> explanation seems to relate to man pages, but man pages come
> *afterwards*, and are a separate improvement. I know you talk about lack
> of proper structure and all that, but *why* can it strictly not be used,
> if the *current* rst clearly can be used?

"afterwards" is the word, that lets me slowly realize, that I have to
stop solving the world's problems with one patch. Now I guess how my
next patch series has to look like. Thanks! ... for being patient with
me.

Before I start, I want to hear your thoughts about the parsing
aspect ...

>>> That said, perhaps having an elegant parser (perhaps based on a compiler
>>> plugin) is incompatible with the idea of making it a bug-for-bug drop-in
>>> replacement of the old one, and it's something we need to think about.

Did you have any suggestions?

-- Markus --