Re: [PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Daniel Vetter]

On Wed, Aug 17, 2016 at 7:44 AM, Markus Heiser
 wrote:
 - Along those lines, is parse-header the right name for this thing?
 "Parsing" isn't necessarily the goal of somebody who uses this directive,
 right?  They want to extract documentation information.  Can we come up
 with a better name?
>>>
>>> Mauro, what is your suggestion and how would we go on in this topic?
>>
>> Maybe we could call it as: "include-c-code-block" or something similar.
>
> Hmm, that's not any better, IMHO ... there is a 'parsed-literal' so, what's
> wrong with a 'parsed-header' directive or for my sake ' parse-c-header'.
> IMHO it is very unspecific what this directive does and it might be changed in
> the near future if someone (e.g. Daniel [1]) see more use cases then the one 
> yet.
>
> [1] https://www.mail-archive.com/linux-media%40vger.kernel.org/msg101129.html

I was wondering more whether we should uplift this to be the canonical
way to document uapi headers. Then we could call it kernel-uapi-header
or whatever, along the lines of our kernel-doc directive. But really
this was just an idea, atm it's a media exclusive feature of our doc
toolchain.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Markus Heiser]

Am 17.08.2016 um 07:44 schrieb Markus Heiser :

> 
> @Daniel: I added you to this discussion. May you are interested in, 
> it is about the parse-headers functionality from Mauro.
> 
> Am 16.08.2016 um 20:22 schrieb Mauro Carvalho Chehab :
> 
>> Em Mon, 15 Aug 2016 10:21:07 +0200
>> Markus Heiser  escreveu:
>> 
>>> Am 14.08.2016 um 20:09 schrieb Jonathan Corbet :
> 
> ...
> 
 but stopped at parse-header.
 At this point, I have a few requests.  These are in approximate order of
 decreasing importance, but they're all important, I think.
>> 
>> After writing the PDF support, I'm starting to think that maybe we
>> should migrate the entire functionality to the Sphinx extension.
>> The rationale is that we won't need to be concerned about output
>> specific escape codes there.
> 
> What do you mean with "output specific escape codes"? This: 
> 
> 
> #
> # Add escape codes for special characters
> #
> $data =~ s,([\_\`\*\<\>\&:\/\|]),\\$1,g;
> 
> $data =~ s,DEPRECATED,**DEPRECATED**,g;
> 
> 
> will be resist, even if you implement it in python.
> 
> 
 - The new directive could really use some ... documentation.  Preferably in
 kernel-documentation.rst with the rest.  What is parse-header, how does
 it differ from kernel-doc, why might a kernel developer doing
 documentation want (or not want) to use it?  That's all pretty obscure
 now.  If we want others to jump onto this little bandwagon of ours, we
 need to make sure it's all really clear.  
>>> 
>>> This could be answered by Mauro.
>> 
>> We use it to allow including an entire header file as-is at the
>> documentation, and cross-reference it with the documents.
>> 
>> IMO, this is very useful to document the ioctl UAPI. There, the most
>> important things to be documented are the ioctl themselves. We don't
>> have any way to document via kernel-doc (as they're #define macros).
>> 
>> Also, when documenting an ioctl, we want to document the structures
>> that are used (most media ioctls use a struct instead of a single value).
>> 
>> So, what we do is that we write a proper description for the ioctl and
>> everything related to it outside the source code. As we want to be
>> sure that everything in the header is documented, we use the
>> "parse-header.pl" (ok, this name really sucks) to create cross-references
>> between the header and the documentation itself.
>> 
>> So, it is actually a script that replaces all occurrences of typedefs,
>> defines, structs, functions, enums into references to the uAPI
>> documentation.
>> 
>> This is is somewhat equivalent to:
>>  .. code-block:: c
>> 
>> Or, even better, it resembles the Doxygen's \example directive:
>>  https://www.stack.nl/~dimitri/doxygen/manual/commands.html#cmdexample
>> 
>> With produces a parsed file, like this one:
>>  https://linuxtv.org/docs/libdvbv5/dvb-fe-tool_8c-example.html
>> 
>> Except that:
>> 
>> 1) It doesn't randomly painting the file;
>> 
>> 2) All places where the a typedef, define, struct, struct member,
>> function or enum exists are replaced by a cross-reference to the
>> documentation (except if explicitly defined to not do that, via a
>> configuration file).
>> 
>> That, plus the nitpick mode at Sphinx, allows us to check what
>> parts of the uAPI file aren't documented.
>> 
 - Along those lines, is parse-header the right name for this thing?
 "Parsing" isn't necessarily the goal of somebody who uses this directive,
 right?  They want to extract documentation information.  Can we come up
 with a better name?  
>>> 
>>> Mauro, what is your suggestion and how would we go on in this topic?
>> 
>> Maybe we could call it as: "include-c-code-block" or something similar.
> 
> Hmm, that's not any better, IMHO ... there is a 'parsed-literal' so, what's
> wrong with a 'parsed-header' directive or for my sake ' parse-c-header'.
> IMHO it is very unspecific what this directive does and it might be changed in
> the near future if someone (e.g. Daniel [1]) see more use cases then the one 
> yet.
> 
> [1] https://www.mail-archive.com/linux-media%40vger.kernel.org/msg101129.html
> 
> -- Markus --
> 

One more thought; parse-header and kernel_doc, are parsing C code.
But both implement their own C-parser ... may it is time to look
for a more general parser solution:

   * http://git.kernel.org/cgit/devel/sparse/sparse.git
   * https://github.com/eliben/pycparser

Only read the documentation of pycparser, sound promising to me.
With something like "parse_file" we could parse the headers

   * https://github.com/eliben/pycparser/blob/master/pycparser/__init__.py#L54

and with subclassing the "CGenerator" class and overwriting some visit-methods:

   * 
https://github.com/eliben/pycparser/blob/master/pycparser/c_generator.py#L12

we can produce the reST output. There is a small example parsing C to
an AST and back to C:

Re: [PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Markus Heiser]

@Daniel: I added you to this discussion. May you are interested in, 
it is about the parse-headers functionality from Mauro.

Am 16.08.2016 um 20:22 schrieb Mauro Carvalho Chehab :

> Em Mon, 15 Aug 2016 10:21:07 +0200
> Markus Heiser  escreveu:
> 
>> Am 14.08.2016 um 20:09 schrieb Jonathan Corbet :

...

>>> but stopped at parse-header.
>>> At this point, I have a few requests.  These are in approximate order of
>>> decreasing importance, but they're all important, I think.
> 
> After writing the PDF support, I'm starting to think that maybe we
> should migrate the entire functionality to the Sphinx extension.
> The rationale is that we won't need to be concerned about output
> specific escape codes there.

What do you mean with "output specific escape codes"? This: 


#
# Add escape codes for special characters
#
$data =~ s,([\_\`\*\<\>\&:\/\|]),\\$1,g;

$data =~ s,DEPRECATED,**DEPRECATED**,g;


will be resist, even if you implement it in python.


>>> - The new directive could really use some ... documentation.  Preferably in
>>> kernel-documentation.rst with the rest.  What is parse-header, how does
>>> it differ from kernel-doc, why might a kernel developer doing
>>> documentation want (or not want) to use it?  That's all pretty obscure
>>> now.  If we want others to jump onto this little bandwagon of ours, we
>>> need to make sure it's all really clear.  
>> 
>> This could be answered by Mauro.
> 
> We use it to allow including an entire header file as-is at the
> documentation, and cross-reference it with the documents.
> 
> IMO, this is very useful to document the ioctl UAPI. There, the most
> important things to be documented are the ioctl themselves. We don't
> have any way to document via kernel-doc (as they're #define macros).
> 
> Also, when documenting an ioctl, we want to document the structures
> that are used (most media ioctls use a struct instead of a single value).
> 
> So, what we do is that we write a proper description for the ioctl and
> everything related to it outside the source code. As we want to be
> sure that everything in the header is documented, we use the
> "parse-header.pl" (ok, this name really sucks) to create cross-references
> between the header and the documentation itself.
> 
> So, it is actually a script that replaces all occurrences of typedefs,
> defines, structs, functions, enums into references to the uAPI
> documentation.
> 
> This is is somewhat equivalent to:
>   .. code-block:: c
> 
> Or, even better, it resembles the Doxygen's \example directive:
>   https://www.stack.nl/~dimitri/doxygen/manual/commands.html#cmdexample
> 
> With produces a parsed file, like this one:
>   https://linuxtv.org/docs/libdvbv5/dvb-fe-tool_8c-example.html
> 
> Except that:
> 
> 1) It doesn't randomly painting the file;
> 
> 2) All places where the a typedef, define, struct, struct member,
> function or enum exists are replaced by a cross-reference to the
> documentation (except if explicitly defined to not do that, via a
> configuration file).
> 
> That, plus the nitpick mode at Sphinx, allows us to check what
> parts of the uAPI file aren't documented.
> 
>>> - Along those lines, is parse-header the right name for this thing?
>>> "Parsing" isn't necessarily the goal of somebody who uses this directive,
>>> right?  They want to extract documentation information.  Can we come up
>>> with a better name?  
>> 
>> Mauro, what is your suggestion and how would we go on in this topic?
> 
> Maybe we could call it as: "include-c-code-block" or something similar.

Hmm, that's not any better, IMHO ... there is a 'parsed-literal' so, what's
wrong with a 'parsed-header' directive or for my sake ' parse-c-header'.
IMHO it is very unspecific what this directive does and it might be changed in
the near future if someone (e.g. Daniel [1]) see more use cases then the one 
yet.

[1] https://www.mail-archive.com/linux-media%40vger.kernel.org/msg101129.html

-- Markus --

> Regards,
> Mauro

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Mauro Carvalho Chehab]

Em Mon, 15 Aug 2016 10:21:07 +0200
Markus Heiser  escreveu:

> Am 14.08.2016 um 20:09 schrieb Jonathan Corbet :
> 
> > On Sat, 13 Aug 2016 16:12:41 +0200
> > Markus Heiser  wrote:
> >   
> >> this series is a consolidation on Jon's docs-next branch. It merges the 
> >> "sphinx
> >> sub-folders" patch [1] and the "parseheaders directive" patch [2] on top of
> >> Jon's docs-next.
> >> 
> >> In sense of consolidation, it also includes:
> >> 
> >> *  doc-rst: add media/conf_nitpick.py
> >> 
> >>   Adds media/conf_nitpick.py from mchehab/docs-next [3].
> >> 
> >> *  doc-rst: migrated media build to parseheaders directive  
> > 
> > OK, I have applied the first five of these,  
> 
> Thanks!
> 
> > but stopped at parse-header.
> > At this point, I have a few requests.  These are in approximate order of
> > decreasing importance, but they're all important, I think.

After writing the PDF support, I'm starting to think that maybe we
should migrate the entire functionality to the Sphinx extension.
The rationale is that we won't need to be concerned about output
specific escape codes there.

> > 
> > - The new directive could really use some ... documentation.  Preferably in
> >  kernel-documentation.rst with the rest.  What is parse-header, how does
> >  it differ from kernel-doc, why might a kernel developer doing
> >  documentation want (or not want) to use it?  That's all pretty obscure
> >  now.  If we want others to jump onto this little bandwagon of ours, we
> >  need to make sure it's all really clear.  
> 
> This could be answered by Mauro.

We use it to allow including an entire header file as-is at the
documentation, and cross-reference it with the documents.

IMO, this is very useful to document the ioctl UAPI. There, the most
important things to be documented are the ioctl themselves. We don't
have any way to document via kernel-doc (as they're #define macros).

Also, when documenting an ioctl, we want to document the structures
that are used (most media ioctls use a struct instead of a single value).

So, what we do is that we write a proper description for the ioctl and
everything related to it outside the source code. As we want to be
sure that everything in the header is documented, we use the
"parse-header.pl" (ok, this name really sucks) to create cross-references
between the header and the documentation itself.

So, it is actually a script that replaces all occurrences of typedefs,
defines, structs, functions, enums into references to the uAPI
documentation.

This is is somewhat equivalent to:
.. code-block:: c

Or, even better, it resembles the Doxygen's \example directive:
https://www.stack.nl/~dimitri/doxygen/manual/commands.html#cmdexample

With produces a parsed file, like this one:
https://linuxtv.org/docs/libdvbv5/dvb-fe-tool_8c-example.html

Except that:

1) It doesn't randomly painting the file;

2) All places where the a typedef, define, struct, struct member,
function or enum exists are replaced by a cross-reference to the
documentation (except if explicitly defined to not do that, via a
configuration file).

That, plus the nitpick mode at Sphinx, allows us to check what
parts of the uAPI file aren't documented.

> > - Along those lines, is parse-header the right name for this thing?
> >  "Parsing" isn't necessarily the goal of somebody who uses this directive,
> >  right?  They want to extract documentation information.  Can we come up
> >  with a better name?  
> 
> Mauro, what is your suggestion and how would we go on in this topic?

Maybe we could call it as: "include-c-code-block" or something similar.

Regards,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Jonathan Corbet]

On Sat, 13 Aug 2016 16:12:41 +0200
Markus Heiser  wrote:

> this series is a consolidation on Jon's docs-next branch. It merges the 
> "sphinx
> sub-folders" patch [1] and the "parseheaders directive" patch [2] on top of
> Jon's docs-next.
> 
> In sense of consolidation, it also includes:
> 
> *  doc-rst: add media/conf_nitpick.py
> 
>Adds media/conf_nitpick.py from mchehab/docs-next [3].
> 
> *  doc-rst: migrated media build to parseheaders directive

OK, I have applied the first five of these, but stopped at parse-header.
At this point, I have a few requests.  These are in approximate order of
decreasing importance, but they're all important, I think.

- The new directive could really use some ... documentation.  Preferably in
  kernel-documentation.rst with the rest.  What is parse-header, how does
  it differ from kernel-doc, why might a kernel developer doing
  documentation want (or not want) to use it?  That's all pretty obscure
  now.  If we want others to jump onto this little bandwagon of ours, we
  need to make sure it's all really clear.

- Along those lines, is parse-header the right name for this thing?
  "Parsing" isn't necessarily the goal of somebody who uses this directive,
  right?  They want to extract documentation information.  Can we come up
  with a better name?

- Can we please try to get the coding style a bit more in line with both
  kernel and Python community norms?  I suspect some people will get grumpy
  if they see this code.  In particular:

- Please try to stick to the 80-column limit when possible.  Python
  makes that a bit harder than C does, and please don't put in
  ridiculous line breaks that make the code worse.  But sticking a bit
  closer to the rule would be good.

- The "#" lines around function/class
  definition lines or other comments are not helpful, please avoid
  them.  Instead, placing a real comment with actual informative text
  above the function/class would be a good thing.  (I could live with
  Python docstrings if you prefer, though I will confess I prefer
  ordinary comments).

- No commas at the beginning of continuation lines, please; that would
  get you yelled at in C code.  If you need to break a function call
  (or whatever), please put the commas at the end of the line as is
  done elsewhere.

  Sorry to poke at nits here, but we want others in the kernel community to
  be able to look at this code, and that will be easier if we stick closer
  to the usual rules.

Thanks,

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 0/7] doc-rst: sphinx sub-folders & parseheaders directive

[Markus Heiser]

From: Markus Heiser 

Hi Jon, Mauro, and Jani,

this series is a consolidation on Jon's docs-next branch. It merges the "sphinx
sub-folders" patch [1] and the "parseheaders directive" patch [2] on top of
Jon's docs-next.

In sense of consolidation, it also includes:

*  doc-rst: add media/conf_nitpick.py

   Adds media/conf_nitpick.py from mchehab/docs-next [3].

*  doc-rst: migrated media build to parseheaders directive

   Remove the media-Makefile and migrate the ".. kernel-include::"
   directive to the new ".. parse-header::" directive.

[1] https://www.mail-archive.com/linux-media@vger.kernel.org/msg101050.html
[2] https://www.mail-archive.com/linux-media@vger.kernel.org/msg101242.html
[3] git.linuxtv.org/mchehab/experimental.git

Thanks,

-- Markus --

Markus Heiser (7):
  doc-rst: generic way to build only sphinx sub-folders
  doc-rst: add stand-alone conf.py to media folder
  doc-rst: add media/conf_nitpick.py
  doc-rst: add stand-alone conf.py to gpu folder
  doc-rst: add docutils config file
  doc-rst: parseheaders directive (inital)
  doc-rst: migrated media build to parseheaders directive

 Documentation/DocBook/Makefile |   7 +
 Documentation/Makefile.sphinx  |  43 +-
 Documentation/conf.py  |   9 +-
 Documentation/docutils.conf|   7 +
 Documentation/gpu/conf.py  |   3 +
 Documentation/index.rst|   7 +-
 Documentation/media/Makefile   |  61 ---
 Documentation/media/audio.h.rst.exceptions |  20 -
 Documentation/media/ca.h.rst.exceptions|  24 -
 Documentation/media/cec.h.rst.exceptions   | 492 ---
 Documentation/media/conf.py|   3 +
 Documentation/media/conf_nitpick.py|  93 
 Documentation/media/dmx.h.rst.exceptions   |  63 ---
 Documentation/media/frontend.h.rst.exceptions  |  47 --
 Documentation/media/index.rst  |  12 +
 Documentation/media/lirc.h.rst.exceptions  |  43 --
 Documentation/media/media.h.rst.exceptions |  30 --
 Documentation/media/net.h.rst.exceptions   |  11 -
 Documentation/media/uapi/cec/cec-header.rst|   4 +-
 Documentation/media/uapi/cec/cec.h.exceptions  | 492 +++
 Documentation/media/uapi/dvb/audio.h.exceptions|  20 +
 Documentation/media/uapi/dvb/audio_h.rst   |   3 +-
 Documentation/media/uapi/dvb/ca.h.exceptions   |  24 +
 Documentation/media/uapi/dvb/ca_h.rst  |   3 +-
 Documentation/media/uapi/dvb/dmx.h.exceptions  |  63 +++
 Documentation/media/uapi/dvb/dmx_h.rst |   3 +-
 Documentation/media/uapi/dvb/frontend.h.exceptions |  47 ++
 Documentation/media/uapi/dvb/frontend_h.rst|   3 +-
 Documentation/media/uapi/dvb/net.h.exceptions  |  11 +
 Documentation/media/uapi/dvb/net_h.rst |   3 +-
 Documentation/media/uapi/dvb/video.h.exceptions|  40 ++
 Documentation/media/uapi/dvb/video_h.rst   |   3 +-
 Documentation/media/uapi/mediactl/media-header.rst |   4 +-
 .../media/uapi/mediactl/media.h.exceptions |  30 ++
 Documentation/media/uapi/rc/lirc-header.rst|   3 +-
 Documentation/media/uapi/rc/lirc.h.exceptions  |  43 ++
 Documentation/media/uapi/v4l/videodev.rst  |   3 +-
 .../media/uapi/v4l/videodev2.h.exceptions  | 535 +
 Documentation/media/video.h.rst.exceptions |  40 --
 Documentation/media/videodev2.h.rst.exceptions | 535 -
 Documentation/sphinx-static/theme_overrides.css|   8 +
 Documentation/sphinx/load_config.py|  33 ++
 Documentation/sphinx/parse-headers.pl  |  17 +-
 Documentation/sphinx/parseheaders.py   | 190 
 44 files changed, 1733 insertions(+), 1402 deletions(-)
 create mode 100644 Documentation/docutils.conf
 create mode 100644 Documentation/gpu/conf.py
 delete mode 100644 Documentation/media/Makefile
 delete mode 100644 Documentation/media/audio.h.rst.exceptions
 delete mode 100644 Documentation/media/ca.h.rst.exceptions
 delete mode 100644 Documentation/media/cec.h.rst.exceptions
 create mode 100644 Documentation/media/conf.py
 create mode 100644 Documentation/media/conf_nitpick.py
 delete mode 100644 Documentation/media/dmx.h.rst.exceptions
 delete mode 100644 Documentation/media/frontend.h.rst.exceptions
 create mode 100644 Documentation/media/index.rst
 delete mode 100644 Documentation/media/lirc.h.rst.exceptions
 delete mode 100644 Documentation/media/media.h.rst.exceptions
 delete mode 100644 Documentation/media/net.h.rst.exceptions
 create mode 100644 Documentation/media/uapi/cec/cec.h.exceptions
 create mode 100644 Documentation/media/uapi/dvb/audio.h.exceptions
 create mode 100644 Documentation/media/uapi/dvb/ca.h.exceptions
 create mode 100644