Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-12-18 Thread Mauro Carvalho Chehab 
Em Wed,  4 Oct 2017 08:48:38 -0300
Mauro Carvalho Chehab  escreveu:

> Right now, it is not possible to document nested struct and nested unions.
> kernel-doc simply ignore them.
> 
> Add support to document them.
> 
> Patches 1 to 6 improve kernel-doc documentation to reflect what
> kernel-doc currently supports and import some stuff from the
> old kernel-doc-nano-HOWTO.txt.
> 
> Patch 7 gets rid of the old documentation (kernel-doc-nano-HOWTO.txt).
> 
> 
> Patch 8 gets rid of the now unused output formats for kernel-doc: since 
> we got rid of all DocBook stuff, we should not need them anymore. The
> reason for dropping it (despite cleaning up), is that it doesn't make 
> sense to invest time on adding new features for formats that aren't
> used anymore.
> 
> Patch 9 improves argument handling, printing script usage if an
> invalid argument is passed and accepting both -cmd and --cmd
> forms.
> 
> Patch 10 changes the default output format to ReST, as this is a way
> more used than man output nowadays.
> 
> Patch 11 solves an issue after ReST conversion, where tabs were not
> properly handled on kernel-doc tags.
> 
> Patch 12 adds support for parsing kernel-doc nested structures and 
> unions.
> 
> Patch 13 does a cleanup, removing $nexted parameter at the
> routines that handle structs.
> 
> Patch 14 Improves warning output by printing the identifier where
> the warning occurred.
> 
> Patch 15 complements patch 12, by adding support for function
> definitions inside nested structures. It is needed by some media
> docs with use such kind of things.
> 
> Patch 16 improves nested struct handling even further, allowing it
> to handle cases where a nested named struct/enum with multiple
> identifiers added (e. g. struct { ... } foo, bar;).
> 
> Patch 17 adds documentation for nested union/struct inside w1_netlink.
> 
> The entire patch series are at my development tree, at:
> https://git.linuxtv.org/mchehab/experimental.git/log/?h=nested-fix-v4b

I'm applying all patches from this series that either didn't have
any comments of whose editorial comments I fully embraced.

The patches that weren't applied from this series are:
0008-media-v4l2-device.h-document-ancillary-macros.patch
0010-media-v4l2-ioctl.h-convert-debug-macros-into-enum-an.patch
0014-media-v4l2-async-simplify-v4l2_async_subdev-structur.patch
0015-media-v4l2-async-better-describe-match-union-at-asyn.patch

I'll send later a patch series with those - probably together with
other documentation patches from the other two series I sent - that
need further reviews.

Thanks,
Mauro

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-10-11 Thread Mauro Carvalho Chehab 
Em Mon, 9 Oct 2017 10:19:55 -0300
Mauro Carvalho Chehab  escreveu:

> > > I really don't want to add that much noise to the docs build; I think it
> > > will defeat any hope of cleaning up the warnings we already have.  I
> > > wonder if we could suppress warnings about nested structs by default, and
> > > add a flag or envar to turn them back on for those who want them?
> > 
> > This is what I vote for: +1
> >   
> > > In truth, now that I look, the real problem is that the warnings are
> > > repeated many times - as in, like 100 times:
> > > 
> > >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> > >> 'wext.ibss' not described in 'wireless_dev'
> > >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> > >> 'wext.ibss' not described in 'wireless_dev'
> > > <107 instances deleted...>
> > >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> > >> 'wext.ibss' not described in 'wireless_dev'
> > > 
> > > That's not something that used to happen, and is certainly not helpful.
> > > Figuring that out would help a lot.  Can I get you to take a look at
> > > what's going on here?
> > 
> > Hi Jon, if you grep for 
> > 
> >  .. kernel-doc:: include/net/cfg80211.h
> > 
> > e.g. by:  grep cfg80211.h Documentation/driver-api/80211/cfg80211.rst | wc 
> > -l
> > you will see, that this directive is used 110 times in one reST file. If you
> > have 5 warnings about the kernel-doc markup in include/net/cfg80211.h you 
> > will
> > get 550 warnings about kernel-doc markup just for just one source code file.
> > 
> > This is because the kernel-doc parser is an external process which is called
> > (in this example) 110 times for one reST file and one source code file. If 
> > include/net/cfg80211.h is referred in other reST files, we got accordingly
> > more and more warnings .. thats really noise.   
> 
> I guess the solution here is simple: if any output selection argument
> is passed (-export, -internal, -function, -nofunction), only report
> warnings for things that match the output criteria. 
> 
> Not sure how easy is to implement that. I'll take a look.

It is actually very easy to suppress those warnings. See the enclosed
proof of concept patch.

Please notice that this patch is likely incomplete: all it does is that,
if --function or --nofunction is used, it won't print any warnings
for structs or enums.

I didn't check yet if this is not making it too silent.

If we're going to follow this way, IMHO, the best would be to define
a warning function and move the needed checks for $output_selection
and for $function for it to do the right thing, printing warnings only
for the stuff that will be output.

Jon,

Please let me know if you prefer go though this way or if, instead, you
opt to replace the kernel-doc perl script by a Sphinx module.

If you decide to keep with the perl script for now, I can work on an
improved version of this PoC.


Regards,
Mauro

scripts: kernel-doc: shut up enum/struct warnings if parsing only functions

There are two command line parameters that restrict the kernel-doc output
to output only functions. If this is used, it doesn't make any sense to
produce warnings about enums and structs. So, shut up such warnings.

Signed-off-by: Mauro Carvalho Chehab 

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 049044d95c0e..b4eebea6a8d6 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1138,16 +1138,20 @@ sub dump_enum($$) {
push @parameterlist, $arg;
if (!$parameterdescs{$arg}) {
$parameterdescs{$arg} = $undescribed;
-   print STDERR "${file}:$.: warning: Enum value '$arg' ".
-   "not described in enum '$declaration_name'\n";
+   if ($output_selection != OUTPUT_INCLUDE &&
+   $output_selection != OUTPUT_EXCLUDE) {
+   print STDERR "${file}:$.: warning: Enum value '$arg' 
not described in enum '$declaration_name'\n";
+   }
}
$_members{$arg} = 1;
}
 
while (my ($k, $v) = each %parameterdescs) {
if (!exists($_members{$k})) {
-print STDERR "${file}:$.: warning: Excess enum value " .
- "'$k' description in '$declaration_name'\n";
+   if ($output_selection != OUTPUT_INCLUDE &&
+   $output_selection != OUTPUT_EXCLUDE) {
+print STDERR "${file}:$.: warning: Excess enum value '$k' 
description in '$declaration_name'\n";
+   }
}
 }
 
@@ -1353,9 +1357,12 @@ sub push_parameter() {
if (!defined $parameterdescs{$param} && $param !~ /^#/) {
$parameterdescs{$param} = $undescribed;
 
-   print STDERR
- "${file}:$.: warning: Function parameter or member 
'$param' not described in '$declaration_name'\n";
-

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-10-09 Thread Mauro Carvalho Chehab 
Em Sun, 8 Oct 2017 12:07:29 +0200
Markus Heiser  escreveu:

> > Am 07.10.2017 um 18:34 schrieb Jonathan Corbet :
> > 
> > On Wed,  4 Oct 2017 08:48:38 -0300
> > Mauro Carvalho Chehab  wrote:
> >   
> >> Right now, it is not possible to document nested struct and nested unions.
> >> kernel-doc simply ignore them.
> >> 
> >> Add support to document them.  
> > 
> > So I've finally found some time to actually look at these; sorry for being
> > so absent from the discussion.  I plead $EXCUSES...

Thanks for looking into it!

> > 
> > Some overall impressions:
> > 
> > - Seems like something we want.
> > - I love hacking all the cruft out of kernel-doc; I've been meaning to
> >  do that for a bit.
> > - It would sure be nice to restore proper man-page generation rather than
> >  documenting a hack with a perl script.  Someday.
> > 
> > I have one real complaint, though: with these patches applied, a "make
> > htmldocs" generates about 5500 (!) more warnings than it did before.  Over
> > the last couple of months, I put a bit of focused time into reducing
> > warnings, and managed to get rid of 20-30 of them.  Now I feel despondent.

Yeah, I know the feeling...

> > 
> > I really don't want to add that much noise to the docs build; I think it
> > will defeat any hope of cleaning up the warnings we already have.  I
> > wonder if we could suppress warnings about nested structs by default, and
> > add a flag or envar to turn them back on for those who want them?  
> 
> This is what I vote for: +1
> 
> > In truth, now that I look, the real problem is that the warnings are
> > repeated many times - as in, like 100 times:
> >   
> >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> >> 'wext.ibss' not described in 'wireless_dev'
> >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> >> 'wext.ibss' not described in 'wireless_dev'  
> > <107 instances deleted...>  
> >> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> >> 'wext.ibss' not described in 'wireless_dev'  
> > 
> > That's not something that used to happen, and is certainly not helpful.
> > Figuring that out would help a lot.  Can I get you to take a look at
> > what's going on here?  
> 
> Hi Jon, if you grep for 
> 
>  .. kernel-doc:: include/net/cfg80211.h
> 
> e.g. by:  grep cfg80211.h Documentation/driver-api/80211/cfg80211.rst | wc -l
> you will see, that this directive is used 110 times in one reST file. If you
> have 5 warnings about the kernel-doc markup in include/net/cfg80211.h you will
> get 550 warnings about kernel-doc markup just for just one source code file.
> 
> This is because the kernel-doc parser is an external process which is called
> (in this example) 110 times for one reST file and one source code file. If 
> include/net/cfg80211.h is referred in other reST files, we got accordingly
> more and more warnings .. thats really noise. 

I guess the solution here is simple: if any output selection argument
is passed (-export, -internal, -function, -nofunction), only report
warnings for things that match the output criteria. 

Not sure how easy is to implement that. I'll take a look.

> So what I see is, that a major part of such noise is self made, it was one 
> major
> goal when I started with the python implementation: run kernel-doc parser in 
> the 
> Sphinx process, cache the results and use it in all reST files where it is 
> required
> from a kernel-doc directive.
> 
> Since there is also a man page solution in the py- version I vote to merge the
> py-version into the kernel (ATM man is similar to what we had in DocBook and
> it is expandable). If you want to have a preview of the result, try this 
> patch:
> 
>  https://return42.github.io/linuxdoc/linux.html
> 
> The only drawback of the py version I see is, that Mauro prefers perl and I do
> not want to lose him as an contributer to the kernel-doc parser. IMO he is an
> experienced C programmer with an excellent regexpr knowledge. This is, what is
> needed for this job. May be we can motivate him to give python one more try,
> this is what I hope.
> 
> The py version includes all patches we had to the perl version, but I also
> know, that you are not 100% compliant with the coding style, this could
> all be fixed in a RFC round. Excuse me if I annoying with this solution;
> IMO it contains what we need, if we really want to make a step forward.

I can probably submit regex patches, no matter if it is perl or python.

I actually submitted several patches patchwork, in the past, and I'm doing
lately some contributions to Solaar, with is written in python, in order
to fix support for my mice.

The thing is that it takes me more time to write code in Python, as I
never had the time to study the language. So, I learned what I needed
there by coding.

Thanks,
Mauro

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-10-08 Thread Markus Heiser 

> Am 07.10.2017 um 18:34 schrieb Jonathan Corbet :
> 
> On Wed,  4 Oct 2017 08:48:38 -0300
> Mauro Carvalho Chehab  wrote:
> 
>> Right now, it is not possible to document nested struct and nested unions.
>> kernel-doc simply ignore them.
>> 
>> Add support to document them.
> 
> So I've finally found some time to actually look at these; sorry for being
> so absent from the discussion.  I plead $EXCUSES...
> 
> Some overall impressions:
> 
> - Seems like something we want.
> - I love hacking all the cruft out of kernel-doc; I've been meaning to
>  do that for a bit.
> - It would sure be nice to restore proper man-page generation rather than
>  documenting a hack with a perl script.  Someday.
> 
> I have one real complaint, though: with these patches applied, a "make
> htmldocs" generates about 5500 (!) more warnings than it did before.  Over
> the last couple of months, I put a bit of focused time into reducing
> warnings, and managed to get rid of 20-30 of them.  Now I feel despondent.
> 
> I really don't want to add that much noise to the docs build; I think it
> will defeat any hope of cleaning up the warnings we already have.  I
> wonder if we could suppress warnings about nested structs by default, and
> add a flag or envar to turn them back on for those who want them?

This is what I vote for: +1

> In truth, now that I look, the real problem is that the warnings are
> repeated many times - as in, like 100 times:
> 
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> <107 instances deleted...>
>> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
>> 'wext.ibss' not described in 'wireless_dev'
> 
> That's not something that used to happen, and is certainly not helpful.
> Figuring that out would help a lot.  Can I get you to take a look at
> what's going on here?

Hi Jon, if you grep for 

 .. kernel-doc:: include/net/cfg80211.h

e.g. by:  grep cfg80211.h Documentation/driver-api/80211/cfg80211.rst | wc -l
you will see, that this directive is used 110 times in one reST file. If you
have 5 warnings about the kernel-doc markup in include/net/cfg80211.h you will
get 550 warnings about kernel-doc markup just for just one source code file.

This is because the kernel-doc parser is an external process which is called
(in this example) 110 times for one reST file and one source code file. If 
include/net/cfg80211.h is referred in other reST files, we got accordingly
more and more warnings .. thats really noise. 

So what I see is, that a major part of such noise is self made, it was one major
goal when I started with the python implementation: run kernel-doc parser in 
the 
Sphinx process, cache the results and use it in all reST files where it is 
required
from a kernel-doc directive.

Since there is also a man page solution in the py- version I vote to merge the
py-version into the kernel (ATM man is similar to what we had in DocBook and
it is expandable). If you want to have a preview of the result, try this patch:

 https://return42.github.io/linuxdoc/linux.html

The only drawback of the py version I see is, that Mauro prefers perl and I do
not want to lose him as an contributer to the kernel-doc parser. IMO he is an
experienced C programmer with an excellent regexpr knowledge. This is, what is
needed for this job. May be we can motivate him to give python one more try,
this is what I hope.

The py version includes all patches we had to the perl version, but I also
know, that you are not 100% compliant with the coding style, this could
all be fixed in a RFC round. Excuse me if I annoying with this solution;
IMO it contains what we need, if we really want to make a step forward.

Thanks,

-- Markus --

> 
> Thanks,
> 
> jon
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-10-07 Thread Jonathan Corbet 
On Wed,  4 Oct 2017 08:48:38 -0300
Mauro Carvalho Chehab  wrote:

> Right now, it is not possible to document nested struct and nested unions.
> kernel-doc simply ignore them.
> 
> Add support to document them.

So I've finally found some time to actually look at these; sorry for being
so absent from the discussion.  I plead $EXCUSES...

Some overall impressions:

 - Seems like something we want.
 - I love hacking all the cruft out of kernel-doc; I've been meaning to
   do that for a bit.
 - It would sure be nice to restore proper man-page generation rather than
   documenting a hack with a perl script.  Someday.

I have one real complaint, though: with these patches applied, a "make
htmldocs" generates about 5500 (!) more warnings than it did before.  Over
the last couple of months, I put a bit of focused time into reducing
warnings, and managed to get rid of 20-30 of them.  Now I feel despondent.

I really don't want to add that much noise to the docs build; I think it
will defeat any hope of cleaning up the warnings we already have.  I
wonder if we could suppress warnings about nested structs by default, and
add a flag or envar to turn them back on for those who want them?

In truth, now that I look, the real problem is that the warnings are
repeated many times - as in, like 100 times:

> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> 'wext.ibss' not described in 'wireless_dev'
> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> 'wext.ibss' not described in 'wireless_dev'
<107 instances deleted...>
> ./include/net/cfg80211.h:4115: warning: Function parameter or member 
> 'wext.ibss' not described in 'wireless_dev'

That's not something that used to happen, and is certainly not helpful.
Figuring that out would help a lot.  Can I get you to take a look at
what's going on here?

Thanks,

jon

[PATCH v3 00/17] kernel-doc: add supported to document nested structs/

2017-10-04 Thread Mauro Carvalho Chehab 

Right now, it is not possible to document nested struct and nested unions.
kernel-doc simply ignore them.

Add support to document them.

Patches 1 to 6 improve kernel-doc documentation to reflect what
kernel-doc currently supports and import some stuff from the
old kernel-doc-nano-HOWTO.txt.

Patch 7 gets rid of the old documentation (kernel-doc-nano-HOWTO.txt).


Patch 8 gets rid of the now unused output formats for kernel-doc: since 
we got rid of all DocBook stuff, we should not need them anymore. The
reason for dropping it (despite cleaning up), is that it doesn't make 
sense to invest time on adding new features for formats that aren't
used anymore.

Patch 9 improves argument handling, printing script usage if an
invalid argument is passed and accepting both -cmd and --cmd
forms.

Patch 10 changes the default output format to ReST, as this is a way
more used than man output nowadays.

Patch 11 solves an issue after ReST conversion, where tabs were not
properly handled on kernel-doc tags.

Patch 12 adds support for parsing kernel-doc nested structures and 
unions.

Patch 13 does a cleanup, removing $nexted parameter at the
routines that handle structs.

Patch 14 Improves warning output by printing the identifier where
the warning occurred.

Patch 15 complements patch 12, by adding support for function
definitions inside nested structures. It is needed by some media
docs with use such kind of things.

Patch 16 improves nested struct handling even further, allowing it
to handle cases where a nested named struct/enum with multiple
identifiers added (e. g. struct { ... } foo, bar;).

Patch 17 adds documentation for nested union/struct inside w1_netlink.

The entire patch series are at my development tree, at:
https://git.linuxtv.org/mchehab/experimental.git/log/?h=nested-fix-v4b

--

v3:
- rebased on the top of docs-next branch at docs git tree;
- patches reordered to a more logical sequence;
- Change script to produce ReST format by default;
- Improve argument handling;
- Accept structs with multiple identifiers.


v2:
  - solved some issues that Randy pointed;
  - added patch 10 as suggested by Markus;
  - Fixed some bugs on patch 9, after parsing nested structs
   on media subsystem;
  - added patch 11 with a warning improvement fixup;
  - added patch 12 in order to handle function parameters
   on nested structures, due to DVB demux kAPI.

v1:
  - original version.

Mauro Carvalho Chehab (17):
  docs: kernel-doc.rst: better describe kernel-doc arguments
  docs: kernel-doc.rst: improve private members description
  docs: kernel-doc.rst: improve function documentation section
  docs: kernel-doc.rst: improve structs chapter
  docs: kernel-doc.rst: improve typedef documentation
  docs: kernel-doc.rst: add documentation about man pages
  docs: get rid of kernel-doc-nano-HOWTO.txt
  scripts: kernel-doc: get rid of unused output formats
  scripts: kernel-doc: improve argument handling
  scripts: kernel-doc: change default to ReST format
  scripts: kernel-doc: replace tabs by spaces
  scripts: kernel-doc: parse next structs/unions
  scripts: kernel-doc: get rid of $nested parameter
  scripts: kernel-doc: print the declaration name on warnings
  scripts: kernel-doc: handle nested struct function arguments
  scripts: kernel-doc: improve nested logic to handle multiple
identifiers
  w1_netlink.h: add support for nested structs

 Documentation/00-INDEX  |2 -
 Documentation/doc-guide/kernel-doc.rst  |  360 +---
 Documentation/kernel-doc-nano-HOWTO.txt |  322 ---
 drivers/w1/w1_netlink.h |6 +-
 scripts/kernel-doc  | 1441 ---
 5 files changed, 414 insertions(+), 1717 deletions(-)
 delete mode 100644 Documentation/kernel-doc-nano-HOWTO.txt

-- 
2.13.6