Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Sat, 22 Oct 2016 08:56:29 -0200
Mauro Carvalho Chehab  escreveu:

> How many special-case commands are we going to need to run?  Does it
> > really need to go beyond what parse-headers is doing now?  
> 
> Right now, we have actually 2 such cases: kernel-doc and parse-headers.
> 
> I also use a set of perl scripts that I manually run, on each Kernel
> version, to update the cards list of several drivers. It is a total of
> 9 such scripts.

I ended by adding a few more scripts, as ivtv, tm6000 and gspca
cardlists were missing. As result, all three had some documentation
gaps.

As an example of such scripts, I added a PoC patch at my development
tree, at the lkml-books-v2 branch:

https://git.linuxtv.org/mchehab/experimental.git/log/?h=lkml-books-v2

Patch follows.

As I mentioned before, usually at -rc6 or -rc7, when I remember that a driver
with a cardlist was touched, I run those scripts to make sure that the
driver documentation will reflect the board additions. The scripts
run quick, so, It doesn't hurt much from my side to keep using the same
procedure. There are just a few drawbacks with this:

1) from time to time, third-party patches touch those cardlist tables,
usually not doing exactly the same thing as the script. So, when I run
the script manually, it will replace by the one at the cards definition,
producing an extra patch;

2) sometimes, I just forget running it;

3) as those cardlists are auto-generated, they need to be on a separate
rst file. If we use the kernel-cmd directive, I could write some text
before the cardlist database, and update it independently from the
scripts.

So, I prefer if we could generate it dynamically while building the
documentation.

Thanks,
Mauro

[RFC] [media] cardlists: dynamically generate cardlists from source

Instead of manually patch the cardlists by hand, on every Kernel
version, use scripts to auto-generate them when building the
documentation.

Signed-off-by: Mauro Carvalho Chehab 

diff --git a/Documentation/media/v4l-drivers/au0828-cardlist.rst 
b/Documentation/media/v4l-drivers/au0828-cardlist.rst
index 82d2567bc7c1..afaef88969b6 100644
--- a/Documentation/media/v4l-drivers/au0828-cardlist.rst
+++ b/Documentation/media/v4l-drivers/au0828-cardlist.rst
@@ -1,13 +1 @@
-AU0828 cards list
-=
-
-=== == 
===
-Card number Card name  USB IDs
-=== == 
===
-0   Unknown board
-1   Hauppauge HVR950Q  2040:7200, 2040:7210, 2040:7217, 
2040:721b, 2040:721e, 2040:721f, 2040:7280, 0fd9:0008, 2040:7260, 2040:7213, 
2040:7270
-2   Hauppauge HVR850   2040:7240
-3   DViCO FusionHDTV USB   0fe9:d620
-4   Hauppauge HVR950Q rev xxF8 2040:7201, 2040:7211, 2040:7281
-5   Hauppauge Woodbury 05e1:0480, 2040:8200
-=== == 
===
+.. kernel-cmd:: au0828-cardlist.pl 
$srctree/drivers/media/usb/au0828/au0828-cards.h 
$srctree/drivers/media/usb/au0828/au0828-cards.c
diff --git a/Documentation/media/v4l-drivers/bttv-cardlist.rst 
b/Documentation/media/v4l-drivers/bttv-cardlist.rst
index 28a01cd6cf2e..3ae5099ffdd3 100644
--- a/Documentation/media/v4l-drivers/bttv-cardlist.rst
+++ b/Documentation/media/v4l-drivers/bttv-cardlist.rst
@@ -1,174 +1 @@
-BTTV cards list
-===
-
-=== 
=
 
==
-Card number Card name  
   PCI IDs
-=== 
=
 
==
-0*** UNKNOWN/GENERIC ***
-1   MIRO PCTV
-2   Hauppauge (bt848)
-3   STB, Gateway P/N 6000699 (bt848)
-4   Intel Create and Share PCI/ Smart Video Recorder III
-5   Diamond DTV2000
-6   AVerMedia TVPhone
-7   MATRIX-Vision MV-Delta
-8   Lifeview FlyVideo II (Bt848) LR26 / MAXI TV Video PCI2 LR26
-9   IMS/IXmicro TurboTV
-10  Hauppauge (bt878)  
   0070:13eb, 0070:3900, 2636:10b4
-11  MIRO PCTV pro
-12 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Sat, 22 Oct 2016 09:04:21 -0600
Jonathan Corbet  escreveu:

> On Sat, 22 Oct 2016 08:56:29 -0200
> Mauro Carvalho Chehab  wrote:
> 
> > The security implications will be the same if either coded as an
> > "ioctl()" or as "syscall", the scripts should be audited. Actually,
> > if we force the need of a "syscall" for every such script, we have
> > twice the code to audit, as both the Sphinx extension and the perl
> > script will need to audit, increasing the attack surface.  
> 
> Just addressing this one part for the moment.  Clearly I've not explained
> my concern well.
> 
> The kernel-cmd directive makes it possible for *any* RST file to run
> arbitrary shell commands.  I'm not concerned about the scripts we add, I
> hope we can get those right.  I'm worried about what slips in via a tweak
> to some obscure .rst file somewhere.
> 
> A quick check says that 932 commits touched Documentation/ since 4.8.  A
> lot of those did not come from either my tree or yours; *everybody* messes
> around in the docs tree.  People know to look closely at changes to
> makefiles and such; nobody thinks to examine documentation changes for
> such things. I think there are attackers out there who would like the
> opportunity to run commands in the settings where kernels are built; we
> need to think pretty hard before we make that easier to do.
> 
> See what I'm getting at here?

Yes, I see your point, but IMHO, if we add an extra logic at kernel-cmd to
restrict it to run scripts *only* from an specific directory 
(like Documentation/sphinx), then you'll have a better control.
There were only 37 commits there, from you, me and Jani (and, AFAIKT, all
of them were sent to the linux-doc ML for review):

$ git log --pretty=fuller Documentation/sphinx|grep Commit:|sort|uniq -c
 11 Commit: Jani Nikula 
 10 Commit: Jonathan Corbet 
 16 Commit: Mauro Carvalho Chehab 

With is, btw, the same rule we have for a Sphinx extension. 

If you thing this isn't enough, we could also add some logic at
checkpatch.pl to check for the usage of Sphinx extensions.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Markus Heiser]

Am 22.10.2016 um 17:04 schrieb Jonathan Corbet :

> On Sat, 22 Oct 2016 08:56:29 -0200
> Mauro Carvalho Chehab  wrote:
> 
>> The security implications will be the same if either coded as an
>> "ioctl()" or as "syscall", the scripts should be audited. Actually,
>> if we force the need of a "syscall" for every such script, we have
>> twice the code to audit, as both the Sphinx extension and the perl
>> script will need to audit, increasing the attack surface.
> 
> Just addressing this one part for the moment.  Clearly I've not explained
> my concern well.
> 
> The kernel-cmd directive makes it possible for *any* RST file to run
> arbitrary shell commands.  I'm not concerned about the scripts we add, I
> hope we can get those right.  I'm worried about what slips in via a tweak
> to some obscure .rst file somewhere.

> 
> A quick check says that 932 commits touched Documentation/ since 4.8.  A
> lot of those did not come from either my tree or yours; *everybody* messes
> around in the docs tree.  People know to look closely at changes to
> makefiles and such; nobody thinks to examine documentation changes for
> such things. I think there are attackers out there who would like the
> opportunity to run commands in the settings where kernels are built; we
> need to think pretty hard before we make that easier to do.

Hmm, I understand, it would not be good, if every .rst (and c-file with
kernel-doc in) becomes the potential to be a *executable* ...

Might it be a compromise, if we reduce kernel-cmd to start only selected
scripts from Documentation/sphinx ?

-- Markus --

--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jonathan Corbet]

On Sat, 22 Oct 2016 08:56:29 -0200
Mauro Carvalho Chehab  wrote:

> The security implications will be the same if either coded as an
> "ioctl()" or as "syscall", the scripts should be audited. Actually,
> if we force the need of a "syscall" for every such script, we have
> twice the code to audit, as both the Sphinx extension and the perl
> script will need to audit, increasing the attack surface.

Just addressing this one part for the moment.  Clearly I've not explained
my concern well.

The kernel-cmd directive makes it possible for *any* RST file to run
arbitrary shell commands.  I'm not concerned about the scripts we add, I
hope we can get those right.  I'm worried about what slips in via a tweak
to some obscure .rst file somewhere.

A quick check says that 932 commits touched Documentation/ since 4.8.  A
lot of those did not come from either my tree or yours; *everybody* messes
around in the docs tree.  People know to look closely at changes to
makefiles and such; nobody thinks to examine documentation changes for
such things. I think there are attackers out there who would like the
opportunity to run commands in the settings where kernels are built; we
need to think pretty hard before we make that easier to do.

See what I'm getting at here?

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Fri, 21 Oct 2016 16:05:43 -0600
Jonathan Corbet  escreveu:

> On Tue, 11 Oct 2016 09:26:48 +0200
> Markus Heiser  wrote:
> 
> > If the kernel-cmd directive gets acked, I will add a description to
> > kernel-documentation.rst and I request Mauro to document the 
> > parse-headers.pl
> > also.
> > 
> > But, let's hear what Jon says.  
> 
> Sigh.
> 
> I've been shunting this discussion aside while I dug out from other
> things.  Now I've pushed through the whole thing; I'm still not sure what
> I think is the best thing to do.
> 
> kernel-cmd scares me.  It looks like the ioctl() of documentation
> building; people will be able to add all kinds of wild things and it will
> take a lot of attention to catch them.  I think we could make things
> pretty messy in a real hurry.  And yes, I do think we should consider the
> security aspects of it; we're talking about adding another shell
> code-execution context in the kernel build, and that can only make things
> harder to audit.
> 
> OTOH, forcing things into dedicated Sphinx extensions doesn't necessarily
> fix the problem.  We're adding system calls rather than ioctl() commands,
> let's say, but we're still adding long-term maintenance complications.

The security implications will be the same if either coded as an
"ioctl()" or as "syscall", the scripts should be audited. Actually,
if we force the need of a "syscall" for every such script, we have
twice the code to audit, as both the Sphinx extension and the perl
script will need to audit, increasing the attack surface.

> How many special-case commands are we going to need to run?  Does it
> really need to go beyond what parse-headers is doing now?

Right now, we have actually 2 such cases: kernel-doc and parse-headers.

I also use a set of perl scripts that I manually run, on each Kernel
version, to update the cards list of several drivers. It is a total of
9 such scripts. See below for one of such example[1]

[1] this is actually new version of the script that produces the
Documentation/media/v4l-drivers/em28xx-cardlist.rst file, using a
flat table. The one used up to 4.9-rc1 were using a code-block tag.

The problem those scripts address is that there are several media boards that
don't have an unique PCI or USB ID, because they don't have eeproms, or
because the vendor sells different models with the very same ID
(for example, using one tuner for US market and another one for EU).

As there are no ways for the Kernel to auto-identify such boards,
the user need to pass two numbers for the driver, in order to set the
card and the tuner type, when modprobing the driver. 

Such drivers have those two parameters:

parm:   tuner:tuner type (array of int)
parm:   card:card type (array of int)

The tuner parameter define what TV tuner is present at the device(s).

The card number define things like GPIOs used to turn on the several
components of the board{s}, and describe the boards' capabilities.

Thankfully, we only have a couple of new additions to those drivers
per Kernel release, so, right now, I take myself the burden of manually
run those scripts, but if we could add it to the building system, that would
reduce a a little bit my workload, with is a good thing :) It would also
prevent the risk of releasing a Kernel with outdated information, with
sometimes happen.

I've no idea if we'll either need scripts or not on other non-media docs.

>  Let's really
> think about what the other use cases might be and whether we can do
> without them. I'm still thoroughly unconvinced about the utility of
> incorporating, say, the MAINTAINERS file into the formatted docs, for
> example, so I'm not yet convinced that making that easier to do is
> something we need.

With regards to MAINTAINERS, what's your suggestion? 

Right now, there are cross-references for it in 58 places, being 25 on a
documentation file, including README and REPORTING-BUGS, with has a broader
audience.

IMHO, we should provide a section somehow at the documentation
in order to give directions if someone reads one of such documents.

Also, the section "List of maintainers and how to submit kernel changes"
Has a short list of rules for people to follow when submitting a patch,
with could make sense at the process book.

So, I can see some ways to handle it:

1) convert everything, including the database to ReST and include
   it at documentation;

2) convert it to ReST format, except for the database, and use some
   extension to parse it, like on my PoC patches;

3) move the "how to submit kernel changes" list to process/ and
   create a small guide at the admin-guide/ book to be used as
   cross-reference to MAINTAINERS, explaining how to get the
   maintainer for a file (e. g. running ./scripts/get-maintainer.pl);

4) do nothing and assume that anyone capable of reading the
   README or REPORTING-BUGS file would guess what "MAINTAINERS" mean
   (or that if then won't 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jonathan Corbet]

On Tue, 11 Oct 2016 09:26:48 +0200
Markus Heiser  wrote:

> If the kernel-cmd directive gets acked, I will add a description to
> kernel-documentation.rst and I request Mauro to document the parse-headers.pl
> also.
> 
> But, let's hear what Jon says.

Sigh.

I've been shunting this discussion aside while I dug out from other
things.  Now I've pushed through the whole thing; I'm still not sure what
I think is the best thing to do.

kernel-cmd scares me.  It looks like the ioctl() of documentation
building; people will be able to add all kinds of wild things and it will
take a lot of attention to catch them.  I think we could make things
pretty messy in a real hurry.  And yes, I do think we should consider the
security aspects of it; we're talking about adding another shell
code-execution context in the kernel build, and that can only make things
harder to audit.

OTOH, forcing things into dedicated Sphinx extensions doesn't necessarily
fix the problem.  We're adding system calls rather than ioctl() commands,
let's say, but we're still adding long-term maintenance complications.

How many special-case commands are we going to need to run?  Does it
really need to go beyond what parse-headers is doing now?  Let's really
think about what the other use cases might be and whether we can do
without them. I'm still thoroughly unconvinced about the utility of
incorporating, say, the MAINTAINERS file into the formatted docs, for
example, so I'm not yet convinced that making that easier to do is
something we need.

Not much clarity here, sorry.

jon
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jani Nikula]

On Tue, 11 Oct 2016, Markus Heiser  wrote:
> Anyway, these are only my 2cent. I'am interested in what Jon says
> in general about using (Perl) scripts to generate reST content.

I think I've said all that I have to say on the subject. Up to Jon.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Markus Heiser]

Am 11.10.2016 um 18:45 schrieb Mauro Carvalho Chehab :

> If we allow such scripts (and we do since day zero, due to
> kernel-doc), then there are 3 options:
> 
> 1) use a single python script to run the scripts needed in the
>   build process (e. g. merging kernel-cmd extension upstream);
> 
> 2) use a dedicated python script for every non-phyton script;
> 
> 3) use only python scripts to extend Sphinx functionality.
> 
> The (2) scenario seems to be the worse case, as it will end by
> having a perl(/shell?) script/python script pair for every 
> non-python script we need to run, we're actually making it twice
> worse.
> 
> For (3) to happen, we'll need to convert both kernel-doc and
> parse-headers.pl to Python. This could be a long term goal,
> but I prefer to not rewrite those scripts for a while, as
> it is a lot easier to maintain them in perl, at least to me, and it
> is less disruptive, as rewriting kernel-doc to Python can introduce
> regressions.
> 

Hi Mauro,

its a bit OT in this thread, but in the linuxdoc project,
the kernel-doc is already converted to python.

  https://return42.github.io/linuxdoc/cmd-line.html#kernel-doc

and you have used it already, when you lint with kernel-lint.
Hence I see more progressions than regression ;-)

> So, the way I see, (1) is the best approach.

agree

--Markus--

> 
>> Anyway, these are only my 2cent. I'am interested in what Jon says
>> in general about using (Perl) scripts to generate reST content.
>> 
>> --Markus--
>> 
> 
> Regards,
> Mauro

--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Tue, 11 Oct 2016 18:06:46 +0200
Markus Heiser  escreveu:

> Am 11.10.2016 um 17:34 schrieb Jani Nikula :
> 
> > On Tue, 11 Oct 2016, Mauro Carvalho Chehab  wrote:  
> >> Em Tue, 11 Oct 2016 09:26:48 +0200
> >> Markus Heiser  escreveu:
> >>   
> >>> Am 07.10.2016 um 07:56 schrieb Jani Nikula :
> >>>   
>  On Thu, 06 Oct 2016, Mauro Carvalho Chehab  
>  wrote:
> > Em Thu, 06 Oct 2016 17:21:36 +0300
> > Jani Nikula  escreveu:
> >> We've seen what happens when we make it easy to add random scripts to
> >> build documentation. We've worked hard to get rid of that. In my books,
> >> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
> >> getting rid of all the hacks required in the build. Things that broke 
> >> in
> >> subtle ways.
> > 
> > I really can't see what scripts it get rids.
>  
>  Really? You don't see why the DocBook build was so fragile and difficult
>  to maintain? That scares me a bit, because then you will not have
>  learned why we should at all costs avoid adding random scripts to
>  produce documentation.
> >>> 
> >>> For me, disassembling the DocBok build was hard and bothersome, I don't
> >>> want this back.
> >>> 
> >>> IMO: old hats are productive with perl and they won't adapt another
> >>> interpreter language (like python) for scripting. 
> >>> 
> >>> This series -- the kernel-cmd -- directive avoid that they build
> >>> fragile and difficult to maintain Makefile constructs, calling their
> >>> perl scripts.
> >>> 
> >>> Am 06.10.2016 um 16:21 schrieb Jani Nikula :
> >>>   
>  This is connected to the above: keeping documentation buildable with
>  sphinx-build directly will force you to avoid the Makefile hacks.
> >>> 
> >>> 
> >>> Thats why I think, that the kernel-cmd directive is a more *straight-
> >>> forward* solution, helps to **avoid** complexity while not everyone
> >>> has to script in python ... 
> >>>   
>  Case in point, parse-headers.pl was added for a specific need of media
>  documentation, and for the life of me I can't figure out by reading the
>  script what good, if any, it would be for gpu documentation. I call
>  *that* unmaintainable.
> >>> 
> >>> 
> >>> If one adds a script like parse-headers.pl to the Documentation/sphinx 
> >>> folder, he/she also has to add a documentation to the 
> >>> kernel-documentation.rst
> >>> 
> >>> If the kernel-cmd directive gets acked, I will add a description to
> >>> kernel-documentation.rst and I request Mauro to document the 
> >>> parse-headers.pl
> >>> also.  
> >> 
> >> I can write documentation for parse-headers.pl, either as a --help/--man
> >> option or at some ReST file (or both). I'll add this to my mental TODO
> >> list.  
> > 
> > Thanks, documentation will help everyone else evaluate whether
> > parse-headers.pl is only useful for some corner case in media docs, or
> > perhaps more generally useful. Currently, it's hard to tell.
> > 
> > Anyway, documentation does not change my view on adding such scripts. As
> > I've said, I think they will make the documentation build more difficult
> > to maintain. They are likely to become special purpose hacks for corner
> > cases across documentation.  
> 
> Hmm, why not generating reST content by (Perl) scripts? From my POV,
> the scripts/kernel-doc is such a script and parse-headers.pl
> is just another. 

And Sphinx is a third one - or, actually, a set of scripts, as each
extension is also a script ;)

> The only difference of both is, that kernel-doc
> has its own integration (directive) while kernel-cmd is a simple
> solution to call scripts (e.g. parse-headers.pl) within sphinx-build.
> 
> Joking: the kernel-doc directive could replaced by
> the kernel-cmd directive ;-)

I'm actually OK with either strategy - although it sounds easier
to maintain if we have a single extension to run an external
script - including kernel-doc - specially if the Sphinx extension
API and/or Python changes in the future in a non-backward compatible way.

> > The rest of what you say is unrelated to the patches at hand.  
> 
> I think Mauro want's to address your (justifiable) fear about
> complexity and hacks. IMO he says (by examples); "there are a lot of
> other hard to maintain hacks required, especially when it comes
> to build PDF".
> 
> IMO, complexity is not reduced by prohibit scripts, it is
> ongoing job of the maintainers to observe. 

Yes, that's exactly my point. IMO, not allowing scripts actually
increase complexity, as we'll need Makefile rules to generate
rst files, if we won't allow doing that via perl/python scripts.

If we allow such scripts (and we do since day zero, due to
kernel-doc), then there are 3 options:

1) use a single python 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Markus Heiser]

Am 11.10.2016 um 17:34 schrieb Jani Nikula :

> On Tue, 11 Oct 2016, Mauro Carvalho Chehab  wrote:
>> Em Tue, 11 Oct 2016 09:26:48 +0200
>> Markus Heiser  escreveu:
>> 
>>> Am 07.10.2016 um 07:56 schrieb Jani Nikula :
>>> 
 On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:  
> Em Thu, 06 Oct 2016 17:21:36 +0300
> Jani Nikula  escreveu:  
>> We've seen what happens when we make it easy to add random scripts to
>> build documentation. We've worked hard to get rid of that. In my books,
>> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
>> getting rid of all the hacks required in the build. Things that broke in
>> subtle ways.  
> 
> I really can't see what scripts it get rids.  
 
 Really? You don't see why the DocBook build was so fragile and difficult
 to maintain? That scares me a bit, because then you will not have
 learned why we should at all costs avoid adding random scripts to
 produce documentation.  
>>> 
>>> For me, disassembling the DocBok build was hard and bothersome, I don't
>>> want this back.
>>> 
>>> IMO: old hats are productive with perl and they won't adapt another
>>> interpreter language (like python) for scripting. 
>>> 
>>> This series -- the kernel-cmd -- directive avoid that they build
>>> fragile and difficult to maintain Makefile constructs, calling their
>>> perl scripts.
>>> 
>>> Am 06.10.2016 um 16:21 schrieb Jani Nikula :
>>> 
 This is connected to the above: keeping documentation buildable with
 sphinx-build directly will force you to avoid the Makefile hacks.  
>>> 
>>> 
>>> Thats why I think, that the kernel-cmd directive is a more *straight-
>>> forward* solution, helps to **avoid** complexity while not everyone
>>> has to script in python ... 
>>> 
 Case in point, parse-headers.pl was added for a specific need of media
 documentation, and for the life of me I can't figure out by reading the
 script what good, if any, it would be for gpu documentation. I call
 *that* unmaintainable.  
>>> 
>>> 
>>> If one adds a script like parse-headers.pl to the Documentation/sphinx 
>>> folder, he/she also has to add a documentation to the 
>>> kernel-documentation.rst
>>> 
>>> If the kernel-cmd directive gets acked, I will add a description to
>>> kernel-documentation.rst and I request Mauro to document the 
>>> parse-headers.pl
>>> also.
>> 
>> I can write documentation for parse-headers.pl, either as a --help/--man
>> option or at some ReST file (or both). I'll add this to my mental TODO
>> list.
> 
> Thanks, documentation will help everyone else evaluate whether
> parse-headers.pl is only useful for some corner case in media docs, or
> perhaps more generally useful. Currently, it's hard to tell.
> 
> Anyway, documentation does not change my view on adding such scripts. As
> I've said, I think they will make the documentation build more difficult
> to maintain. They are likely to become special purpose hacks for corner
> cases across documentation.

Hmm, why not generating reST content by (Perl) scripts? From my POV,
the scripts/kernel-doc is such a script and parse-headers.pl
is just another. The only difference of both is, that kernel-doc
has its own integration (directive) while kernel-cmd is a simple
solution to call scripts (e.g. parse-headers.pl) within sphinx-build.

Joking: the kernel-doc directive could replaced by
the kernel-cmd directive ;-)

> The rest of what you say is unrelated to the patches at hand.

I think Mauro want's to address your (justifiable) fear about
complexity and hacks. IMO he says (by examples); "there are a lot of
other hard to maintain hacks required, especially when it comes
to build PDF".

IMO, complexity is not reduced by prohibit scripts, it is
ongoing job of the maintainers to observe. 

Anyway, these are only my 2cent. I'am interested in what Jon says
in general about using (Perl) scripts to generate reST content.

--Markus--

--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jani Nikula]

On Tue, 11 Oct 2016, Mauro Carvalho Chehab  wrote:
> Em Tue, 11 Oct 2016 09:26:48 +0200
> Markus Heiser  escreveu:
>
>> Am 07.10.2016 um 07:56 schrieb Jani Nikula :
>> 
>> > On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:  
>> >> Em Thu, 06 Oct 2016 17:21:36 +0300
>> >> Jani Nikula  escreveu:  
>> >>> We've seen what happens when we make it easy to add random scripts to
>> >>> build documentation. We've worked hard to get rid of that. In my books,
>> >>> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
>> >>> getting rid of all the hacks required in the build. Things that broke in
>> >>> subtle ways.  
>> >> 
>> >> I really can't see what scripts it get rids.  
>> > 
>> > Really? You don't see why the DocBook build was so fragile and difficult
>> > to maintain? That scares me a bit, because then you will not have
>> > learned why we should at all costs avoid adding random scripts to
>> > produce documentation.  
>> 
>> For me, disassembling the DocBok build was hard and bothersome, I don't
>> want this back.
>> 
>> IMO: old hats are productive with perl and they won't adapt another
>> interpreter language (like python) for scripting. 
>> 
>> This series -- the kernel-cmd -- directive avoid that they build
>> fragile and difficult to maintain Makefile constructs, calling their
>> perl scripts.
>> 
>> Am 06.10.2016 um 16:21 schrieb Jani Nikula :
>> 
>> > This is connected to the above: keeping documentation buildable with
>> > sphinx-build directly will force you to avoid the Makefile hacks.  
>> 
>> 
>> Thats why I think, that the kernel-cmd directive is a more *straight-
>> forward* solution, helps to **avoid** complexity while not everyone
>> has to script in python ... 
>> 
>> > Case in point, parse-headers.pl was added for a specific need of media
>> > documentation, and for the life of me I can't figure out by reading the
>> > script what good, if any, it would be for gpu documentation. I call
>> > *that* unmaintainable.  
>> 
>> 
>> If one adds a script like parse-headers.pl to the Documentation/sphinx 
>> folder, he/she also has to add a documentation to the 
>> kernel-documentation.rst
>> 
>> If the kernel-cmd directive gets acked, I will add a description to
>> kernel-documentation.rst and I request Mauro to document the parse-headers.pl
>> also.
>
> I can write documentation for parse-headers.pl, either as a --help/--man
> option or at some ReST file (or both). I'll add this to my mental TODO
> list.

Thanks, documentation will help everyone else evaluate whether
parse-headers.pl is only useful for some corner case in media docs, or
perhaps more generally useful. Currently, it's hard to tell.

Anyway, documentation does not change my view on adding such scripts. As
I've said, I think they will make the documentation build more difficult
to maintain. They are likely to become special purpose hacks for corner
cases across documentation.

The rest of what you say is unrelated to the patches at hand.


BR,
Jani.


>
> -
>
> With regards to Sphinx x DocBook, in terms of functionality, Sphinx
> is an improvement, as we can now use the same markup for everything,
> and cross-reference all documents. Unfortunately, it has less
> functionality than DocBook, and requires more magic to work.
>
> However, there are costs to pay on using a Python script like Sphinx.
> I won't mention again the issues with Python itself and its unstable
> API, but, instead, focus on Sphinx script itself.
>
> First of all, installing packages for Sphinx script to work is a lot
> more complex, specially for the PDF output, as the LaTeX requirements are
> very distribution specific, as some distributions package each LaTeX
> extension as optional packages, and the required extensions used by
> the Sphinx script are not documented.
>
> Also, the Sphinx script and its build logic is a lot more fragile than 
> the Makefile/xmlto that we use on DocBook, and this has nothing to do 
> with adding extra perl/python scripts.
>
> While it is clean for html output, it is very dirty when producing
> a PDF output.
>
> It starts by generating its own Makefile for PDF builds, at the output
> directory, with we have little control on it. 
>
> Also, it seems that almost all books will need hacks to produce proper PDF,
> as neither ReST or Sphinx extensions currently have proper support to adjust
> things provided on DocBook, like setting page properties, enabling 
> image/table scaling, auto-numbering images/tables/examples or changing page
> orientation in the middle of a document.
>
> The biggest issue I found is related to table outputs in PDF format, as
> it does a very poor job adjusting table margins to fit inside the paper
> margins. Also, the auto-generated TOC index on PDF doesn't match the
> numeration of the HTML output. Currently, I didn't find any solution
> for the latter 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Tue, 11 Oct 2016 09:26:48 +0200
Markus Heiser  escreveu:

> Am 07.10.2016 um 07:56 schrieb Jani Nikula :
> 
> > On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:  
> >> Em Thu, 06 Oct 2016 17:21:36 +0300
> >> Jani Nikula  escreveu:  
> >>> We've seen what happens when we make it easy to add random scripts to
> >>> build documentation. We've worked hard to get rid of that. In my books,
> >>> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
> >>> getting rid of all the hacks required in the build. Things that broke in
> >>> subtle ways.  
> >> 
> >> I really can't see what scripts it get rids.  
> > 
> > Really? You don't see why the DocBook build was so fragile and difficult
> > to maintain? That scares me a bit, because then you will not have
> > learned why we should at all costs avoid adding random scripts to
> > produce documentation.  
> 
> For me, disassembling the DocBok build was hard and bothersome, I don't
> want this back.
> 
> IMO: old hats are productive with perl and they won't adapt another
> interpreter language (like python) for scripting. 
> 
> This series -- the kernel-cmd -- directive avoid that they build
> fragile and difficult to maintain Makefile constructs, calling their
> perl scripts.
> 
> Am 06.10.2016 um 16:21 schrieb Jani Nikula :
> 
> > This is connected to the above: keeping documentation buildable with
> > sphinx-build directly will force you to avoid the Makefile hacks.  
> 
> 
> Thats why I think, that the kernel-cmd directive is a more *straight-
> forward* solution, helps to **avoid** complexity while not everyone
> has to script in python ... 
> 
> > Case in point, parse-headers.pl was added for a specific need of media
> > documentation, and for the life of me I can't figure out by reading the
> > script what good, if any, it would be for gpu documentation. I call
> > *that* unmaintainable.  
> 
> 
> If one adds a script like parse-headers.pl to the Documentation/sphinx 
> folder, he/she also has to add a documentation to the kernel-documentation.rst
> 
> If the kernel-cmd directive gets acked, I will add a description to
> kernel-documentation.rst and I request Mauro to document the parse-headers.pl
> also.

I can write documentation for parse-headers.pl, either as a --help/--man
option or at some ReST file (or both). I'll add this to my mental TODO
list.

-

With regards to Sphinx x DocBook, in terms of functionality, Sphinx
is an improvement, as we can now use the same markup for everything,
and cross-reference all documents. Unfortunately, it has less
functionality than DocBook, and requires more magic to work.

However, there are costs to pay on using a Python script like Sphinx.
I won't mention again the issues with Python itself and its unstable
API, but, instead, focus on Sphinx script itself.

First of all, installing packages for Sphinx script to work is a lot
more complex, specially for the PDF output, as the LaTeX requirements are
very distribution specific, as some distributions package each LaTeX
extension as optional packages, and the required extensions used by
the Sphinx script are not documented.

Also, the Sphinx script and its build logic is a lot more fragile than 
the Makefile/xmlto that we use on DocBook, and this has nothing to do 
with adding extra perl/python scripts.

While it is clean for html output, it is very dirty when producing
a PDF output.

It starts by generating its own Makefile for PDF builds, at the output
directory, with we have little control on it. 

Also, it seems that almost all books will need hacks to produce proper PDF,
as neither ReST or Sphinx extensions currently have proper support to adjust
things provided on DocBook, like setting page properties, enabling 
image/table scaling, auto-numbering images/tables/examples or changing page
orientation in the middle of a document.

The biggest issue I found is related to table outputs in PDF format, as
it does a very poor job adjusting table margins to fit inside the paper
margins. Also, the auto-generated TOC index on PDF doesn't match the
numeration of the HTML output. Currently, I didn't find any solution
for the latter issue.

As Sphinx markup doesn't have any way to tell how the output would be
formatted, a lot of magic is required at Sphinx conf.py for PDF
output to start working. Worse than that, several tables require
extra tags for PDF output, specifying the column sizes, and forcing
them to be handled as longtables[1]. Also, if the table is too big,
the rst files require raw latex macros/code before/after such
tables, in order for them to fit inside a paper page - either by
changing their orientation or enabling auto-scale, if the table is
not a longtable. Currently, no way to tell Sphinx to enable auto-scale
on big tables. 

So, in practice, it means that Sphinx build is a house of cards, 
at least for PDF output, as every new 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Markus Heiser]

Am 07.10.2016 um 07:56 schrieb Jani Nikula :

> On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:
>> Em Thu, 06 Oct 2016 17:21:36 +0300
>> Jani Nikula  escreveu:
>>> We've seen what happens when we make it easy to add random scripts to
>>> build documentation. We've worked hard to get rid of that. In my books,
>>> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
>>> getting rid of all the hacks required in the build. Things that broke in
>>> subtle ways.
>> 
>> I really can't see what scripts it get rids.
> 
> Really? You don't see why the DocBook build was so fragile and difficult
> to maintain? That scares me a bit, because then you will not have
> learned why we should at all costs avoid adding random scripts to
> produce documentation.

For me, disassembling the DocBok build was hard and bothersome, I don't
want this back.

IMO: old hats are productive with perl and they won't adapt another
interpreter language (like python) for scripting. 

This series -- the kernel-cmd -- directive avoid that they build
fragile and difficult to maintain Makefile constructs, calling their
perl scripts.

Am 06.10.2016 um 16:21 schrieb Jani Nikula :

> This is connected to the above: keeping documentation buildable with
> sphinx-build directly will force you to avoid the Makefile hacks.


Thats why I think, that the kernel-cmd directive is a more *straight-
forward* solution, helps to **avoid** complexity while not everyone
has to script in python ... 

> Case in point, parse-headers.pl was added for a specific need of media
> documentation, and for the life of me I can't figure out by reading the
> script what good, if any, it would be for gpu documentation. I call
> *that* unmaintainable.


If one adds a script like parse-headers.pl to the Documentation/sphinx 
folder, he/she also has to add a documentation to the kernel-documentation.rst

If the kernel-cmd directive gets acked, I will add a description to
kernel-documentation.rst and I request Mauro to document the parse-headers.pl
also.

But, let's hear what Jon says.

-- Markus --



--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jani Nikula]

On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:
> Em Thu, 06 Oct 2016 17:21:36 +0300
> Jani Nikula  escreveu:
>> We've seen what happens when we make it easy to add random scripts to
>> build documentation. We've worked hard to get rid of that. In my books,
>> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
>> getting rid of all the hacks required in the build. Things that broke in
>> subtle ways.
>
> I really can't see what scripts it get rids.

Really? You don't see why the DocBook build was so fragile and difficult
to maintain? That scares me a bit, because then you will not have
learned why we should at all costs avoid adding random scripts to
produce documentation.

The DocBook build was designed by Rube Goldberg, and this video
accurately illustrates how it works:
https://www.youtube.com/watch?v=qybUFnY7Y8w

I don't want the Sphinx build to end up like that.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Thu, 06 Oct 2016 17:21:36 +0300
Jani Nikula  escreveu:

> On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:
> > Em Thu, 06 Oct 2016 11:42:14 +0300
> > Jani Nikula  escreveu:
> > Just curious here: what use case do you see by building the Kernel
> > documentation without the Kernel tree?  
> 
> Not without the kernel tree, but without the kernel build system. If
> sphinx-build works directly, https://readthedocs.org/ just works when
> you point it at a kernel git repo and the conf.py inside it.
> 
> It would be important to get Sphinx working over at
> https://www.kernel.org/doc/htmldocs/ (which still looks kind of sad) but
> in the mean time we could have had that at https://readthedocs.org/. If
> it weren't for parse-headers.pl and the build hacks around it.

I'm not sure how readthedocs.org work, but if it doesn't use the
extensions inside the git tree, it won't work anyway, and if it uses,
it will run the right scripts, including kernel-doc and parse-headers.pl,
if our tree would contain kernel-cmd or a parse-headers extension.

In the specific case of kernel.org, fixing it to produce all docs
is likely just a matter of installing Sphinx there and be sure that
the documentation will be generated at the right place, like at an
output dir inside https://www.kernel.org/doc/Documentation/. 

I guess we could talk to kernel.org maintainers asking them to do that
for us.

It probably makes sense to create a global index.html file that will
point to both html DocBook and Sphinx output dirs, while we there are
still some DocBook files somewhere - or to finish their conversion for
Kernel 4.10.

> At least there's one at https://01.org/linuxgraphics/gfx-docs/drm/ now.

Good! We can also host a complete Sphinx output at linuxtv.org if 
worth. I'll likely do it anyway, in order to point to the user and
development-process books once we merge those patches, in order to do
some cleanup at the linux media wiki pages there.

> >> However, I would have much preferred the approach I proposed months ago,
> >> having the extension itself do specifically what parse-headers.pl does
> >> now. While it may seem generic on the surface, I don't think it's a
> >> clean or a secure approach to allow running of arbitrary scripts from
> >> PATH while building documentation. It's certainly not an approach that
> >> should be encouraged.  
> >
> > Sorry, but I disagree. The security threat of having a random command
> > doing something wrong is the same as we already have with the Kernel
> > Makefiles, as they can also run a random command. All it is needed
> > is to add this to a Makefile:  
> 
> My intention was to emphasize the importance of the clarity of the
> documentation build, and not get hung up on the security aspect.
> 
> This is connected to the above: keeping documentation buildable with
> sphinx-build directly will force you to avoid the Makefile hacks.

A generic kernel-cmd extension will avoid Makefile hacks. It will
also prevent the addition of a new extension for every new script
we may need to run, in order to cover the Sphinx deficiencies.

> > IMO, a generic solution is a way better, as it sounds easier to
> > maintain.  
> 
> We've seen what happens when we make it easy to add random scripts to
> build documentation. We've worked hard to get rid of that. In my books,
> one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
> getting rid of all the hacks required in the build. Things that broke in
> subtle ways.

I really can't see what scripts it get rids. with both Sphinx and
AsciiDoc(tor), the kernel-doc perl/python script is still needed.

On media books, as Sphinx is incapable of generate cross-references
from C code to the documentation, we needed to add the parse_headers.pl.

So, at the end of the day, we still need the same scripts as before.

If the choice was for Doxygen, then we would get rid of both scripts,
at the price of needing to write tables in HTML, and the need to convert
all kernel-doc markups to Doxygen syntax, with would be worse, IMHO.

> I think having people write Sphinx extensions for the special needs have
> a better chance of solving the problems in more generic ways than
> writing scripts for each specific need. Ideally, we can push those
> extensions to upstream Sphinx, but at least make them easily usable
> across the kernel documentation.

Well, writing a Sphinx extension would require a deep knowledge about
Sphinx internals and a python programmer. As I don't have any, nor
intend to invest some time to be expert on writing Sphinx extensions
any time soon, it means that for me a Sphinx extension is unmaintainable.

> Case in point, parse-headers.pl was added for a specific need of media
> documentation, and for the life of me I can't figure out by reading the
> script what good, if any, it would be for gpu documentation. I call
> *that* unmaintainable.

Actually, parse-headers.pl was added to do 

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jani Nikula]

On Thu, 06 Oct 2016, Mauro Carvalho Chehab  wrote:
> Em Thu, 06 Oct 2016 11:42:14 +0300
> Jani Nikula  escreveu:
> Just curious here: what use case do you see by building the Kernel
> documentation without the Kernel tree?

Not without the kernel tree, but without the kernel build system. If
sphinx-build works directly, https://readthedocs.org/ just works when
you point it at a kernel git repo and the conf.py inside it.

It would be important to get Sphinx working over at
https://www.kernel.org/doc/htmldocs/ (which still looks kind of sad) but
in the mean time we could have had that at https://readthedocs.org/. If
it weren't for parse-headers.pl and the build hacks around it.

At least there's one at https://01.org/linuxgraphics/gfx-docs/drm/ now.

>> However, I would have much preferred the approach I proposed months ago,
>> having the extension itself do specifically what parse-headers.pl does
>> now. While it may seem generic on the surface, I don't think it's a
>> clean or a secure approach to allow running of arbitrary scripts from
>> PATH while building documentation. It's certainly not an approach that
>> should be encouraged.
>
> Sorry, but I disagree. The security threat of having a random command
> doing something wrong is the same as we already have with the Kernel
> Makefiles, as they can also run a random command. All it is needed
> is to add this to a Makefile:

My intention was to emphasize the importance of the clarity of the
documentation build, and not get hung up on the security aspect.

This is connected to the above: keeping documentation buildable with
sphinx-build directly will force you to avoid the Makefile hacks.

> IMO, a generic solution is a way better, as it sounds easier to
> maintain.

We've seen what happens when we make it easy to add random scripts to
build documentation. We've worked hard to get rid of that. In my books,
one of the bigger points in favor of Sphinx over AsciiDoc(tor) was
getting rid of all the hacks required in the build. Things that broke in
subtle ways.

I think having people write Sphinx extensions for the special needs have
a better chance of solving the problems in more generic ways than
writing scripts for each specific need. Ideally, we can push those
extensions to upstream Sphinx, but at least make them easily usable
across the kernel documentation.

Case in point, parse-headers.pl was added for a specific need of media
documentation, and for the life of me I can't figure out by reading the
script what good, if any, it would be for gpu documentation. I call
*that* unmaintainable.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Mauro Carvalho Chehab]

Em Thu, 06 Oct 2016 11:42:14 +0300
Jani Nikula  escreveu:

> On Thu, 06 Oct 2016, Markus Heiser  wrote:
> > with this series a reST-directive kernel-cmd is introduced. The kernel-cmd
> > directive includes contend from the stdout of a command-line (@mchehab asked
> > for).  

Ok, I ran some tests here and it works as expected.

> I like the fact that this removes Documentation/media/Makefile, and
> cleans up the Sphinx build rule in Documentation/Makefile.sphinx.

Yeah, that sounds great.

> Does
> this also make the documentation buildable with sphinx-build directly,
> without the kernel build system? If so, great.

I guess it probably allows that, if you include the extension on
some other tree.

Just curious here: what use case do you see by building the Kernel
documentation without the Kernel tree?

> However, I would have much preferred the approach I proposed months ago,
> having the extension itself do specifically what parse-headers.pl does
> now. While it may seem generic on the surface, I don't think it's a
> clean or a secure approach to allow running of arbitrary scripts from
> PATH while building documentation. It's certainly not an approach that
> should be encouraged.

Sorry, but I disagree. The security threat of having a random command
doing something wrong is the same as we already have with the Kernel
Makefiles, as they can also run a random command. All it is needed
is to add this to a Makefile:

subdir-y += run_some_evil_cmd

If we accept the fact that we do need to run commands when running "make",
it doesn't really matter if such command is at a makefile, inside a 
perl/python script or called via some Sphinx directive. In all cases,
patches need to be reviewed by the community, to be sure that they won't
introduce any vulnerabilities.

Btw, with regards to security, a way bigger threat is if someone
introduces a vulnerable code inside the Kernel code, as this will
affect a lot more systems than a vulnerability at the documentation
build process.

Yet, if you think security is still a high risk, my suggestion
would be to restrict the kernel-cmd script to only run scripts
inside trusted places, like Documentation/sphinx.


-

The real issue here is that Sphinx itself doesn't provide what
it is needed to build the Kernel documentation. Some extra
scripts are required. Right now, we converted maybe 5% of the
documentation to ReST, and we're using running two perl scripts:
- kernel-doc
- parse-headers.pl

We also identified that, if we want to add the MAINTAINERS file to
some documentation (or a parsed version of it), we would need an extra
script to filter it[1].

I can think on other use cases to run such scripts[2].

What I'm saying is that, we can keep adding a Sphinx-specific
extension for every such needs, with will result in code duplication
and will make harder to maintain it, or we can use a generic
solution like this kernel-cmd extension.

IMO, a generic solution is a way better, as it sounds easier to
maintain.

Regards,
Mauro

[1] 
https://git.linuxtv.org/mchehab/experimental.git/commit/?h=lkml-books=c8b07684c0278d7f9d0e30f575eb4be3a2da4c3b

[2] There's another possible usecase, with I'm not convinced yet
whether should be addressed or not.

At the media subsystem, we use 8 out-of-tree scripts that update
the list of supported USB and PCI media boards, e. g. the files under:
Documentation/media/v4l-drivers/*cardlist.rst

Such scripts used to be part of the mercurial tree, before we moved the 
media development to git, like this one:
https://linuxtv.org/hg/v4l-dvb/file/3724e93f7af5/v4l/scripts/bttv.pl

Currently, I'm using a local fork of the old scripts, and, when I
notice a patch for a driver with a cardlist, if I remember, I run the
scripts to add to update the cardlists. While it could be kept
OOT forever, if moved it to the Kernel tree,  the documentation will 
always reflect the Kernel status, with is, IMHO, a good thing.
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/4] reST-directive kernel-cmd / include contentent from scripts

[Jani Nikula]

On Thu, 06 Oct 2016, Markus Heiser  wrote:
> with this series a reST-directive kernel-cmd is introduced. The kernel-cmd
> directive includes contend from the stdout of a command-line (@mchehab asked
> for).

I like the fact that this removes Documentation/media/Makefile, and
cleans up the Sphinx build rule in Documentation/Makefile.sphinx. Does
this also make the documentation buildable with sphinx-build directly,
without the kernel build system? If so, great.

However, I would have much preferred the approach I proposed months ago,
having the extension itself do specifically what parse-headers.pl does
now. While it may seem generic on the surface, I don't think it's a
clean or a secure approach to allow running of arbitrary scripts from
PATH while building documentation. It's certainly not an approach that
should be encouraged.

In part, the reason the DocBook build became so unwieldy was the
proliferation of arbitrary scripts and tools required to make it
happen. I think it would be really sad to let this happen to the Sphinx
build. I am *already* sad about how parse-headers.pl was bolted on to
the build.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html