Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Keith Packard  writes:

> The goal would be to create an html document which could be used without
> javascript, and that would work without css as well.

I've managed to hack up asciidoc to generate the TOC within the
document, rather than requiring javascript. The changes are fairly
minor, and seem to add a nice generalization to the asciidoc environment
which should be useful in other contexts.

The changes consist of two bits -- the first is to allow the diversion
of some text from .conf file sections, the second is to postpone some
attribute processing to a second pass over the document so that the TOC
can be inserted in the desired location, instead of requiring that it be
placed at the bottom.

I've sent these changes upstream, and also pushed them to a personal
asciidoc git repository at :

git clone git://keithp.com/git/asciidoc

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Keith Packard  writes:

> The goal would be to create an html document which could be used without
> javascript, and that would work without css as well.

I've managed to hack up asciidoc to generate the TOC within the
document, rather than requiring javascript. The changes are fairly
minor, and seem to add a nice generalization to the asciidoc environment
which should be useful in other contexts.

The changes consist of two bits -- the first is to allow the diversion
of some text from .conf file sections, the second is to postpone some
attribute processing to a second pass over the document so that the TOC
can be inserted in the desired location, instead of requiring that it be
placed at the bottom.

I've sent these changes upstream, and also pushed them to a personal
asciidoc git repository at :

git clone git://keithp.com/git/asciidoc

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jani Nikula  writes:

> One of the chief complaints with the current pipeline (and some of the
> proposals) has been the need to install lots of tools with lots of
> dependencies. I would like to avoid the need to install bleeding edge
> tools and stick to what's already widely available in distros. Thus I
> would like to avoid hacking asciidoc for our needs.

Agreed. That means using docbook for now; the native html output from
asciidoc is simply not usable for anything more complicated than a short
web page. However, getting ready to collapse the pipeline by eliminating
docbook seems like a good medium-term goal.

> Also, I'd really like to not have to decide between asciidoc and
> asciidoctor, and only use features supported by both. Let the users pick
> which one suits them better.

That's harder; you'll have much different output from the two
processors. I'd encourage the selection of one of these two tools
instead of trying to support both. I've settled on using only asciidoc
for my other projects because it doesn't require the installation of a
whole new language environment.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jonathan Corbet  writes:

> asciidoc->HTML on its own isn't viable, I think; we do have people wanting
> other formats.  Though one might well ask when somebody last successfully
> generated PDF...maybe it's not worth the trouble.  I would like epub
> someday...

I'm hopeful that I can hack up asciidoc to generate usable HTML
directly. Once we've got HTML, we've got epub.

If we don't want to use docbook for pdf, asciidoc has a native latex
backend. That's in about the same shape as the html backend. Would that
be better than docbook?

> There's also people who actually use the man-page output.  I don't think
> that should require the xml step; getting rid of that might make it
> possible to do "make mandocs" and have it finish before the next merge
> window opens...

Adding a troff backend to asciidoc would be simple enough; I'm not sure
what other method you'd suggest here.

> We talked about that a bit in Geelong; the short-term idea was to generate
> a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
> you thought that would be a fine solution that would remove the need for
> chunked output.  Keith seemed interested in looking into this too.

Here's an example that takes the docbook output with some simple CSS
hacks to place the TOC alongside the document in a separate scrolling
list. With a small bit of javascript, I'm pretty sure that could have
collapsible entries.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Thu, 11 Feb 2016, Keith Packard  wrote:
> I think it should be pretty easy to hack asciidoc to add diversions to
> hold TOC contents while generating the rest of the doc and then replay
> the diversion into the final document.

One of the chief complaints with the current pipeline (and some of the
proposals) has been the need to install lots of tools with lots of
dependencies. I would like to avoid the need to install bleeding edge
tools and stick to what's already widely available in distros. Thus I
would like to avoid hacking asciidoc for our needs.

Also, I'd really like to not have to decide between asciidoc and
asciidoctor, and only use features supported by both. Let the users pick
which one suits them better.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jonathan Corbet  writes:

> asciidoc->HTML on its own isn't viable, I think; we do have people wanting
> other formats.  Though one might well ask when somebody last successfully
> generated PDF...maybe it's not worth the trouble.  I would like epub
> someday...

I'm hopeful that I can hack up asciidoc to generate usable HTML
directly. Once we've got HTML, we've got epub.

If we don't want to use docbook for pdf, asciidoc has a native latex
backend. That's in about the same shape as the html backend. Would that
be better than docbook?

> There's also people who actually use the man-page output.  I don't think
> that should require the xml step; getting rid of that might make it
> possible to do "make mandocs" and have it finish before the next merge
> window opens...

Adding a troff backend to asciidoc would be simple enough; I'm not sure
what other method you'd suggest here.

> We talked about that a bit in Geelong; the short-term idea was to generate
> a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
> you thought that would be a fine solution that would remove the need for
> chunked output.  Keith seemed interested in looking into this too.

Here's an example that takes the docbook output with some simple CSS
hacks to place the TOC alongside the document in a separate scrolling
list. With a small bit of javascript, I'm pretty sure that could have
collapsible entries.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jani Nikula  writes:

> One of the chief complaints with the current pipeline (and some of the
> proposals) has been the need to install lots of tools with lots of
> dependencies. I would like to avoid the need to install bleeding edge
> tools and stick to what's already widely available in distros. Thus I
> would like to avoid hacking asciidoc for our needs.

Agreed. That means using docbook for now; the native html output from
asciidoc is simply not usable for anything more complicated than a short
web page. However, getting ready to collapse the pipeline by eliminating
docbook seems like a good medium-term goal.

> Also, I'd really like to not have to decide between asciidoc and
> asciidoctor, and only use features supported by both. Let the users pick
> which one suits them better.

That's harder; you'll have much different output from the two
processors. I'd encourage the selection of one of these two tools
instead of trying to support both. I've settled on using only asciidoc
for my other projects because it doesn't require the installation of a
whole new language environment.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Thu, 11 Feb 2016, Keith Packard  wrote:
> I think it should be pretty easy to hack asciidoc to add diversions to
> hold TOC contents while generating the rest of the doc and then replay
> the diversion into the final document.

One of the chief complaints with the current pipeline (and some of the
proposals) has been the need to install lots of tools with lots of
dependencies. I would like to avoid the need to install bleeding edge
tools and stick to what's already widely available in distros. Thus I
would like to avoid hacking asciidoc for our needs.

Also, I'd really like to not have to decide between asciidoc and
asciidoctor, and only use features supported by both. Let the users pick
which one suits them better.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jonathan Corbet  writes:

> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]

Thanks.

>  - I would like to format directly to HTML if at all possible.

Agreed. asciidoc's docbook path seems to only increase the amount of
software involved.

>It seems it should be possible to get a table of contents into the
>files, and the feedback I got was that a TOC would be enough for
>navigation

I spent a few hours on the flight home reading asciidoc source code, and
it is a stream processor with a stack. The output is generated with some
simple templates, one for each backend. Here's the xhtml1.1 template for
section level sections: (sect1 in the .conf file):

[sect1]

{numbered?{sectnum} }{title}

|



The contents of the section get inserted at the |; it's nesting, so
[sect2] bits would get expanded while being processed.

Each asciidoc backend has dramatically different functionality. It's
pretty clear to me that the 'docbook' backend has the best support for
larger documents as that provides 'book-scale' processing bits. I've
recently written a book in asciidoc using the docbook backend, and the
html and pdf results are quite comparable. Using the html backend from
asciidoc yields a significantly different result.

I think it should be pretty easy to hack asciidoc to add diversions to
hold TOC contents while generating the rest of the doc and then replay
the diversion into the final document. Something like:

[sect1]

{numbered?{sectnum} }{title}
|>"table-of-contents"


{numbered?{sectnum} {title}




|



At the end of the document, we'd have some way of wrapping the diversion
in suitable additional bits to  complete the TOC, which would then be
formatted by CSS.

This same technique could be used to create lists of figures and tables.

The goal would be to create an html document which could be used without
javascript, and that would work without css as well.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 16:03:38 +0200
Jani Nikula  wrote:

> I'd like to clarify the end goal a bit more before deciding what to do
> next. In particular, is the aim to have asciidoc->HTML only or dual
> asciidoc->HTML and asciidoc->XML->whatever? Or independent
> asciidoc->HTML first, with the existing DocBook on the side until
> everything's converted? Something else?

asciidoc->HTML on its own isn't viable, I think; we do have people wanting
other formats.  Though one might well ask when somebody last successfully
generated PDF...maybe it's not worth the trouble.  I would like epub
someday...

There's also people who actually use the man-page output.  I don't think
that should require the xml step; getting rid of that might make it
possible to do "make mandocs" and have it finish before the next merge
window opens...

> Direct asciidoc->HTML has the problem I mentioned that there is no
> chunked output. If the source is big (as-is or via asciidoc includes)
> the output is big. The current gpu.tmpl turned way too big. We could
> alleviate that by splitting the source documents into smaller pieces (in
> gpu.tmpl case it's desirable no matter what), and tying them together
> via cross-references and TOC rather than asciidoc includes.

We talked about that a bit in Geelong; the short-term idea was to generate
a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
you thought that would be a fine solution that would remove the need for
chunked output.  Keith seemed interested in looking into this too.

I would still like to look into splitting up the output.  One would *think*
we ought to be able to do that without the whole docbook infrastructure,
but, then, I'm known to be a naive optimist...

> The problem with this, in turn, is that I don't really know how
> automatic cross-referencing between kernel-doc comments would turn out
> then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
> after gpu.tmpl split) as asciidoc would process the files
> independently. A kernel-doc comment writer shouldn't have to know which
> document the referenced symbol is in... We could do post-processing I
> guess, but I'd really like to get rid of the homebrew aspects here.
> 
> Is it acceptable to have dead links when referencing symbols outside of
> the document in question, for the time being, until someone figures out
> a nice way to do this?

Short-term *maybe*, but I think we'd want to figure that one out quickly. 

> Also in my dream world you could have asciidoc files anywhere in the
> Documentation tree, with a Makefile per directory identifying which ones
> should be processed as asciidoc. I might even name them all .txt, and
> you wouldn't have to rename existing "almost markup" plain text files to
> have them processed, just fix the markup and update the Makefile. (FWIW
> asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
> .asciidoc.) I think this would better promote a gradual transition to
> lightweight markup, with easier to review patches. Also you mentioned
> there's no structure under Documentation. Allowing asciidoc files
> anywhere would, I think, help gradual restructuring.

I agree with all of this, but I still think that, for the short term while
we're figuring out how all this works, it's better to concentrate it in one
place where people can actually find it...

> I'd turn this around. IMO the problem isn't insisting EXPORT_SYMBOL is
> in the same file as the definition of the symbol. The problem is
> insisting that the kernel-doc comment is in the same file as the
> EXPORT_SYMBOL and the definition. Particularly include/media has plenty
> of kernel-doc in headers with the declarations.
> 
> If we can't insist on that, we could teach kernel-doc to scan a list of
> other files for the EXPORT_SYMBOLs, instead of having that logic
> externally in docproc. This should be trivial, especially if you know
> perl. (Unfortunately this might get a little tricky with the include
> syntax.)
> 
> This was mostly driven by the desire to get rid of the docproc
> preprocessing step.

...and that's a worthy goal.  In an ideal world, it's all found together.
I think we should probably proceed with the idea that the EXPORT_SYMBOL
issue can be dealt with.

> >> Please let me know your thoughts on the above.  Do you think you can find
> >> some time over the next month for this?  I'll try to shake loose some time
> >> too, but, well, $EXCUSES...  
> 
> If we can come up with a plan where I can be reasonably sure the
> polished effort isn't going down the drain... ;)

Seems we should be able to do that.  We want this stuff, even I'm not so
dumb as to send it down the drain when things are so close...:)

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 09:07:22 +0100
Daniel Vetter  wrote:

> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

That can be fine, I'd just like to have the end goal in mind.  For the near
future we should go with what actually works, definitely.

> >  - Asciidoc templates and processing should happen in a new directory
> >(perhaps imaginatively called "asciidoc"); having them in a directory
> >called "DocBook" seems a little weird.  More importantly, though, I'd
> >like to separate them out as a fresh start, and not mess with the
> >existing DocBook templates until we decide we don't need them anymore.
> >If we could end up with a cleaner, simpler makefile in the process,
> >that would be a bonus.  
> 
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

That's a good question.  I can certainly see the value of mixing the
templates with the rest, but that's a longer-term thing.  I'd sure like to
clean up the main Documentation/ directory before we start scattering
asciidoc stuff around there.  For the moment, I think my preference is
still to focus this work in one place where it's easily found and played
with.  Straightening out this directory is going to involve a fair amount
of moving stuff around, adding a few template files to that debt isn't
going to change the situation much.

> >  - I'm not sold on the new inclusion mechanism.  Creating thousands of
> >little files and tracking them for dependencies and such doesn't seem
> >like a simplification or a path toward better performance.  I would
> >like to at least consider keeping the direct-from-source inclusion.  
> 
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With those you can pass options to e.g. shift the hierarchy.
> With that you can do subheadings in DOC: sections and then seamlessly
> include them. Or similar stuff.
> 
> The speed-up due to parallelization is just a small bonus.
> 
> Also generating thousands of files is totally not unheard of in the kernel:
> 
> $ find include/config | wc -l
> 2623
> 
> None of those are in git.

Well, we are talking about an order of magnitude more files...  Still, I
said "not sold on" rather than "violently opposed to".  It does seem that
there are some good reasons for doing things this way, including, as Jani
said, getting rid of docproc and not mixing in a weird alien include
syntax.  I'm still not 100% sold, but I'll not hold up a working patch on
this point.

> >  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
> >it's going to work for now; that could maybe change after Al's work
> >goes in, which could be fairly soon.  
> 
> Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
> holds true for gpu docs already I think, and most other subsystems.
> The trouble iirc is all around asm and similar stuff, and we can't
> kerneldoc asm afaik.

asm stuff and things built into libraries.  But it does seem that this is
well on the way toward being fixed.

> One more thing we discussed: Did you ping kbuild folks already? Or
> want to get some agreement on the overall build process first?

Not yet.  It's worth doing...it would be nice, someday, if docs makefiles
could just have lines like:

  adoc-y += drm.txt ...

but I think we should work out how the pieces fit together before we get
too worried about the details of the build system.  What we have now isn't
particularly well integrated, we're not likely to make it all that much
worse...

Thanks,

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Wed, 10 Feb 2016, Jani Nikula  wrote:
>> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
>>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>>little files and tracking them for dependencies and such doesn't seem
>>>like a simplification or a path toward better performance.  I would
>>>like to at least consider keeping the direct-from-source inclusion.

...

> Yes, my main motivation here was to get rid of the preprocessing step
> (currently tmpl->xml). I wanted to have the source documents in pure
> markup which could be directly processed by asciidoc. I wanted to have
> the editor markup helpers and syntax highlighting just work, with no
> extra non-markup cruft to confuse it. (For example, emacs tells me the
> current tmpl files are invalid XML because of the docproc directives.)
> This ties back to the dream above; just have .txt files with no
> preprocessing step, IMO it's less confusing for actually writing the
> docs.

I suppose a compromise could be to put the docproc directives in
asciidoc comments to keep the files pure asciidoc and to hide the
preprocessing step from the document writers, i.e. call them asciidoc
and name them .txt instead of .tmpl or something. While I'm not thrilled
about the idea of keeping docproc around, this would be progress, would
avoid the EXPORT_SYMBOL problem for now, and, most importantly, wouldn't
block us from doing what I suggested as a future iteration.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

[Sorry this turned out a long email, I didn't have the time to write a
short one.]

On Wed, 10 Feb 2016, Daniel Vetter  wrote:
> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
>> On Tue, 26 Jan 2016 14:08:45 +0200
>> Jani Nikula  wrote:
>>
>>> I'm afraid we've done some overlapping work in the mean time, but I'm
>>> happy we've both looked at the tool chain, and can have a more
>>> meaningful conversation now.
>>
>> [Adding Keith since you said you wanted to be a part of this - let us know
>> when you've had enough!]
>>
>> So I've spent a bit of time looking at this, and quite a bit more time
>> talking with various folks at LCA.  There is pretty much universal
>> agreement that this is interesting work and the direction we'd like to
>> go.  My current hope is that we can merge some version of it for 4.6 and
>> see where it goes from there.
>>
>> So naturally I have some thoughts on the whole thing...
>>
>>  - I would like to format directly to HTML if at all possible.  It seems
>>it should be possible to get a table of contents into the files, and
>>the feedback I got was that a TOC would be enough for navigation - it
>>would not be necessary to split the files at that point.  We might
>>still want to try to figure that out too, though.  In any case, this
>>isn't a show stopper, in that we can change it anytime if a better way
>>shows up.  But I'd like to have it in mind.
>
> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

I'd like to clarify the end goal a bit more before deciding what to do
next. In particular, is the aim to have asciidoc->HTML only or dual
asciidoc->HTML and asciidoc->XML->whatever? Or independent
asciidoc->HTML first, with the existing DocBook on the side until
everything's converted? Something else?

Direct asciidoc->HTML has the problem I mentioned that there is no
chunked output. If the source is big (as-is or via asciidoc includes)
the output is big. The current gpu.tmpl turned way too big. We could
alleviate that by splitting the source documents into smaller pieces (in
gpu.tmpl case it's desirable no matter what), and tying them together
via cross-references and TOC rather than asciidoc includes.

The problem with this, in turn, is that I don't really know how
automatic cross-referencing between kernel-doc comments would turn out
then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
after gpu.tmpl split) as asciidoc would process the files
independently. A kernel-doc comment writer shouldn't have to know which
document the referenced symbol is in... We could do post-processing I
guess, but I'd really like to get rid of the homebrew aspects here.

Is it acceptable to have dead links when referencing symbols outside of
the document in question, for the time being, until someone figures out
a nice way to do this?

> Once that's solid we can look into the icing on the cake for later
> kernels I think.
>
>>  - Asciidoc templates and processing should happen in a new directory
>>(perhaps imaginatively called "asciidoc"); having them in a directory
>>called "DocBook" seems a little weird.  More importantly, though, I'd
>>like to separate them out as a fresh start, and not mess with the
>>existing DocBook templates until we decide we don't need them anymore.
>>If we could end up with a cleaner, simpler makefile in the process,
>>that would be a bonus.
>
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

Also in my dream world you could have asciidoc files anywhere in the
Documentation tree, with a Makefile per directory identifying which ones
should be processed as asciidoc. I might even name them all .txt, and
you wouldn't have to rename existing "almost markup" plain text files to
have them processed, just fix the markup and update the Makefile. (FWIW
asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
.asciidoc.) I think this would better promote a gradual transition to
lightweight markup, with easier to review patches. Also you mentioned
there's no structure under Documentation. Allowing asciidoc files
anywhere would, I think, help gradual restructuring.

The output could be a subdirectory (one per output format?) under
Documentation.

>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>little files and tracking them for dependencies and such doesn't seem
>>like a simplification or a path toward better performance.  I would
>>like to at least consider keeping the direct-from-source inclusion.
>
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
> On Tue, 26 Jan 2016 14:08:45 +0200
> Jani Nikula  wrote:
>
>> I'm afraid we've done some overlapping work in the mean time, but I'm
>> happy we've both looked at the tool chain, and can have a more
>> meaningful conversation now.
>
> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]
>
> So I've spent a bit of time looking at this, and quite a bit more time
> talking with various folks at LCA.  There is pretty much universal
> agreement that this is interesting work and the direction we'd like to
> go.  My current hope is that we can merge some version of it for 4.6 and
> see where it goes from there.
>
> So naturally I have some thoughts on the whole thing...
>
>  - I would like to format directly to HTML if at all possible.  It seems
>it should be possible to get a table of contents into the files, and
>the feedback I got was that a TOC would be enough for navigation - it
>would not be necessary to split the files at that point.  We might
>still want to try to figure that out too, though.  In any case, this
>isn't a show stopper, in that we can change it anytime if a better way
>shows up.  But I'd like to have it in mind.

I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
toolchain, since that's less disruptive. And with that we can also
fully concentrating on the frontend, and how it'll look and behave.

Once that's solid we can look into the icing on the cake for later
kernels I think.

>  - Asciidoc templates and processing should happen in a new directory
>(perhaps imaginatively called "asciidoc"); having them in a directory
>called "DocBook" seems a little weird.  More importantly, though, I'd
>like to separate them out as a fresh start, and not mess with the
>existing DocBook templates until we decide we don't need them anymore.
>If we could end up with a cleaner, simpler makefile in the process,
>that would be a bonus.

For the long term dream plan of including other .txt files from the
existing pile of unstructured docs, do we really want a separate
asciidoc directory? Or just .asciidoc as a special extension?

>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>little files and tracking them for dependencies and such doesn't seem
>like a simplification or a path toward better performance.  I would
>like to at least consider keeping the direct-from-source inclusion.

The motivation behind the new inclusion mechanism isn't the speed-up
due to parallelization, but being able to use native asciidoc
includes. With those you can pass options to e.g. shift the hierarchy.
With that you can do subheadings in DOC: sections and then seamlessly
include them. Or similar stuff.

The speed-up due to parallelization is just a small bonus.

Also generating thousands of files is totally not unheard of in the kernel:

$ find include/config | wc -l
2623

None of those are in git.

>  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
>it's going to work for now; that could maybe change after Al's work
>goes in, which could be fairly soon.

Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
holds true for gpu docs already I think, and most other subsystems.
The trouble iirc is all around asm and similar stuff, and we can't
kerneldoc asm afaik.

> Please let me know your thoughts on the above.  Do you think you can find
> some time over the next month for this?  I'll try to shake loose some time
> too, but, well, $EXCUSES...

One more thing we discussed: Did you ping kbuild folks already? Or
want to get some agreement on the overall build process first?

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

[Sorry this turned out a long email, I didn't have the time to write a
short one.]

On Wed, 10 Feb 2016, Daniel Vetter  wrote:
> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
>> On Tue, 26 Jan 2016 14:08:45 +0200
>> Jani Nikula  wrote:
>>
>>> I'm afraid we've done some overlapping work in the mean time, but I'm
>>> happy we've both looked at the tool chain, and can have a more
>>> meaningful conversation now.
>>
>> [Adding Keith since you said you wanted to be a part of this - let us know
>> when you've had enough!]
>>
>> So I've spent a bit of time looking at this, and quite a bit more time
>> talking with various folks at LCA.  There is pretty much universal
>> agreement that this is interesting work and the direction we'd like to
>> go.  My current hope is that we can merge some version of it for 4.6 and
>> see where it goes from there.
>>
>> So naturally I have some thoughts on the whole thing...
>>
>>  - I would like to format directly to HTML if at all possible.  It seems
>>it should be possible to get a table of contents into the files, and
>>the feedback I got was that a TOC would be enough for navigation - it
>>would not be necessary to split the files at that point.  We might
>>still want to try to figure that out too, though.  In any case, this
>>isn't a show stopper, in that we can change it anytime if a better way
>>shows up.  But I'd like to have it in mind.
>
> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

I'd like to clarify the end goal a bit more before deciding what to do
next. In particular, is the aim to have asciidoc->HTML only or dual
asciidoc->HTML and asciidoc->XML->whatever? Or independent
asciidoc->HTML first, with the existing DocBook on the side until
everything's converted? Something else?

Direct asciidoc->HTML has the problem I mentioned that there is no
chunked output. If the source is big (as-is or via asciidoc includes)
the output is big. The current gpu.tmpl turned way too big. We could
alleviate that by splitting the source documents into smaller pieces (in
gpu.tmpl case it's desirable no matter what), and tying them together
via cross-references and TOC rather than asciidoc includes.

The problem with this, in turn, is that I don't really know how
automatic cross-referencing between kernel-doc comments would turn out
then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
after gpu.tmpl split) as asciidoc would process the files
independently. A kernel-doc comment writer shouldn't have to know which
document the referenced symbol is in... We could do post-processing I
guess, but I'd really like to get rid of the homebrew aspects here.

Is it acceptable to have dead links when referencing symbols outside of
the document in question, for the time being, until someone figures out
a nice way to do this?

> Once that's solid we can look into the icing on the cake for later
> kernels I think.
>
>>  - Asciidoc templates and processing should happen in a new directory
>>(perhaps imaginatively called "asciidoc"); having them in a directory
>>called "DocBook" seems a little weird.  More importantly, though, I'd
>>like to separate them out as a fresh start, and not mess with the
>>existing DocBook templates until we decide we don't need them anymore.
>>If we could end up with a cleaner, simpler makefile in the process,
>>that would be a bonus.
>
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

Also in my dream world you could have asciidoc files anywhere in the
Documentation tree, with a Makefile per directory identifying which ones
should be processed as asciidoc. I might even name them all .txt, and
you wouldn't have to rename existing "almost markup" plain text files to
have them processed, just fix the markup and update the Makefile. (FWIW
asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
.asciidoc.) I think this would better promote a gradual transition to
lightweight markup, with easier to review patches. Also you mentioned
there's no structure under Documentation. Allowing asciidoc files
anywhere would, I think, help gradual restructuring.

The output could be a subdirectory (one per output format?) under
Documentation.

>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>little files and tracking them for dependencies and such doesn't seem
>>like a simplification or a path toward better performance.  I would
>>like to at least consider keeping the direct-from-source inclusion.
>
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Wed, 10 Feb 2016, Jani Nikula  wrote:
>> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
>>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>>little files and tracking them for dependencies and such doesn't seem
>>>like a simplification or a path toward better performance.  I would
>>>like to at least consider keeping the direct-from-source inclusion.

...

> Yes, my main motivation here was to get rid of the preprocessing step
> (currently tmpl->xml). I wanted to have the source documents in pure
> markup which could be directly processed by asciidoc. I wanted to have
> the editor markup helpers and syntax highlighting just work, with no
> extra non-markup cruft to confuse it. (For example, emacs tells me the
> current tmpl files are invalid XML because of the docproc directives.)
> This ties back to the dream above; just have .txt files with no
> preprocessing step, IMO it's less confusing for actually writing the
> docs.

I suppose a compromise could be to put the docproc directives in
asciidoc comments to keep the files pure asciidoc and to hide the
preprocessing step from the document writers, i.e. call them asciidoc
and name them .txt instead of .tmpl or something. While I'm not thrilled
about the idea of keeping docproc around, this would be progress, would
avoid the EXPORT_SYMBOL problem for now, and, most importantly, wouldn't
block us from doing what I suggested as a future iteration.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 16:03:38 +0200
Jani Nikula  wrote:

> I'd like to clarify the end goal a bit more before deciding what to do
> next. In particular, is the aim to have asciidoc->HTML only or dual
> asciidoc->HTML and asciidoc->XML->whatever? Or independent
> asciidoc->HTML first, with the existing DocBook on the side until
> everything's converted? Something else?

asciidoc->HTML on its own isn't viable, I think; we do have people wanting
other formats.  Though one might well ask when somebody last successfully
generated PDF...maybe it's not worth the trouble.  I would like epub
someday...

There's also people who actually use the man-page output.  I don't think
that should require the xml step; getting rid of that might make it
possible to do "make mandocs" and have it finish before the next merge
window opens...

> Direct asciidoc->HTML has the problem I mentioned that there is no
> chunked output. If the source is big (as-is or via asciidoc includes)
> the output is big. The current gpu.tmpl turned way too big. We could
> alleviate that by splitting the source documents into smaller pieces (in
> gpu.tmpl case it's desirable no matter what), and tying them together
> via cross-references and TOC rather than asciidoc includes.

We talked about that a bit in Geelong; the short-term idea was to generate
a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
you thought that would be a fine solution that would remove the need for
chunked output.  Keith seemed interested in looking into this too.

I would still like to look into splitting up the output.  One would *think*
we ought to be able to do that without the whole docbook infrastructure,
but, then, I'm known to be a naive optimist...

> The problem with this, in turn, is that I don't really know how
> automatic cross-referencing between kernel-doc comments would turn out
> then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
> after gpu.tmpl split) as asciidoc would process the files
> independently. A kernel-doc comment writer shouldn't have to know which
> document the referenced symbol is in... We could do post-processing I
> guess, but I'd really like to get rid of the homebrew aspects here.
> 
> Is it acceptable to have dead links when referencing symbols outside of
> the document in question, for the time being, until someone figures out
> a nice way to do this?

Short-term *maybe*, but I think we'd want to figure that one out quickly. 

> Also in my dream world you could have asciidoc files anywhere in the
> Documentation tree, with a Makefile per directory identifying which ones
> should be processed as asciidoc. I might even name them all .txt, and
> you wouldn't have to rename existing "almost markup" plain text files to
> have them processed, just fix the markup and update the Makefile. (FWIW
> asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
> .asciidoc.) I think this would better promote a gradual transition to
> lightweight markup, with easier to review patches. Also you mentioned
> there's no structure under Documentation. Allowing asciidoc files
> anywhere would, I think, help gradual restructuring.

I agree with all of this, but I still think that, for the short term while
we're figuring out how all this works, it's better to concentrate it in one
place where people can actually find it...

> I'd turn this around. IMO the problem isn't insisting EXPORT_SYMBOL is
> in the same file as the definition of the symbol. The problem is
> insisting that the kernel-doc comment is in the same file as the
> EXPORT_SYMBOL and the definition. Particularly include/media has plenty
> of kernel-doc in headers with the declarations.
> 
> If we can't insist on that, we could teach kernel-doc to scan a list of
> other files for the EXPORT_SYMBOLs, instead of having that logic
> externally in docproc. This should be trivial, especially if you know
> perl. (Unfortunately this might get a little tricky with the include
> syntax.)
> 
> This was mostly driven by the desire to get rid of the docproc
> preprocessing step.

...and that's a worthy goal.  In an ideal world, it's all found together.
I think we should probably proceed with the idea that the EXPORT_SYMBOL
issue can be dealt with.

> >> Please let me know your thoughts on the above.  Do you think you can find
> >> some time over the next month for this?  I'll try to shake loose some time
> >> too, but, well, $EXCUSES...  
> 
> If we can come up with a plan where I can be reasonably sure the
> polished effort isn't going down the drain... ;)

Seems we should be able to do that.  We want this stuff, even I'm not so
dumb as to send it down the drain when things are so close...:)

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 09:07:22 +0100
Daniel Vetter  wrote:

> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

That can be fine, I'd just like to have the end goal in mind.  For the near
future we should go with what actually works, definitely.

> >  - Asciidoc templates and processing should happen in a new directory
> >(perhaps imaginatively called "asciidoc"); having them in a directory
> >called "DocBook" seems a little weird.  More importantly, though, I'd
> >like to separate them out as a fresh start, and not mess with the
> >existing DocBook templates until we decide we don't need them anymore.
> >If we could end up with a cleaner, simpler makefile in the process,
> >that would be a bonus.  
> 
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

That's a good question.  I can certainly see the value of mixing the
templates with the rest, but that's a longer-term thing.  I'd sure like to
clean up the main Documentation/ directory before we start scattering
asciidoc stuff around there.  For the moment, I think my preference is
still to focus this work in one place where it's easily found and played
with.  Straightening out this directory is going to involve a fair amount
of moving stuff around, adding a few template files to that debt isn't
going to change the situation much.

> >  - I'm not sold on the new inclusion mechanism.  Creating thousands of
> >little files and tracking them for dependencies and such doesn't seem
> >like a simplification or a path toward better performance.  I would
> >like to at least consider keeping the direct-from-source inclusion.  
> 
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With those you can pass options to e.g. shift the hierarchy.
> With that you can do subheadings in DOC: sections and then seamlessly
> include them. Or similar stuff.
> 
> The speed-up due to parallelization is just a small bonus.
> 
> Also generating thousands of files is totally not unheard of in the kernel:
> 
> $ find include/config | wc -l
> 2623
> 
> None of those are in git.

Well, we are talking about an order of magnitude more files...  Still, I
said "not sold on" rather than "violently opposed to".  It does seem that
there are some good reasons for doing things this way, including, as Jani
said, getting rid of docproc and not mixing in a weird alien include
syntax.  I'm still not 100% sold, but I'll not hold up a working patch on
this point.

> >  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
> >it's going to work for now; that could maybe change after Al's work
> >goes in, which could be fairly soon.  
> 
> Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
> holds true for gpu docs already I think, and most other subsystems.
> The trouble iirc is all around asm and similar stuff, and we can't
> kerneldoc asm afaik.

asm stuff and things built into libraries.  But it does seem that this is
well on the way toward being fixed.

> One more thing we discussed: Did you ping kbuild folks already? Or
> want to get some agreement on the overall build process first?

Not yet.  It's worth doing...it would be nice, someday, if docs makefiles
could just have lines like:

  adoc-y += drm.txt ...

but I think we should work out how the pieces fit together before we get
too worried about the details of the build system.  What we have now isn't
particularly well integrated, we're not likely to make it all that much
worse...

Thanks,

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet  wrote:
> On Tue, 26 Jan 2016 14:08:45 +0200
> Jani Nikula  wrote:
>
>> I'm afraid we've done some overlapping work in the mean time, but I'm
>> happy we've both looked at the tool chain, and can have a more
>> meaningful conversation now.
>
> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]
>
> So I've spent a bit of time looking at this, and quite a bit more time
> talking with various folks at LCA.  There is pretty much universal
> agreement that this is interesting work and the direction we'd like to
> go.  My current hope is that we can merge some version of it for 4.6 and
> see where it goes from there.
>
> So naturally I have some thoughts on the whole thing...
>
>  - I would like to format directly to HTML if at all possible.  It seems
>it should be possible to get a table of contents into the files, and
>the feedback I got was that a TOC would be enough for navigation - it
>would not be necessary to split the files at that point.  We might
>still want to try to figure that out too, though.  In any case, this
>isn't a show stopper, in that we can change it anytime if a better way
>shows up.  But I'd like to have it in mind.

I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
toolchain, since that's less disruptive. And with that we can also
fully concentrating on the frontend, and how it'll look and behave.

Once that's solid we can look into the icing on the cake for later
kernels I think.

>  - Asciidoc templates and processing should happen in a new directory
>(perhaps imaginatively called "asciidoc"); having them in a directory
>called "DocBook" seems a little weird.  More importantly, though, I'd
>like to separate them out as a fresh start, and not mess with the
>existing DocBook templates until we decide we don't need them anymore.
>If we could end up with a cleaner, simpler makefile in the process,
>that would be a bonus.

For the long term dream plan of including other .txt files from the
existing pile of unstructured docs, do we really want a separate
asciidoc directory? Or just .asciidoc as a special extension?

>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>little files and tracking them for dependencies and such doesn't seem
>like a simplification or a path toward better performance.  I would
>like to at least consider keeping the direct-from-source inclusion.

The motivation behind the new inclusion mechanism isn't the speed-up
due to parallelization, but being able to use native asciidoc
includes. With those you can pass options to e.g. shift the hierarchy.
With that you can do subheadings in DOC: sections and then seamlessly
include them. Or similar stuff.

The speed-up due to parallelization is just a small bonus.

Also generating thousands of files is totally not unheard of in the kernel:

$ find include/config | wc -l
2623

None of those are in git.

>  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
>it's going to work for now; that could maybe change after Al's work
>goes in, which could be fairly soon.

Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
holds true for gpu docs already I think, and most other subsystems.
The trouble iirc is all around asm and similar stuff, and we can't
kerneldoc asm afaik.

> Please let me know your thoughts on the above.  Do you think you can find
> some time over the next month for this?  I'll try to shake loose some time
> too, but, well, $EXCUSES...

One more thing we discussed: Did you ping kbuild folks already? Or
want to get some agreement on the overall build process first?

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

Jonathan Corbet  writes:

> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]

Thanks.

>  - I would like to format directly to HTML if at all possible.

Agreed. asciidoc's docbook path seems to only increase the amount of
software involved.

>It seems it should be possible to get a table of contents into the
>files, and the feedback I got was that a TOC would be enough for
>navigation

I spent a few hours on the flight home reading asciidoc source code, and
it is a stream processor with a stack. The output is generated with some
simple templates, one for each backend. Here's the xhtml1.1 template for
section level sections: (sect1 in the .conf file):

[sect1]

{numbered?{sectnum} }{title}

|



The contents of the section get inserted at the |; it's nesting, so
[sect2] bits would get expanded while being processed.

Each asciidoc backend has dramatically different functionality. It's
pretty clear to me that the 'docbook' backend has the best support for
larger documents as that provides 'book-scale' processing bits. I've
recently written a book in asciidoc using the docbook backend, and the
html and pdf results are quite comparable. Using the html backend from
asciidoc yields a significantly different result.

I think it should be pretty easy to hack asciidoc to add diversions to
hold TOC contents while generating the rest of the doc and then replay
the diversion into the final document. Something like:

[sect1]

{numbered?{sectnum} }{title}
|>"table-of-contents"


{numbered?{sectnum} {title}




|



At the end of the document, we'd have some way of wrapping the diversion
in suitable additional bits to  complete the TOC, which would then be
formatted by CSS.

This same technique could be used to create lists of figures and tables.

The goal would be to create an html document which could be used without
javascript, and that would work without css as well.

-- 
-keith


signature.asc
Description: PGP signature

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula  wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

[Adding Keith since you said you wanted to be a part of this - let us know
when you've had enough!]

So I've spent a bit of time looking at this, and quite a bit more time
talking with various folks at LCA.  There is pretty much universal
agreement that this is interesting work and the direction we'd like to
go.  My current hope is that we can merge some version of it for 4.6 and
see where it goes from there.

So naturally I have some thoughts on the whole thing...

 - I would like to format directly to HTML if at all possible.  It seems
   it should be possible to get a table of contents into the files, and
   the feedback I got was that a TOC would be enough for navigation - it
   would not be necessary to split the files at that point.  We might
   still want to try to figure that out too, though.  In any case, this
   isn't a show stopper, in that we can change it anytime if a better way
   shows up.  But I'd like to have it in mind.

 - Asciidoc templates and processing should happen in a new directory
   (perhaps imaginatively called "asciidoc"); having them in a directory
   called "DocBook" seems a little weird.  More importantly, though, I'd
   like to separate them out as a fresh start, and not mess with the
   existing DocBook templates until we decide we don't need them anymore.
   If we could end up with a cleaner, simpler makefile in the process,
   that would be a bonus.

 - I'm not sold on the new inclusion mechanism.  Creating thousands of
   little files and tracking them for dependencies and such doesn't seem
   like a simplification or a path toward better performance.  I would
   like to at least consider keeping the direct-from-source inclusion.  

 - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
   it's going to work for now; that could maybe change after Al's work
   goes in, which could be fairly soon.

Please let me know your thoughts on the above.  Do you think you can find
some time over the next month for this?  I'll try to shake loose some time
too, but, well, $EXCUSES...

Many thanks for doing this work!

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula  wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

[Adding Keith since you said you wanted to be a part of this - let us know
when you've had enough!]

So I've spent a bit of time looking at this, and quite a bit more time
talking with various folks at LCA.  There is pretty much universal
agreement that this is interesting work and the direction we'd like to
go.  My current hope is that we can merge some version of it for 4.6 and
see where it goes from there.

So naturally I have some thoughts on the whole thing...

 - I would like to format directly to HTML if at all possible.  It seems
   it should be possible to get a table of contents into the files, and
   the feedback I got was that a TOC would be enough for navigation - it
   would not be necessary to split the files at that point.  We might
   still want to try to figure that out too, though.  In any case, this
   isn't a show stopper, in that we can change it anytime if a better way
   shows up.  But I'd like to have it in mind.

 - Asciidoc templates and processing should happen in a new directory
   (perhaps imaginatively called "asciidoc"); having them in a directory
   called "DocBook" seems a little weird.  More importantly, though, I'd
   like to separate them out as a fresh start, and not mess with the
   existing DocBook templates until we decide we don't need them anymore.
   If we could end up with a cleaner, simpler makefile in the process,
   that would be a bonus.

 - I'm not sold on the new inclusion mechanism.  Creating thousands of
   little files and tracking them for dependencies and such doesn't seem
   like a simplification or a path toward better performance.  I would
   like to at least consider keeping the direct-from-source inclusion.  

 - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
   it's going to work for now; that could maybe change after Al's work
   goes in, which could be fairly soon.

Please let me know your thoughts on the above.  Do you think you can find
some time over the next month for this?  I'll try to shake loose some time
too, but, well, $EXCUSES...

Many thanks for doing this work!

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 07:48:21 -0700
Jonathan Corbet  wrote:

> For HTML-page splitting, we can see if the tools can help us, consider
> splitting the template files, or do the splitting in a postprocessing
> step.  Docproc (or whatever replaces it) could also maybe do that work.
> It doesn't seem to me something that should force the inclusion of an
> entire XML-based processing step.  
> 
> 
> 
> But then, I'm here spouting ideas without any proof to back them up again,
> so who knows...:)
> 
> Thanks for doing this work.  As you said, it has been demonstrated 

So let's try this again without hitting that "send" button prematurely,
sigh.  Claws is annoying sometimes.

You'd said:

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)

I'd like to solve the cross-reference problem anyway; the documents
shouldn't be silos unto themselves.  Sphinx and ReST can do that; I *bet*
the other formats can too.  Maybe I can dig into some of that in my
upcoming airplane time (LAX->MEL, 15:45, sigh).

> I should also remind us that the original goal was to enable lightweight
> markup for documentation. This seems very much achievable now.

Agreed, nice work.

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula  wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

Overlapping work is just how this kernel thing works :)

> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

So I'm not really going to have time to dig too much more into this until
after LCA, so these are really quick impressions for now.

I would *really* like to get the XML step out of the processing path if
possible.  It adds complexity, makes it harder for others to build the
docs, makes things more fragile, and slows it all down.  It seems to me
that it should be possible to do that.

The issues, it seems, are splitting the output files and format support.
The latter isn't really an issue, I don't think; there are tools to do all
kinds of format conversions.  The only one that's even slightly weird is
man, and kernel-doc already has some (unused, I think) provisions for
doing that.  We could generate man pages directly without much pain.

For HTML-page splitting, we can see if the tools can help us, consider
splitting the template files, or do the splitting in a postprocessing
step.  Docproc (or whatever replaces it) could also maybe do that work.
It doesn't seem to me something that should force the inclusion of an
entire XML-based processing step.  



But then, I'm here spouting ideas without any proof to back them up again,
so who knows...:)

Thanks for doing this work.  As you said, it has been demonstrated 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Daniel Vetter  wrote:
> On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula  wrote:
>> I first took roughly the same approach as you did. I was really
>> impressed with the speed and the beauty of the produced HTML. The
>> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
>> to several pages) HTML directly. This is a showstopper for the gpu
>> document which turns into 1.3 MB of HTML, which looks pretty but is a
>> paint to navigate. To do chunked output, you have to output DocBook and
>> handle that like we do now. So while I would like to have asciidoc
>> generate HTML directly for speed and beauty, I ended up going the
>> asciidoc to DocBook path. The upside is all the output formats are
>> supported.
>
> This is a big bummer since with the parralized kernel-doc processing
> using Makefiles and using asciidoctor even building something big like
> the gpu docs is down to 2-3 seconds. From a clean tree, so not even
> counting incremental speed-ups. Unfortunately asciidoc doesn't have an
> built-in tooling (there's some experimental extensions) to split
> things up.

Basically asciidoc -> HTML is about as fast as asciidoc -> XML, and with
parallel kernel-doc it really is fast. Sadly the XML -> HTML part still
takes forever.

I just want to emphasize that we can get parallel kernel-doc with either
pipeline. It is also possible to enable both pipelines, i.e. have a fast
path HTML generation with few external dependencies and the Swiss army
knife slow path with XML.

I should also remind us that the original goal was to enable lightweight
markup for documentation. This seems very much achievable now. We don't
have to solve all the existing problems with the XML pipeline right
now. And asciidoc suits this well, as it can also feed to the existing
pipeline.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula  wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet  wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work.  Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start.  What this series has is:
>>
>>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>>doing this!  It was the kickstart that was needed to get this process
>>going.
>
> Hooray! :)
>
>>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>>".adt", it's an asciidoc template; it's processed pretty much the same
>>way, except that kernel-doc gets the -asciidoc argument.
>>
>>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>>Naturally this was where most of the time got spent.  *Only* HTML output
>>works at the moment.
>>
>>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>>and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats.  asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
>> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
>   kernel-doc: rewrite usage description, remove duplicated comments
>   kernel-doc: add support for asciidoc output
>   kernel-doc: support printing exported and non-exported symbols
>   kernel-doc: add support for printing DOC: comments with escaped names
>   scripts: add asciidoc-includes to extract includes from asciidoc
>   scripts: add a kernel-doc helper for special invocation
>   scripts: add tool for generating asciidoc dependencies and rules
>   scripts: add a crude converter from DocBook tmpl to asciidoc
>   Documentation: convert gpu.tmpl to gpu.txt
>   Documentation: build asciidoc documentation
>
>  Documentation/DocBook/Makefile |   37 +-
>  Documentation/DocBook/gpu.tmpl | 3499 
> 
>  Documentation/DocBook/gpu.txt  | 1183 ++
>  scripts/asciidoc-includes  |6 +
>  scripts/kernel-doc |  365 -
>  scripts/kernel-doc-deps| 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Jonathan Corbet  wrote:
> So here is a proof-of-concept series showing how a fully asciidoc-based
> toolchain might work.  Lots of hackery here, this isn't meant to be applied
> to anything at this point, but it's a good start.  What this series has is:
>
>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>doing this!  It was the kickstart that was needed to get this process
>going.

Hooray! :)

>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>".adt", it's an asciidoc template; it's processed pretty much the same
>way, except that kernel-doc gets the -asciidoc argument.
>
>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>Naturally this was where most of the time got spent.  *Only* HTML output
>works at the moment.
>
>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>and the output is much pleasing, IMO.
>
> I'm sure there's a thousand details to deal with, and there is the issue of
> the other output formats.  asciidoctor claims to be able to create man
> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
> straightforward, so it should be possible to use xmlto as is done now.
>
> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
> anybody wants to mess with it.
>
> Comments?

I'm afraid we've done some overlapping work in the mean time, but I'm
happy we've both looked at the tool chain, and can have a more
meaningful conversation now.

I first took roughly the same approach as you did. I was really
impressed with the speed and the beauty of the produced HTML. The
trouble is, neither asciidoc nor asciidoctor can produce chunked (split
to several pages) HTML directly. This is a showstopper for the gpu
document which turns into 1.3 MB of HTML, which looks pretty but is a
paint to navigate. To do chunked output, you have to output DocBook and
handle that like we do now. So while I would like to have asciidoc
generate HTML directly for speed and beauty, I ended up going the
asciidoc to DocBook path. The upside is all the output formats are
supported.

(We could of course split the source documents, but then I believe we'd
have lots of trouble cross-referencing between the documents. I could be
proven wrong. I'd *like* to be proven wrong.)

One significant difference between our approaches is that I ditched
docproc out of the equation. Instead of having the docproc ! directives
in the asciidoc, I opted for using asciidoc's native include macro, with
specially crafted filename suffixes to specify what parts of the source
to include. Those files are then generated as intermediate asciidoc
files using kernel-doc, with dependencies set right. There's a bunch of
scripting involved, but it's pretty straight forward. 

So you'd have, for example,

include::drivers/gpu/drm/drm_ioctl.c,export[]

to include all the exported functions, or

include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]

to include the DOC: section. I think I'd like some version of this,
regardless of whether asciidoc generates HTML or DocBook. It lets make
parallelize all of kernel-doc processing for free, which is a big win.

Patches follow, also available in kernel-asciidoc branch of
git://people.freedesktop.org/~jani/drm (web interface
http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)

BR,
Jani.


Jani Nikula (10):
  kernel-doc: rewrite usage description, remove duplicated comments
  kernel-doc: add support for asciidoc output
  kernel-doc: support printing exported and non-exported symbols
  kernel-doc: add support for printing DOC: comments with escaped names
  scripts: add asciidoc-includes to extract includes from asciidoc
  scripts: add a kernel-doc helper for special invocation
  scripts: add tool for generating asciidoc dependencies and rules
  scripts: add a crude converter from DocBook tmpl to asciidoc
  Documentation: convert gpu.tmpl to gpu.txt
  Documentation: build asciidoc documentation

 Documentation/DocBook/Makefile |   37 +-
 Documentation/DocBook/gpu.tmpl | 3499 
 Documentation/DocBook/gpu.txt  | 1183 ++
 scripts/asciidoc-includes  |6 +
 scripts/kernel-doc |  365 -
 scripts/kernel-doc-deps|   66 +
 scripts/kernel-doc-helper  |   70 +
 scripts/tmpl2asciidoc  |   39 +
 8 files changed, 1709 insertions(+), 3556 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt
 create mode 100755 scripts/asciidoc-includes
 create mode 100755 scripts/kernel-doc-deps
 create mode 100755 scripts/kernel-doc-helper
 create mode 100755 scripts/tmpl2asciidoc

-- 
2.1.4

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula  wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet  wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work.  Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start.  What this series has is:
>>
>>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>>doing this!  It was the kickstart that was needed to get this process
>>going.
>
> Hooray! :)
>
>>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>>".adt", it's an asciidoc template; it's processed pretty much the same
>>way, except that kernel-doc gets the -asciidoc argument.
>>
>>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>>Naturally this was where most of the time got spent.  *Only* HTML output
>>works at the moment.
>>
>>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>>and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats.  asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
>> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
>   kernel-doc: rewrite usage description, remove duplicated comments
>   kernel-doc: add support for asciidoc output
>   kernel-doc: support printing exported and non-exported symbols
>   kernel-doc: add support for printing DOC: comments with escaped names
>   scripts: add asciidoc-includes to extract includes from asciidoc
>   scripts: add a kernel-doc helper for special invocation
>   scripts: add tool for generating asciidoc dependencies and rules
>   scripts: add a crude converter from DocBook tmpl to asciidoc
>   Documentation: convert gpu.tmpl to gpu.txt
>   Documentation: build asciidoc documentation
>
>  Documentation/DocBook/Makefile |   37 +-
>  Documentation/DocBook/gpu.tmpl | 3499 
> 
>  Documentation/DocBook/gpu.txt  | 1183 ++
>  scripts/asciidoc-includes  |6 +
>  scripts/kernel-doc |  365 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Jonathan Corbet  wrote:
> So here is a proof-of-concept series showing how a fully asciidoc-based
> toolchain might work.  Lots of hackery here, this isn't meant to be applied
> to anything at this point, but it's a good start.  What this series has is:
>
>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>doing this!  It was the kickstart that was needed to get this process
>going.

Hooray! :)

>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>".adt", it's an asciidoc template; it's processed pretty much the same
>way, except that kernel-doc gets the -asciidoc argument.
>
>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>Naturally this was where most of the time got spent.  *Only* HTML output
>works at the moment.
>
>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>and the output is much pleasing, IMO.
>
> I'm sure there's a thousand details to deal with, and there is the issue of
> the other output formats.  asciidoctor claims to be able to create man
> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
> straightforward, so it should be possible to use xmlto as is done now.
>
> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
> anybody wants to mess with it.
>
> Comments?

I'm afraid we've done some overlapping work in the mean time, but I'm
happy we've both looked at the tool chain, and can have a more
meaningful conversation now.

I first took roughly the same approach as you did. I was really
impressed with the speed and the beauty of the produced HTML. The
trouble is, neither asciidoc nor asciidoctor can produce chunked (split
to several pages) HTML directly. This is a showstopper for the gpu
document which turns into 1.3 MB of HTML, which looks pretty but is a
paint to navigate. To do chunked output, you have to output DocBook and
handle that like we do now. So while I would like to have asciidoc
generate HTML directly for speed and beauty, I ended up going the
asciidoc to DocBook path. The upside is all the output formats are
supported.

(We could of course split the source documents, but then I believe we'd
have lots of trouble cross-referencing between the documents. I could be
proven wrong. I'd *like* to be proven wrong.)

One significant difference between our approaches is that I ditched
docproc out of the equation. Instead of having the docproc ! directives
in the asciidoc, I opted for using asciidoc's native include macro, with
specially crafted filename suffixes to specify what parts of the source
to include. Those files are then generated as intermediate asciidoc
files using kernel-doc, with dependencies set right. There's a bunch of
scripting involved, but it's pretty straight forward. 

So you'd have, for example,

include::drivers/gpu/drm/drm_ioctl.c,export[]

to include all the exported functions, or

include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]

to include the DOC: section. I think I'd like some version of this,
regardless of whether asciidoc generates HTML or DocBook. It lets make
parallelize all of kernel-doc processing for free, which is a big win.

Patches follow, also available in kernel-asciidoc branch of
git://people.freedesktop.org/~jani/drm (web interface
http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)

BR,
Jani.


Jani Nikula (10):
  kernel-doc: rewrite usage description, remove duplicated comments
  kernel-doc: add support for asciidoc output
  kernel-doc: support printing exported and non-exported symbols
  kernel-doc: add support for printing DOC: comments with escaped names
  scripts: add asciidoc-includes to extract includes from asciidoc
  scripts: add a kernel-doc helper for special invocation
  scripts: add tool for generating asciidoc dependencies and rules
  scripts: add a crude converter from DocBook tmpl to asciidoc
  Documentation: convert gpu.tmpl to gpu.txt
  Documentation: build asciidoc documentation

 Documentation/DocBook/Makefile |   37 +-
 Documentation/DocBook/gpu.tmpl | 3499 
 Documentation/DocBook/gpu.txt  | 1183 ++
 scripts/asciidoc-includes  |6 +
 scripts/kernel-doc |  365 -
 scripts/kernel-doc-deps|   66 +
 scripts/kernel-doc-helper  |   70 +
 scripts/tmpl2asciidoc  |   39 +
 8 files changed, 1709 insertions(+), 3556 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt
 create mode 100755 scripts/asciidoc-includes
 create mode 100755 scripts/kernel-doc-deps
 create mode 100755 scripts/kernel-doc-helper
 create mode 100755 scripts/tmpl2asciidoc

-- 
2.1.4

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 07:48:21 -0700
Jonathan Corbet  wrote:

> For HTML-page splitting, we can see if the tools can help us, consider
> splitting the template files, or do the splitting in a postprocessing
> step.  Docproc (or whatever replaces it) could also maybe do that work.
> It doesn't seem to me something that should force the inclusion of an
> entire XML-based processing step.  
> 
> 
> 
> But then, I'm here spouting ideas without any proof to back them up again,
> so who knows...:)
> 
> Thanks for doing this work.  As you said, it has been demonstrated 

So let's try this again without hitting that "send" button prematurely,
sigh.  Claws is annoying sometimes.

You'd said:

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)

I'd like to solve the cross-reference problem anyway; the documents
shouldn't be silos unto themselves.  Sphinx and ReST can do that; I *bet*
the other formats can too.  Maybe I can dig into some of that in my
upcoming airplane time (LAX->MEL, 15:45, sigh).

> I should also remind us that the original goal was to enable lightweight
> markup for documentation. This seems very much achievable now.

Agreed, nice work.

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula  wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

Overlapping work is just how this kernel thing works :)

> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

So I'm not really going to have time to dig too much more into this until
after LCA, so these are really quick impressions for now.

I would *really* like to get the XML step out of the processing path if
possible.  It adds complexity, makes it harder for others to build the
docs, makes things more fragile, and slows it all down.  It seems to me
that it should be possible to do that.

The issues, it seems, are splitting the output files and format support.
The latter isn't really an issue, I don't think; there are tools to do all
kinds of format conversions.  The only one that's even slightly weird is
man, and kernel-doc already has some (unused, I think) provisions for
doing that.  We could generate man pages directly without much pain.

For HTML-page splitting, we can see if the tools can help us, consider
splitting the template files, or do the splitting in a postprocessing
step.  Docproc (or whatever replaces it) could also maybe do that work.
It doesn't seem to me something that should force the inclusion of an
entire XML-based processing step.  



But then, I'm here spouting ideas without any proof to back them up again,
so who knows...:)

Thanks for doing this work.  As you said, it has been demonstrated 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Daniel Vetter  wrote:
> On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula  wrote:
>> I first took roughly the same approach as you did. I was really
>> impressed with the speed and the beauty of the produced HTML. The
>> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
>> to several pages) HTML directly. This is a showstopper for the gpu
>> document which turns into 1.3 MB of HTML, which looks pretty but is a
>> paint to navigate. To do chunked output, you have to output DocBook and
>> handle that like we do now. So while I would like to have asciidoc
>> generate HTML directly for speed and beauty, I ended up going the
>> asciidoc to DocBook path. The upside is all the output formats are
>> supported.
>
> This is a big bummer since with the parralized kernel-doc processing
> using Makefiles and using asciidoctor even building something big like
> the gpu docs is down to 2-3 seconds. From a clean tree, so not even
> counting incremental speed-ups. Unfortunately asciidoc doesn't have an
> built-in tooling (there's some experimental extensions) to split
> things up.

Basically asciidoc -> HTML is about as fast as asciidoc -> XML, and with
parallel kernel-doc it really is fast. Sadly the XML -> HTML part still
takes forever.

I just want to emphasize that we can get parallel kernel-doc with either
pipeline. It is also possible to enable both pipelines, i.e. have a fast
path HTML generation with few external dependencies and the Swiss army
knife slow path with XML.

I should also remind us that the original goal was to enable lightweight
markup for documentation. This seems very much achievable now. We don't
have to solve all the existing problems with the XML pipeline right
now. And asciidoc suits this well, as it can also feed to the existing
pipeline.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center