Re: PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[James Graham]

On 27/08/2019 16:24, Andrew Halberstadt wrote:
RST is foreign to most 
It turns out* that firefox-source-docs also supports markdown format, so 
if you have existing docs in that format, or simply feel like needing to 
learn ReST is a significant impediment to writing docs, that option is 
available.


The tradeoff is that the featureset of ReST is much larger than that of 
markdown, so you have to give up some expressiveness and docs/source 
integration features of Sphinx.


* After some discussion with ahal about the merits of enabling markdown, 
it turned out that maja_zf already had and the marionette docs are using it.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[Nicholas Nethercote]

I recently wrote my first firefox-source-docs and I was surprised and
impressed by one thing in particular: when you run `./mach docs` or `./mach
docs ` it opens a Firefox instance that shows the generated HTML, and
it *also* starts up a server that automatically regenerates and reloads
that HTML every time a .rst file is saved. It makes for a very responsive
and pleasant editing experience.

Nick

On Wed, 28 Aug 2019 at 01:25, Andrew Halberstadt  wrote:

> Documentation is hard. It's difficult to write, difficult to find and
> always out
> of date. That's why we implemented our in-tree documentation system that
> underpins firefox-source-docs.mozilla.org. The documentation lives next
> to the
> code that it documents, so it can be updated within the same commit that
> makes
> the underlying changes. Unfortunately, this system has never been a
> pleasure to
> work with. Docs are slow to build, RST is foreign to most and the moz.build
> syntax to integrate them is confusing. Not to mention some serious bugs
> around
> how files are uploaded and stored in S3.
>
> I'm happy to announce many great new features for firefox-source-docs,
> courtesy
> of our GSoC student Shivam (aka :championshuttler). You can see a blog
> post he
> wrote on his experience here
> . Below
> is a list of all the work and awesome new
> features he added to our documentation infrastructure. Please note that bug
> 1576912 
> tracks saving this information into the docs themselves.
>
>
> *Server Synchronization*
>
> Previously, the doc system had a surprisingly major flaw. Docs were only
> ever
> uploaded to S3, never deleted. This meant that even if a source file no
> longer
> exists (e.g it was removed or renamed), the related page would continue to
> live
> on the server indefinitely. External links to the page would continue to
> function,
> the users of the link being none-the-wiser. Yikes!
>
> We now compare all the files that exist on the server against the list of
> source
> files in `mozilla-central`. Any files on the server that no longer exist in
> `mozilla-central` are removed.
>
>
> *Redirects*
>
> We now have the ability to define redirects in-tree! This will allow us to
> refactor and move docs around to our hearts content without needing to
> worry
> about stale external URLs. To set up a redirect simply add a line to this
> file:
>
> https://searchfox.org/mozilla-central/source/tools/docs/redirects.yml
>
> Any request starting with the prefix on the left, will be rewritten to the
> prefix on the right by the server. So for example a request to
> `/testing/marionette/marionette/index.html` will be re-written to
> `/testing/marionette/index.html`. Amazon's API only supports prefix
> redirects,
> so anything more complex isn't supported.
>
>
> *Nested Doc Trees*
>
> This has been implemented for awhile and wasn't part of the GSoC project,
> but
> due to a lack of redirects it didn't make sense to advertise too broadly.
> Up
> until recently, we've tended to add all doc folders to the root index. In
> other
> words each subsection is visible at the top level, creating a rather long
> and 
> incoherent  list of pages to
> the left.
>
> This feature essentially means we can now group related docs together under
> common "landing pages". This will allow us to refactor the docs into a
> structure
> that makes more sense. For example we could have a landing page for docs
> describing Gecko's internals, and another one for docs describing developer
> workflows in `mozilla-central`.
>
> Mark Banner has already started to organize some of the docs under some of
> these
> landing pages. See this commit
>  for a rough
> example.
>
> To clarify a few things:
>
> 1. The path specified in `SPHINX_TREES` *does not* need to correspond to
> a path
> in `mozilla-central`. For example, I could register my docs using
> `SPHINX_TREES["/foo"] = "docs"`, which would make that doc tree accessible
> at
> `firefox-source-docs.mozilla.org/foo`
> .
>
> 2. Any subtrees that are nested under another index will automatically be
> hidden
> from the main index.  This means you should make sure to link to any
> subtrees
> from somewhere in the landing page. So given my earlier doc tree at
> `/foo`, if I
> now created a subtree and registered it using `SPHINX_TREES["/foo/bar"] =
> "docs"`, those docs would *not* show up in the main index.
>
> 3. The relation between subtrees and their parents does not necessarily
> have any
> bearing with their relation on the file system. For example, a doc tree
> that
> lives under `/devtools` can be nested under an index that lives under
> `/browser`.
>
> I realize this explanation is about as clear as mud, so feel free 

Re: PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[Andrew Halberstadt]

We have some *very* basic documentation here:
https://firefox-source-docs.mozilla.org/#managing-documentation

But it's not good enough and is missing all of the features outlined here.
I've filed bug 1576912
 to make these docs
better. Shivam and I will try and get this updated shortly.

On Tue, Aug 27, 2019 at 1:20 PM David Teller  wrote:

> That sounds useful :)
>
> Do we have any documentation on how to add documentation?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[David Teller]

That sounds useful :)

Do we have any documentation on how to add documentation?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[Andrew Halberstadt]

Forgot to mention, if you run into any bugs using this infrastructure, file
under `Firefox Build System :: Generated Documentation`. If you have
feature requests or ideas to improve the system, please file a bug and make
it block the first-class-docs
 alias. If you would
like to attempt to restructure your docs using these new tools, please file
a bug and make it block bug 1513821
.

Thanks!

On Tue, Aug 27, 2019 at 11:24 AM Andrew Halberstadt 
wrote:

> Documentation is hard. It's difficult to write, difficult to find and
> always out
> of date. That's why we implemented our in-tree documentation system that
> underpins firefox-source-docs.mozilla.org. The documentation lives next
> to the
> code that it documents, so it can be updated within the same commit that
> makes
> the underlying changes. Unfortunately, this system has never been a
> pleasure to
> work with. Docs are slow to build, RST is foreign to most and the moz.build
> syntax to integrate them is confusing. Not to mention some serious bugs
> around
> how files are uploaded and stored in S3.
>
> I'm happy to announce many great new features for firefox-source-docs,
> courtesy
> of our GSoC student Shivam (aka :championshuttler). You can see a blog
> post he
> wrote on his experience here
> . Below
> is a list of all the work and awesome new
> features he added to our documentation infrastructure. Please note that bug
> 1576912 
> tracks saving this information into the docs themselves.
>
>
> *Server Synchronization*
>
> Previously, the doc system had a surprisingly major flaw. Docs were only
> ever
> uploaded to S3, never deleted. This meant that even if a source file no
> longer
> exists (e.g it was removed or renamed), the related page would continue to
> live
> on the server indefinitely. External links to the page would continue to
> function,
> the users of the link being none-the-wiser. Yikes!
>
> We now compare all the files that exist on the server against the list of
> source
> files in `mozilla-central`. Any files on the server that no longer exist in
> `mozilla-central` are removed.
>
>
> *Redirects*
>
> We now have the ability to define redirects in-tree! This will allow us to
> refactor and move docs around to our hearts content without needing to
> worry
> about stale external URLs. To set up a redirect simply add a line to this
> file:
>
> https://searchfox.org/mozilla-central/source/tools/docs/redirects.yml
>
> Any request starting with the prefix on the left, will be rewritten to the
> prefix on the right by the server. So for example a request to
> `/testing/marionette/marionette/index.html` will be re-written to
> `/testing/marionette/index.html`. Amazon's API only supports prefix
> redirects,
> so anything more complex isn't supported.
>
>
> *Nested Doc Trees*
>
> This has been implemented for awhile and wasn't part of the GSoC project,
> but
> due to a lack of redirects it didn't make sense to advertise too broadly.
> Up
> until recently, we've tended to add all doc folders to the root index. In
> other
> words each subsection is visible at the top level, creating a rather long
> and 
> incoherent  list of pages to
> the left.
>
> This feature essentially means we can now group related docs together under
> common "landing pages". This will allow us to refactor the docs into a
> structure
> that makes more sense. For example we could have a landing page for docs
> describing Gecko's internals, and another one for docs describing developer
> workflows in `mozilla-central`.
>
> Mark Banner has already started to organize some of the docs under some of
> these
> landing pages. See this commit
>  for a rough
> example.
>
> To clarify a few things:
>
> 1. The path specified in `SPHINX_TREES` *does not* need to correspond to
> a path
> in `mozilla-central`. For example, I could register my docs using
> `SPHINX_TREES["/foo"] = "docs"`, which would make that doc tree accessible
> at
> `firefox-source-docs.mozilla.org/foo`
> .
>
> 2. Any subtrees that are nested under another index will automatically be
> hidden
> from the main index.  This means you should make sure to link to any
> subtrees
> from somewhere in the landing page. So given my earlier doc tree at
> `/foo`, if I
> now created a subtree and registered it using `SPHINX_TREES["/foo/bar"] =
> "docs"`, those docs would *not* show up in the main index.
>
> 3. The relation between subtrees and their parents does not necessarily
> have any
> bearing with their relation on the file system. For example, a doc tree
> that
> lives under `/devtools` can be nested under an 

PSA: Improvements to infrastructure underpinning `firefox-source-docs`

[Andrew Halberstadt]

Documentation is hard. It's difficult to write, difficult to find and
always out
of date. That's why we implemented our in-tree documentation system that
underpins firefox-source-docs.mozilla.org. The documentation lives next to
the
code that it documents, so it can be updated within the same commit that
makes
the underlying changes. Unfortunately, this system has never been a
pleasure to
work with. Docs are slow to build, RST is foreign to most and the moz.build
syntax to integrate them is confusing. Not to mention some serious bugs
around
how files are uploaded and stored in S3.

I'm happy to announce many great new features for firefox-source-docs,
courtesy
of our GSoC student Shivam (aka :championshuttler). You can see a blog post
he
wrote on his experience here
. Below is
a list of all the work and awesome new
features he added to our documentation infrastructure. Please note that bug
1576912 
tracks saving this information into the docs themselves.


*Server Synchronization*

Previously, the doc system had a surprisingly major flaw. Docs were only
ever
uploaded to S3, never deleted. This meant that even if a source file no
longer
exists (e.g it was removed or renamed), the related page would continue to
live
on the server indefinitely. External links to the page would continue to
function,
the users of the link being none-the-wiser. Yikes!

We now compare all the files that exist on the server against the list of
source
files in `mozilla-central`. Any files on the server that no longer exist in
`mozilla-central` are removed.


*Redirects*

We now have the ability to define redirects in-tree! This will allow us to
refactor and move docs around to our hearts content without needing to worry
about stale external URLs. To set up a redirect simply add a line to this
file:

https://searchfox.org/mozilla-central/source/tools/docs/redirects.yml

Any request starting with the prefix on the left, will be rewritten to the
prefix on the right by the server. So for example a request to
`/testing/marionette/marionette/index.html` will be re-written to
`/testing/marionette/index.html`. Amazon's API only supports prefix
redirects,
so anything more complex isn't supported.


*Nested Doc Trees*

This has been implemented for awhile and wasn't part of the GSoC project,
but
due to a lack of redirects it didn't make sense to advertise too broadly. Up
until recently, we've tended to add all doc folders to the root index. In
other
words each subsection is visible at the top level, creating a rather long
and 
incoherent  list of pages to the
left.

This feature essentially means we can now group related docs together under
common "landing pages". This will allow us to refactor the docs into a
structure
that makes more sense. For example we could have a landing page for docs
describing Gecko's internals, and another one for docs describing developer
workflows in `mozilla-central`.

Mark Banner has already started to organize some of the docs under some of
these
landing pages. See this commit
 for a rough
example.

To clarify a few things:

1. The path specified in `SPHINX_TREES` *does not* need to correspond to a
path
in `mozilla-central`. For example, I could register my docs using
`SPHINX_TREES["/foo"] = "docs"`, which would make that doc tree accessible
at
`firefox-source-docs.mozilla.org/foo`.

2. Any subtrees that are nested under another index will automatically be
hidden
from the main index.  This means you should make sure to link to any
subtrees
from somewhere in the landing page. So given my earlier doc tree at `/foo`,
if I
now created a subtree and registered it using `SPHINX_TREES["/foo/bar"] =
"docs"`, those docs would *not* show up in the main index.

3. The relation between subtrees and their parents does not necessarily
have any
bearing with their relation on the file system. For example, a doc tree that
lives under `/devtools` can be nested under an index that lives under
`/browser`.

I realize this explanation is about as clear as mud, so feel free to reach
out
to me if you need a hand. Also keep an eye on bug 1576912
 which will provide a
better explanation of how registration works.


*ReStructuredText Linter*

RST isn't the easiest of markup languages, but it's powerful and  what
`Sphinx`
(the library used to build our docs) uses, so we're stuck with it. But at
least we
now have a linter which will catch basic problems in `.rst` files early. Be
sure to
run `./mach lint -o` to test your outgoing changes before submitting to
review!


*Mermaid Integration*

Mermaid  is a tool that lets you generate
flow charts, sequence diagrams, gantt
charts, class diagrams and vcs