[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Tom Stellard via Phabricator via cfe-commits]

tstellar added a subscriber: gkistanova.
tstellar added a comment.

@luke You'll need to contact the bot owners so they can update machine.  The 
workers running the publish-sphinx-docs will need to be updated too.  
@gkistanova should be able to help.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke added a comment.

In D142199#4076287 , 
@kiranchandramohan wrote:

> The flang website (flang.llvm.org) or flang.llvm.org/docs are built from 
> these. Would you know whether there are changes required are there? 
> https://github.com/llvm/llvm-zorg/blob/main/zorg/buildbot/builders/SphinxDocsBuilder.py

The `docs-flang-html` target should remain the same, I quickly checked that 
script and it seems fine. The main thing is that the builders would need to 
have `myst-parser` installed on them. Are these python dependencies installed 
via a script or is it done manually?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Kiran Chandramohan via Phabricator via cfe-commits]

kiranchandramohan added a comment.

The flang website (flang.llvm.org) or flang.llvm.org/docs are built from these. 
Would you know whether there are changes required are there? 
https://github.com/llvm/llvm-zorg/blob/main/zorg/buildbot/builders/SphinxDocsBuilder.py


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke added a comment.

In D142199#4073815 , @tstellar wrote:

> LGTM.  Can you add something to the release notes about this.

Done. Do any build machines need to be upgraded before this is landed?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke updated this revision to Diff 491667.
luke added a comment.

Update release notes


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

Files:
  .github/workflows/release-tasks.yml
  clang/docs/conf.py
  flang/docs/ComplexOperations.md
  flang/docs/DiagnosticsReference.md
  flang/docs/FIRArrayOperations.md
  flang/docs/FlangDriver.md
  flang/docs/HighLevelFIR.md
  flang/docs/OptionComparison.md
  flang/docs/ParameterizedDerivedTypes.md
  flang/docs/PolymorphicEntities.md
  flang/docs/PullRequestChecklist.md
  flang/docs/conf.py
  flang/docs/index.md
  flang/include/flang/Optimizer/Dialect/FIROps.td
  
llvm/docs/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack.md
  llvm/docs/DeveloperPolicy.rst
  llvm/docs/GitRepositoryPolicy.md
  llvm/docs/MarkdownQuickstartTemplate.md
  llvm/docs/PointerAuth.md
  llvm/docs/ReleaseNotes.rst
  llvm/docs/SpeculativeLoadHardening.md
  llvm/docs/SphinxQuickstartTemplate.rst
  llvm/docs/conf.py
  llvm/utils/release/build-docs.sh

Index: llvm/utils/release/build-docs.sh
===
--- llvm/utils/release/build-docs.sh
+++ llvm/utils/release/build-docs.sh
@@ -15,7 +15,7 @@
 # ninja-build gcc-c++
 #   * pip install sphinx-markdown-tables
 # * Ubuntu:
-#   * apt-get install doxygen sphinx-common python3-recommonmark \
+#   * apt-get install doxygen sphinx-common python3-myst-parser \
 # ninja-build graphviz texlive-font-utils
 #   * pip install sphinx-markdown-tables
 #======#
Index: llvm/docs/conf.py
===
--- llvm/docs/conf.py
+++ llvm/docs/conf.py
@@ -37,7 +37,7 @@
 }
 
 try:
-  import recommonmark
+  import myst_parser
 except ImportError:
   # manpages do not use any .md sources
   if not tags.has('builder-man'):
@@ -46,10 +46,11 @@
   import sphinx
   if sphinx.version_info >= (3, 0):
 # This requires 0.5 or later.
-extensions.append('recommonmark')
+extensions.append('myst_parser')
   else:
-source_parsers = {'.md': 'recommonmark.parser.CommonMarkParser'}
+source_parsers = {'.md': 'myst_parser.parsers.sphinx_.MystParser'}
   source_suffix['.md'] = 'markdown'
+  myst_heading_anchors = 7
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
Index: llvm/docs/SphinxQuickstartTemplate.rst
===
--- llvm/docs/SphinxQuickstartTemplate.rst
+++ llvm/docs/SphinxQuickstartTemplate.rst
@@ -164,7 +164,8 @@
  .++:..
   ...
 
-
+.. _generating the documentation:
+			 
 Generating the documentation
 
 
@@ -172,19 +173,19 @@
 see what they would look like. In addition to the normal
 `build tools `_
 you need to install `Sphinx`_ and the
-`recommonmark `_ extension.
+`myst-parser `_ extension.
 
 On Debian you can install these with:
 
 .. code-block:: console
 
-   sudo apt install -y sphinx-doc python-recommonmark-doc
+   sudo apt install -y sphinx-doc python3-myst-parser
 
-On Ubuntu use pip to get an up-to-date version of recommonmark:
+On Ubuntu use pip to get an up-to-date version of myst-parser:
 
 .. code-block:: console
 
-   sudo pip install sphinx recommonmark
+   sudo pip install sphinx myst-parser
 
 Then run cmake to build the documentation inside the ``llvm-project`` checkout:
 
Index: llvm/docs/SpeculativeLoadHardening.md
===
--- llvm/docs/SpeculativeLoadHardening.md
+++ llvm/docs/SpeculativeLoadHardening.md
@@ -1,6 +1,6 @@
 # Speculative Load Hardening
 
-### A Spectre Variant #1 Mitigation Technique
+## A Spectre Variant #1 Mitigation Technique
 
 Author: Chandler Carruth - [chandl...@google.com](mailto:chandl...@google.com)
 
Index: llvm/docs/ReleaseNotes.rst
===
--- llvm/docs/ReleaseNotes.rst
+++ llvm/docs/ReleaseNotes.rst
@@ -316,6 +316,16 @@
   would now be ``UNSUPPORTED: target=arm{{.*}}`` and ``XFAIL: windows``
   would now be ``XFAIL: target={{.*}}-windows{{.*}}``.
 
+* The HTML documentation now uses myst-parser instead of recommonmark, since the
+  `latter has been sunset
+  `_.
+  It can be installed via either apt or pip:
+
+  .. code-block:: bash
+
+ apt install python3-myst-parser
+ pip install myst-parser
+
 External Open Source Projects Using LLVM 15
 ===
 
Index: llvm/docs/PointerAuth.md
===
--- 

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Tom Stellard via Phabricator via cfe-commits]

tstellar accepted this revision.
tstellar added a comment.
This revision is now accepted and ready to land.

LGTM.  Can you add something to the release notes about this.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke updated this revision to Diff 491318.
luke added a comment.

Don't suppress misc.highlighting_failure warning

Instead just fix the flang code block that was failing to highlight


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

Files:
  .github/workflows/release-tasks.yml
  clang/docs/conf.py
  flang/docs/ComplexOperations.md
  flang/docs/DiagnosticsReference.md
  flang/docs/FIRArrayOperations.md
  flang/docs/FlangDriver.md
  flang/docs/HighLevelFIR.md
  flang/docs/OptionComparison.md
  flang/docs/ParameterizedDerivedTypes.md
  flang/docs/PolymorphicEntities.md
  flang/docs/PullRequestChecklist.md
  flang/docs/conf.py
  flang/docs/index.md
  flang/include/flang/Optimizer/Dialect/FIROps.td
  
llvm/docs/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack.md
  llvm/docs/DeveloperPolicy.rst
  llvm/docs/GitRepositoryPolicy.md
  llvm/docs/MarkdownQuickstartTemplate.md
  llvm/docs/PointerAuth.md
  llvm/docs/SpeculativeLoadHardening.md
  llvm/docs/SphinxQuickstartTemplate.rst
  llvm/docs/conf.py
  llvm/utils/release/build-docs.sh

Index: llvm/utils/release/build-docs.sh
===
--- llvm/utils/release/build-docs.sh
+++ llvm/utils/release/build-docs.sh
@@ -15,7 +15,7 @@
 # ninja-build gcc-c++
 #   * pip install sphinx-markdown-tables
 # * Ubuntu:
-#   * apt-get install doxygen sphinx-common python3-recommonmark \
+#   * apt-get install doxygen sphinx-common python3-myst-parser \
 # ninja-build graphviz texlive-font-utils
 #   * pip install sphinx-markdown-tables
 #======#
Index: llvm/docs/conf.py
===
--- llvm/docs/conf.py
+++ llvm/docs/conf.py
@@ -37,7 +37,7 @@
 }
 
 try:
-  import recommonmark
+  import myst_parser
 except ImportError:
   # manpages do not use any .md sources
   if not tags.has('builder-man'):
@@ -46,10 +46,11 @@
   import sphinx
   if sphinx.version_info >= (3, 0):
 # This requires 0.5 or later.
-extensions.append('recommonmark')
+extensions.append('myst_parser')
   else:
-source_parsers = {'.md': 'recommonmark.parser.CommonMarkParser'}
+source_parsers = {'.md': 'myst_parser.parsers.sphinx_.MystParser'}
   source_suffix['.md'] = 'markdown'
+  myst_heading_anchors = 7
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
Index: llvm/docs/SphinxQuickstartTemplate.rst
===
--- llvm/docs/SphinxQuickstartTemplate.rst
+++ llvm/docs/SphinxQuickstartTemplate.rst
@@ -164,7 +164,8 @@
  .++:..
   ...
 
-
+.. _generating the documentation:
+			 
 Generating the documentation
 
 
@@ -172,19 +173,19 @@
 see what they would look like. In addition to the normal
 `build tools `_
 you need to install `Sphinx`_ and the
-`recommonmark `_ extension.
+`myst-parser `_ extension.
 
 On Debian you can install these with:
 
 .. code-block:: console
 
-   sudo apt install -y sphinx-doc python-recommonmark-doc
+   sudo apt install -y sphinx-doc python3-myst-parser
 
-On Ubuntu use pip to get an up-to-date version of recommonmark:
+On Ubuntu use pip to get an up-to-date version of myst-parser:
 
 .. code-block:: console
 
-   sudo pip install sphinx recommonmark
+   sudo pip install sphinx myst-parser
 
 Then run cmake to build the documentation inside the ``llvm-project`` checkout:
 
Index: llvm/docs/SpeculativeLoadHardening.md
===
--- llvm/docs/SpeculativeLoadHardening.md
+++ llvm/docs/SpeculativeLoadHardening.md
@@ -1,6 +1,6 @@
 # Speculative Load Hardening
 
-### A Spectre Variant #1 Mitigation Technique
+## A Spectre Variant #1 Mitigation Technique
 
 Author: Chandler Carruth - [chandl...@google.com](mailto:chandl...@google.com)
 
Index: llvm/docs/PointerAuth.md
===
--- llvm/docs/PointerAuth.md
+++ llvm/docs/PointerAuth.md
@@ -16,7 +16,7 @@
 * a [call operand bundle](#operand-bundle) (to authenticate called pointers)
 
 The current implementation leverages the
-[Armv8.3-A PAuth/Pointer Authentication Code](#armv8-3-a-pauth-pointer-authentication-code)
+[Armv8.3-A PAuth/Pointer Authentication Code](#armv83-a-pauth-pointer-authentication-code)
 instructions in the [AArch64 backend](#aarch64-support).
 This support is used to implement the Darwin arm64e ABI, as well as the
 [PAuth ABI Extension to ELF](https://github.com/ARM-software/abi-aa/blob/main/pauthabielf64/pauthabielf64.rst).
@@ -117,7 +117,7 @@
 signature is valid.
 
 

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke added inline comments.



Comment at: flang/docs/ComplexOperations.md:37
 **FIR**
-```c
 func.func @_QPpow_self(%arg0: !fir.ref>) -> !fir.complex<4> {

luke wrote:
> tstellar wrote:
> > Do these changes have any affected on the output html?  Is there another 
> > way to specify the language?
> I’m not sure, I think I might be able to add this back in though since I 
> suppressed a warning about highlight errors.
> I had to remove the ones that specified MLIR as the lexer though since there 
> didn’t seem to be an option to suppress those warnings. They didn’t seem to 
> receive any syntax highlighting in the output html anyway
Looks like these weren't being highlighted anyway: 
https://flang.llvm.org/docs/ComplexOperations.html?highlight=complex#fir-representation

Ideally there would eventually be an MLIR lexer added to pygments. Until then 
is it better to remove the incorrect lexers?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke added inline comments.



Comment at: flang/docs/ComplexOperations.md:37
 **FIR**
-```c
 func.func @_QPpow_self(%arg0: !fir.ref>) -> !fir.complex<4> {

tstellar wrote:
> Do these changes have any affected on the output html?  Is there another way 
> to specify the language?
I’m not sure, I think I might be able to add this back in though since I 
suppressed a warning about highlight errors.
I had to remove the ones that specified MLIR as the lexer though since there 
didn’t seem to be an option to suppress those warnings. They didn’t seem to 
receive any syntax highlighting in the output html anyway


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Tom Stellard via Phabricator via cfe-commits]

tstellar added inline comments.
Herald added a subscriber: asb.



Comment at: flang/docs/ComplexOperations.md:37
 **FIR**
-```c
 func.func @_QPpow_self(%arg0: !fir.ref>) -> !fir.complex<4> {

Do these changes have any affected on the output html?  Is there another way to 
specify the language?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke added inline comments.



Comment at: flang/docs/DiagnosticsReference.md:1
+% This file intentionally left blank. Silences a warning about a missing 
reference from the automatically generated FlangCommandLineReference.rst

This is pretty hairy and it will link users to a blank document. Open to any 
other ideas to workaround this


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D142199/new/

https://reviews.llvm.org/D142199

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D142199: [Docs] Replace recommonmark with myst-parser

[Luke Lau via Phabricator via cfe-commits]

luke created this revision.
luke added reviewers: kuhnel, Shivamgupta1234, mehdi_amini.
Herald added subscribers: kosarev, pmatos, kerbowa, arphaman, jvesely.
Herald added a reviewer: sscalpone.
Herald added projects: Flang, All.
luke requested review of this revision.
Herald added subscribers: llvm-commits, cfe-commits, jdoerfert.
Herald added a reviewer: nicolasvasilache.
Herald added projects: clang, LLVM.

Now that recommonmark is deprecated 
https://github.com/readthedocs/recommonmark/issues/221,
myst-parser seems to be the generally accepted drop in replacement.
This swaps recommonmark out of the sphinx config and upgrades any
markdown document that needs adjustment due to myst-parser's stricter
parsing.
In particular, these things needed addressed as myst-parser complained
about them:

- Broken references were replaced
- Invalid lexers in code blocks were replaced or removed (It didn't look like 
they were being syntax highlighted anyway)
- Non-contiguous headers (# -> ###) were made contiguous  (# -> ##)

Closes #55787


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D142199

Files:
  .github/workflows/release-tasks.yml
  clang/docs/conf.py
  flang/docs/ComplexOperations.md
  flang/docs/DiagnosticsReference.md
  flang/docs/FIRArrayOperations.md
  flang/docs/FlangDriver.md
  flang/docs/HighLevelFIR.md
  flang/docs/OptionComparison.md
  flang/docs/ParameterizedDerivedTypes.md
  flang/docs/PullRequestChecklist.md
  flang/docs/conf.py
  flang/docs/index.md
  flang/include/flang/Optimizer/Dialect/FIROps.td
  
llvm/docs/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack.md
  llvm/docs/DeveloperPolicy.rst
  llvm/docs/GitRepositoryPolicy.md
  llvm/docs/MarkdownQuickstartTemplate.md
  llvm/docs/PointerAuth.md
  llvm/docs/SpeculativeLoadHardening.md
  llvm/docs/SphinxQuickstartTemplate.rst
  llvm/docs/conf.py
  llvm/utils/release/build-docs.sh

Index: llvm/utils/release/build-docs.sh
===
--- llvm/utils/release/build-docs.sh
+++ llvm/utils/release/build-docs.sh
@@ -15,7 +15,7 @@
 # ninja-build gcc-c++
 #   * pip install sphinx-markdown-tables
 # * Ubuntu:
-#   * apt-get install doxygen sphinx-common python3-recommonmark \
+#   * apt-get install doxygen sphinx-common python3-myst-parser \
 # ninja-build graphviz texlive-font-utils
 #   * pip install sphinx-markdown-tables
 #======#
Index: llvm/docs/conf.py
===
--- llvm/docs/conf.py
+++ llvm/docs/conf.py
@@ -37,7 +37,7 @@
 }
 
 try:
-  import recommonmark
+  import myst_parser
 except ImportError:
   # manpages do not use any .md sources
   if not tags.has('builder-man'):
@@ -46,10 +46,11 @@
   import sphinx
   if sphinx.version_info >= (3, 0):
 # This requires 0.5 or later.
-extensions.append('recommonmark')
+extensions.append('myst_parser')
   else:
-source_parsers = {'.md': 'recommonmark.parser.CommonMarkParser'}
+source_parsers = {'.md': 'myst_parser.parsers.sphinx_.MystParser'}
   source_suffix['.md'] = 'markdown'
+  myst_heading_anchors = 7
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
Index: llvm/docs/SphinxQuickstartTemplate.rst
===
--- llvm/docs/SphinxQuickstartTemplate.rst
+++ llvm/docs/SphinxQuickstartTemplate.rst
@@ -164,7 +164,8 @@
  .++:..
   ...
 
-
+.. _generating the documentation:
+			 
 Generating the documentation
 
 
@@ -172,19 +173,19 @@
 see what they would look like. In addition to the normal
 `build tools `_
 you need to install `Sphinx`_ and the
-`recommonmark `_ extension.
+`myst-parser `_ extension.
 
 On Debian you can install these with:
 
 .. code-block:: console
 
-   sudo apt install -y sphinx-doc python-recommonmark-doc
+   sudo apt install -y sphinx-doc python3-myst-parser
 
-On Ubuntu use pip to get an up-to-date version of recommonmark:
+On Ubuntu use pip to get an up-to-date version of myst-parser:
 
 .. code-block:: console
 
-   sudo pip install sphinx recommonmark
+   sudo pip install sphinx myst-parser
 
 Then run cmake to build the documentation inside the ``llvm-project`` checkout:
 
Index: llvm/docs/SpeculativeLoadHardening.md
===
--- llvm/docs/SpeculativeLoadHardening.md
+++ llvm/docs/SpeculativeLoadHardening.md
@@ -1,6 +1,6 @@
 # Speculative Load Hardening
 
-### A Spectre Variant #1 Mitigation Technique
+## A Spectre Variant #1 Mitigation Technique
 
 Author: Chandler Carruth - [chandl...@google.com](mailto:chandl...@google.com)
 
Index: