This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 46241cac6a46 [SPARK-58817][DOC] Eliminate absolute line references for
`literalinclude`
46241cac6a46 is described below
commit 46241cac6a46ac1866eb533cef69375043016900
Author: Tian Gao <[email protected]>
AuthorDate: Mon Aug 17 18:56:58 2026 -0700
[SPARK-58817][DOC] Eliminate absolute line references for `literalinclude`
### What changes were proposed in this pull request?
* Replaced all the `:line:` reference for `literalinclude` with markers
* Added a simple safeguard to forbid future usages for `:line:` reference
### Why are the changes needed?
`:line:` is super fragile - any changes to the example code will either
loudly or silently fail the documentation build. We have no protection
mechanism for the source file itself (except some comments in the file which is
often ignored), so we should not rely on the assumption that the file will not
be changed.
marker based reference is much more resilient to changes to the file and
it's still very simple and readable.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI and Claude confirmed the output is byte-identical.
### Was this patch authored or co-authored using generative AI tooling?
Claude Code (Opus 4.8 high).
Closes #58035 from gaogaotiantian/eliminate-literalinclude.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
examples/src/main/python/sql/arrow.py | 26 +++++++++-
examples/src/main/python/sql/udtf.py | 29 +++++++----
python/docs/source/conf.py | 2 +
python/docs/source/forbid_literalinclude_lines.py | 62 +++++++++++++++++++++++
python/docs/source/tutorial/sql/arrow_pandas.rst | 33 ++++++++----
python/docs/source/tutorial/sql/python_udtf.rst | 27 ++++++----
6 files changed, 149 insertions(+), 30 deletions(-)
diff --git a/examples/src/main/python/sql/arrow.py
b/examples/src/main/python/sql/arrow.py
index 62d8385849ca..af51fb553205 100644
--- a/examples/src/main/python/sql/arrow.py
+++ b/examples/src/main/python/sql/arrow.py
@@ -22,7 +22,8 @@ Run with:
"""
# NOTE that this file is imported in tutorials in PySpark documentation.
-# The codes are referred via line numbers. See also `literalinclude` directive
in Sphinx.
+# The code blocks are referred to via `$example on/off$` markers, included in
the
+# docs with the `literalinclude` directive's `:start-after:`/`:end-before:`
options.
import pandas as pd
from typing import Iterator
@@ -34,6 +35,7 @@ require_minimum_pyarrow_version()
def dataframe_to_from_arrow_table_example(spark: SparkSession) -> None:
+ # $example on:dataframe_to_from_arrow_table$
import pyarrow as pa
import numpy as np
@@ -50,9 +52,11 @@ def dataframe_to_from_arrow_table_example(spark:
SparkSession) -> None:
# a: double
# b: double
# c: double
+ # $example off:dataframe_to_from_arrow_table$
def dataframe_with_arrow_example(spark: SparkSession) -> None:
+ # $example on:dataframe_with_arrow$
import numpy as np
import pandas as pd
@@ -69,9 +73,11 @@ def dataframe_with_arrow_example(spark: SparkSession) ->
None:
result_pdf = df.select("*").toPandas()
print("Pandas DataFrame result statistics:\n%s\n" %
str(result_pdf.describe()))
+ # $example off:dataframe_with_arrow$
def ser_to_frame_pandas_udf_example(spark: SparkSession) -> None:
+ # $example on:ser_to_frame_pandas_udf$
import pandas as pd
from pyspark.sql.functions import pandas_udf
@@ -97,9 +103,11 @@ def ser_to_frame_pandas_udf_example(spark: SparkSession) ->
None:
# |-- func(long_col, string_col, struct_col): struct (nullable = true)
# | |-- col1: string (nullable = true)
# | |-- col2: long (nullable = true)
+ # $example off:ser_to_frame_pandas_udf$
def ser_to_ser_pandas_udf_example(spark: SparkSession) -> None:
+ # $example on:ser_to_ser_pandas_udf$
import pandas as pd
from pyspark.sql.functions import col, pandas_udf
@@ -131,9 +139,11 @@ def ser_to_ser_pandas_udf_example(spark: SparkSession) ->
None:
# | 4|
# | 9|
# +-------------------+
+ # $example off:ser_to_ser_pandas_udf$
def iter_ser_to_iter_ser_pandas_udf_example(spark: SparkSession) -> None:
+ # $example on:iter_ser_to_iter_ser_pandas_udf$
from typing import Iterator
import pandas as pd
@@ -157,9 +167,11 @@ def iter_ser_to_iter_ser_pandas_udf_example(spark:
SparkSession) -> None:
# | 3|
# | 4|
# +-----------+
+ # $example off:iter_ser_to_iter_ser_pandas_udf$
def iter_sers_to_iter_ser_pandas_udf_example(spark: SparkSession) -> None:
+ # $example on:iter_sers_to_iter_ser_pandas_udf$
from typing import Iterator, Tuple
import pandas as pd
@@ -184,9 +196,11 @@ def iter_sers_to_iter_ser_pandas_udf_example(spark:
SparkSession) -> None:
# | 4|
# | 9|
# +-----------------------+
+ # $example off:iter_sers_to_iter_ser_pandas_udf$
def ser_to_scalar_pandas_udf_example(spark: SparkSession) -> None:
+ # $example on:ser_to_scalar_pandas_udf$
import pandas as pd
from pyspark.sql.functions import pandas_udf
@@ -229,9 +243,11 @@ def ser_to_scalar_pandas_udf_example(spark: SparkSession)
-> None:
# | 2| 5.0| 6.0|
# | 2|10.0| 6.0|
# +---+----+------+
+ # $example off:ser_to_scalar_pandas_udf$
def grouped_apply_in_pandas_example(spark: SparkSession) -> None:
+ # $example on:grouped_apply_in_pandas$
df = spark.createDataFrame(
[(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
("id", "v"))
@@ -251,9 +267,11 @@ def grouped_apply_in_pandas_example(spark: SparkSession)
-> None:
# | 2|-1.0|
# | 2| 4.0|
# +---+----+
+ # $example off:grouped_apply_in_pandas$
def map_in_pandas_example(spark: SparkSession) -> None:
+ # $example on:map_in_pandas$
df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
def filter_func(iterator: Iterator[pd.DataFrame]) ->
Iterator[pd.DataFrame]:
@@ -266,9 +284,11 @@ def map_in_pandas_example(spark: SparkSession) -> None:
# +---+---+
# | 1| 21|
# +---+---+
+ # $example off:map_in_pandas$
def cogrouped_apply_in_pandas_example(spark: SparkSession) -> None:
+ # $example on:cogrouped_apply_in_pandas$
import pandas as pd
df1 = spark.createDataFrame(
@@ -292,9 +312,11 @@ def cogrouped_apply_in_pandas_example(spark: SparkSession)
-> None:
# |20000101| 2|2.0| y|
# |20000102| 2|4.0|null|
# +--------+---+---+----+
+ # $example off:cogrouped_apply_in_pandas$
def arrow_python_udf_example(spark: SparkSession) -> None:
+ # $example on:arrow_python_udf$
from pyspark.sql.functions import udf
@udf(returnType='int') # A default, pickled Python UDF
@@ -314,6 +336,8 @@ def arrow_python_udf_example(spark: SparkSession) -> None:
# | 8| 8|
# +----------+----------------+
+ # $example off:arrow_python_udf$
+
if __name__ == "__main__":
spark = SparkSession \
diff --git a/examples/src/main/python/sql/udtf.py
b/examples/src/main/python/sql/udtf.py
index bff5182f8e16..ecb87c4e72ba 100644
--- a/examples/src/main/python/sql/udtf.py
+++ b/examples/src/main/python/sql/udtf.py
@@ -22,7 +22,8 @@ Run with:
"""
# NOTE that this file is imported in the tutorials in PySpark documentation.
-# The codes are referred via line numbers. See also `literalinclude` directive
in Sphinx.
+# The code blocks are referred to via `$example on/off$` markers, included in
the
+# docs with the `literalinclude` directive's `:start-after:`/`:end-before:`
options.
from pyspark.sql import SparkSession
from pyspark.sql.pandas.utils import require_minimum_pandas_version,
require_minimum_pyarrow_version
@@ -32,13 +33,15 @@ require_minimum_pyarrow_version()
def python_udtf_simple_example(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_simple_class$
# Define the UDTF class and implement the required `eval` method.
class SquareNumbers:
def eval(self, start: int, end: int):
for num in range(start, end + 1):
yield (num, num * num)
+ # $example off:python_udtf_simple_class$
+ # $example on:python_udtf_simple_udtf$
from pyspark.sql.functions import lit, udtf
# Create a UDTF using the class definition and the `udtf` function.
@@ -53,10 +56,11 @@ def python_udtf_simple_example(spark: SparkSession) -> None:
# | 2| 4|
# | 3| 9|
# +---+-------+
+ # $example off:python_udtf_simple_udtf$
def python_udtf_decorator_example(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_decorator$
from pyspark.sql.functions import lit, udtf
# Define a UDTF using the `udtf` decorator directly on the class.
@@ -75,10 +79,11 @@ def python_udtf_decorator_example(spark: SparkSession) ->
None:
# | 2| 4|
# | 3| 9|
# +---+-------+
+ # $example off:python_udtf_decorator$
def python_udtf_registration(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_registration$
from pyspark.sql.functions import udtf
@udtf(returnType="word: string")
@@ -114,20 +119,22 @@ def python_udtf_registration(spark: SparkSession) -> None:
# |Apache Spark|Apache|
# |Apache Spark| Spark|
# +------------+------+
+ # $example off:python_udtf_registration$
def python_udtf_arrow_example(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_arrow$
from pyspark.sql.functions import udtf
@udtf(returnType="c1: int, c2: int", useArrow=True)
class PlusOne:
def eval(self, x: int):
yield x, x + 1
+ # $example off:python_udtf_arrow$
def python_udtf_date_expander_example(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_date_expander$
from datetime import datetime, timedelta
from pyspark.sql.functions import lit, udtf
@@ -150,10 +157,11 @@ def python_udtf_date_expander_example(spark:
SparkSession) -> None:
# |2023-02-28|
# |2023-03-01|
# +----------+
+ # $example off:python_udtf_date_expander$
def python_udtf_terminate_example(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_terminate$
from pyspark.sql.functions import udtf
@udtf(returnType="cnt: int")
@@ -184,10 +192,11 @@ def python_udtf_terminate_example(spark: SparkSession) ->
None:
# | 4| 5|
# | 9| 5|
# +---+---+
+ # $example off:python_udtf_terminate$
def python_udtf_table_argument(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_table_argument$
from pyspark.sql.functions import udtf
from pyspark.sql.types import Row
@@ -208,10 +217,11 @@ def python_udtf_table_argument(spark: SparkSession) ->
None:
# | 8|
# | 9|
# +---+
+ # $example off:python_udtf_table_argument$
def python_udtf_table_argument_with_partitioning(spark: SparkSession) -> None:
-
+ # $example on:python_udtf_table_argument_with_partitioning$
from pyspark.sql.functions import udtf
from pyspark.sql.types import Row
@@ -285,6 +295,7 @@ def python_udtf_table_argument_with_partitioning(spark:
SparkSession) -> None:
# Clean up.
spark.sql("DROP TABLE values_table")
+ # $example off:python_udtf_table_argument_with_partitioning$
if __name__ == "__main__":
diff --git a/python/docs/source/conf.py b/python/docs/source/conf.py
index 53e940add09e..36da7608c05f 100644
--- a/python/docs/source/conf.py
+++ b/python/docs/source/conf.py
@@ -73,6 +73,8 @@ extensions = [
'IPython.sphinxext.ipython_console_highlighting',
'numpydoc', # handle NumPy documentation formatted docstrings.
'sphinx_plotly_directive', # For visualize plot result
+ # Local: forbid ':lines:' on 'literalinclude' (fragile line-number
pinning).
+ 'forbid_literalinclude_lines',
]
# sphinx copy button
diff --git a/python/docs/source/forbid_literalinclude_lines.py
b/python/docs/source/forbid_literalinclude_lines.py
new file mode 100644
index 000000000000..a8fa76672cb3
--- /dev/null
+++ b/python/docs/source/forbid_literalinclude_lines.py
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+A tiny Sphinx extension that forbids the ``:lines:`` option on
``literalinclude``.
+
+Pinning a snippet to absolute line numbers rots silently: any edit to the
included
+source file (adding an import, re-sorting imports, inserting a blank line)
shifts the
+range, so the docs either render the wrong code or fail the build with a
+"non-whitespace stripped by dedent" warning. Prefer selecting the region by
name with
+``:pyobject:`` or by sentinel comments with ``:start-after:`` /
``:end-before:``, both
+of which are robust to line movement.
+
+This overrides the built-in ``literalinclude`` directive to raise a build
error the
+moment ``:lines:`` is used. As the docs build treats warnings as errors, the
error is
+surfaced immediately in CI.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from docutils.nodes import Node
+from sphinx.application import Sphinx
+from sphinx.directives.code import LiteralInclude
+
+
+class LiteralIncludeNoLines(LiteralInclude):
+ """``literalinclude`` that rejects the fragile ``:lines:`` option."""
+
+ def run(self) -> list[Node]:
+ if "lines" in self.options:
+ raise self.error(
+ "literalinclude with ':lines:' is not allowed: it pins
absolute line "
+ "numbers, which silently break when the included file changes.
Select "
+ "the region by name with ':pyobject:', or bracket it with
sentinel "
+ "comments and use ':start-after:'/':end-before:' instead."
+ )
+ return super().run()
+
+
+def setup(app: Sphinx) -> Dict[str, Any]:
+ # override=True replaces the built-in ``literalinclude`` registration.
+ app.add_directive("literalinclude", LiteralIncludeNoLines, override=True)
+ return {
+ "parallel_read_safe": True,
+ "parallel_write_safe": True,
+ }
diff --git a/python/docs/source/tutorial/sql/arrow_pandas.rst
b/python/docs/source/tutorial/sql/arrow_pandas.rst
index 5e60090ec3db..2ef306b0460a 100644
--- a/python/docs/source/tutorial/sql/arrow_pandas.rst
+++ b/python/docs/source/tutorial/sql/arrow_pandas.rst
@@ -48,7 +48,8 @@ with :meth:`DataFrame.toArrow`.
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 37-52
+ :start-after: $example on:dataframe_to_from_arrow_table$
+ :end-before: $example off:dataframe_to_from_arrow_table$
:dedent: 4
Note that :meth:`DataFrame.toArrow` results in the collection of all records
in the DataFrame to
@@ -69,7 +70,8 @@ This can be controlled by
``spark.sql.execution.arrow.pyspark.fallback.enabled``
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 56-71
+ :start-after: $example on:dataframe_with_arrow$
+ :end-before: $example off:dataframe_with_arrow$
:dedent: 4
Using the above optimizations with Arrow will produce the same results as when
Arrow is not
@@ -106,7 +108,8 @@ specify the type hints of ``pandas.Series`` and
``pandas.DataFrame`` as below:
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 75-99
+ :start-after: $example on:ser_to_frame_pandas_udf$
+ :end-before: $example off:ser_to_frame_pandas_udf$
:dedent: 4
In the following sections, it describes the combinations of the supported type
hints. For simplicity,
@@ -129,7 +132,8 @@ The following example shows how to create this Pandas UDF
that computes the prod
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 103-133
+ :start-after: $example on:ser_to_ser_pandas_udf$
+ :end-before: $example off:ser_to_ser_pandas_udf$
:dedent: 4
For detailed usage, please see :func:`pandas_udf`.
@@ -168,7 +172,8 @@ The following example shows how to create this Pandas UDF:
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 137-159
+ :start-after: $example on:iter_ser_to_iter_ser_pandas_udf$
+ :end-before: $example off:iter_ser_to_iter_ser_pandas_udf$
:dedent: 4
For detailed usage, please see :func:`pandas_udf`.
@@ -190,7 +195,8 @@ The following example shows how to create this Pandas UDF:
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 163-186
+ :start-after: $example on:iter_sers_to_iter_ser_pandas_udf$
+ :end-before: $example off:iter_sers_to_iter_ser_pandas_udf$
:dedent: 4
For detailed usage, please see :func:`pandas_udf`.
@@ -221,7 +227,8 @@ and window operations:
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 190-231
+ :start-after: $example on:ser_to_scalar_pandas_udf$
+ :end-before: $example off:ser_to_scalar_pandas_udf$
:dedent: 4
.. currentmodule:: pyspark.sql.functions
@@ -286,7 +293,8 @@ in the group.
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 235-253
+ :start-after: $example on:grouped_apply_in_pandas$
+ :end-before: $example off:grouped_apply_in_pandas$
:dedent: 4
For detailed usage, please see please see :meth:`GroupedData.applyInPandas`
@@ -304,7 +312,8 @@ The following example shows how to use
:meth:`DataFrame.mapInPandas`:
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 257-268
+ :start-after: $example on:map_in_pandas$
+ :end-before: $example off:map_in_pandas$
:dedent: 4
For detailed usage, please see :meth:`DataFrame.mapInPandas`.
@@ -343,7 +352,8 @@ The following example shows how to use
``DataFrame.groupby().cogroup().applyInPa
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 272-294
+ :start-after: $example on:cogrouped_apply_in_pandas$
+ :end-before: $example off:cogrouped_apply_in_pandas$
:dedent: 4
@@ -365,7 +375,8 @@ Here's an example that demonstrates the usage of both a
default, pickled Python
.. literalinclude:: ../../../../../examples/src/main/python/sql/arrow.py
:language: python
- :lines: 298-316
+ :start-after: $example on:arrow_python_udf$
+ :end-before: $example off:arrow_python_udf$
:dedent: 4
Type coercion:
diff --git a/python/docs/source/tutorial/sql/python_udtf.rst
b/python/docs/source/tutorial/sql/python_udtf.rst
index e2d240a6f9de..8afc8dc007cd 100644
--- a/python/docs/source/tutorial/sql/python_udtf.rst
+++ b/python/docs/source/tutorial/sql/python_udtf.rst
@@ -421,7 +421,8 @@ Python UDTFs can be registered and used in SQL queries.
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 82-116
+ :start-after: $example on:python_udtf_registration$
+ :end-before: $example off:python_udtf_registration$
:dedent: 4
@@ -440,7 +441,8 @@ when declaring the UDTF.
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 121-126
+ :start-after: $example on:python_udtf_arrow$
+ :end-before: $example off:python_udtf_arrow$
:dedent: 4
@@ -454,7 +456,8 @@ Here is a simple example of a UDTF class implementation:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 36-40
+ :start-after: $example on:python_udtf_simple_class$
+ :end-before: $example off:python_udtf_simple_class$
:dedent: 4
@@ -462,7 +465,8 @@ To make use of the UDTF, you'll first need to instantiate
it using the ``@udtf``
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 42-55
+ :start-after: $example on:python_udtf_simple_udtf$
+ :end-before: $example off:python_udtf_simple_udtf$
:dedent: 4
@@ -470,21 +474,24 @@ An alternative way to create a UDTF is to use the
:func:`udtf` function:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 60-77
+ :start-after: $example on:python_udtf_decorator$
+ :end-before: $example off:python_udtf_decorator$
:dedent: 4
Here is a Python UDTF that expands date ranges into individual dates:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 131-152
+ :start-after: $example on:python_udtf_date_expander$
+ :end-before: $example off:python_udtf_date_expander$
:dedent: 4
Here is a Python UDTF with ``__init__`` and ``terminate``:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 157-186
+ :start-after: $example on:python_udtf_terminate$
+ :end-before: $example off:python_udtf_terminate$
:dedent: 4
@@ -510,7 +517,8 @@ For example:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 191-210
+ :start-after: $example on:python_udtf_table_argument$
+ :end-before: $example off:python_udtf_table_argument$
:dedent: 4
When calling a UDTF with a table argument, any SQL query can request that the
input table be
@@ -535,7 +543,8 @@ For example:
.. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py
:language: python
- :lines: 215-287
+ :start-after: $example on:python_udtf_table_argument_with_partitioning$
+ :end-before: $example off:python_udtf_table_argument_with_partitioning$
:dedent: 4
Note that in for each of these ways of partitioning the input table when
calling UDTFs in SQL
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
