This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new bbe0487d43d5 Revert "[MINOR][PYTHON][DOC] Fixed some links in
documentation."
bbe0487d43d5 is described below
commit bbe0487d43d5cfb6145312be878eb979c888f6d5
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Jul 22 10:03:12 2026 +0800
Revert "[MINOR][PYTHON][DOC] Fixed some links in documentation."
This reverts commit c198adcf3268e934f14a1bb76504b84163ef3bb5.
since it's failing the master CI
https://github.com/apache/spark/actions/runs/29875592188/job/88788942949
```
*****************************
* Building Python API docs. *
*****************************
Running Sphinx v8.2.3
loading translations [en]... done
making output directory... done
Converting `source_suffix = '.rst'` to `source_suffix = {'.rst':
'restructuredtext'}`.
[autosummary] generating autosummary for: development/contributing.rst,
development/debugging.rst, development/errors.rst, development/index.rst,
development/logger.rst, development/setting_ide.rst, development/testing.rst,
getting_started/index.rst, getting_started/install.rst,
getting_started/quickstart_connect.ipynb, ...,
tutorial/sql/type_conversions.rst, user_guide/ansi_migration_guide.ipynb,
user_guide/bugbusting.ipynb, user_guide/dataframes.ipynb,
user_guide/dataprep.ipynb, use [...]
WARNING: [autosummary] failed to import pyspark.sql.DataStreamReader.
Possible hints:
* AttributeError: module 'pyspark.sql' has no attribute 'DataStreamReader'
* ModuleNotFoundError: No module named 'pyspark.sql.DataStreamReader'
* ImportError:
WARNING: [autosummary] failed to import pyspark.sql.DataStreamWriter.
Possible hints:
* ModuleNotFoundError: No module named 'pyspark.sql.DataStreamWriter'
* AttributeError: module 'pyspark.sql' has no attribute 'DataStreamWriter'
* ImportError:
```
Closes #57415 from zhengruifeng/revert-57359-dev0.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit dccff5579c4e2f200dc4e7f1bb5d41ec1edccce0)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../source/reference/pyspark.sql/core_classes.rst | 3 ---
python/pyspark/pandas/frame.py | 24 +++++++++----------
python/pyspark/sql/dataframe.py | 4 +---
python/pyspark/sql/pandas/functions.py | 2 +-
python/pyspark/sql/tvf.py | 28 +++++++++++-----------
5 files changed, 27 insertions(+), 34 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/core_classes.rst
b/python/docs/source/reference/pyspark.sql/core_classes.rst
index 1ba4c62994b7..af38ef1c4f5e 100644
--- a/python/docs/source/reference/pyspark.sql/core_classes.rst
+++ b/python/docs/source/reference/pyspark.sql/core_classes.rst
@@ -38,9 +38,6 @@ Core Classes
DataFrameReader
DataFrameWriter
DataFrameWriterV2
- MergeIntoWriter
- DataStreamReader
- DataStreamWriter
UDFRegistration
UDTFRegistration
udf.UserDefinedFunction
diff --git a/python/pyspark/pandas/frame.py b/python/pyspark/pandas/frame.py
index 5bdc665a15e7..4b9ce84b3b7a 100644
--- a/python/pyspark/pandas/frame.py
+++ b/python/pyspark/pandas/frame.py
@@ -390,19 +390,6 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically.
This holds Spark
DataFrame internally.
- .. versionadded:: 3.2.0
-
- .. versionchanged:: 3.4.0
- Since 3.4.0, it deals with `data` and `index` in this approach:
- 1, when `data` is a distributed dataset (Internal DataFrame/Spark
DataFrame/
- pandas-on-Spark DataFrame/pandas-on-Spark Series), it will first
parallelize
- the `index` if necessary, and then try to combine the `data` and
`index`;
- Note that if `data` and `index` doesn't have the same anchor, then
- `compute.ops_on_diff_frames` should be turned on;
- 2, when `data` is a local dataset (Pandas DataFrame/numpy
ndarray/list/etc),
- it will first collect the `index` to driver if necessary, and then
apply
- the `pandas.DataFrame(...)` creation internally;
-
:ivar _internal: an internal immutable Frame to manage metadata.
:type _internal: InternalFrame
@@ -422,6 +409,17 @@ class DataFrame(Frame, Generic[T]):
copy : boolean, default False
Copy data from inputs. Only affects DataFrame / 2d ndarray input
+ .. versionchanged:: 3.4.0
+ Since 3.4.0, it deals with `data` and `index` in this approach:
+ 1, when `data` is a distributed dataset (Internal DataFrame/Spark
DataFrame/
+ pandas-on-Spark DataFrame/pandas-on-Spark Series), it will first
parallelize
+ the `index` if necessary, and then try to combine the `data` and
`index`;
+ Note that if `data` and `index` doesn't have the same anchor, then
+ `compute.ops_on_diff_frames` should be turned on;
+ 2, when `data` is a local dataset (Pandas DataFrame/numpy
ndarray/list/etc),
+ it will first collect the `index` to driver if necessary, and then
apply
+ the `pandas.DataFrame(...)` creation internally;
+
Examples
--------
Constructing DataFrame from a dictionary.
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index 508e83f84d4b..6c4d32ea1797 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -1206,8 +1206,6 @@ class DataFrame:
:class:`DataFrame`
Hinted DataFrame
- .. note:: See also `Hints
<https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html>`_
-
Examples
--------
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")],
schema=["age", "name"])
@@ -6484,7 +6482,7 @@ class DataFrame:
See Also
--------
- pyspark.pandas.DataFrame.to_spark
+ pyspark.pandas.frame.DataFrame.to_spark
Examples
--------
diff --git a/python/pyspark/sql/pandas/functions.py
b/python/pyspark/sql/pandas/functions.py
index 4b6699829996..b14b10b44859 100644
--- a/python/pyspark/sql/pandas/functions.py
+++ b/python/pyspark/sql/pandas/functions.py
@@ -74,7 +74,7 @@ def arrow_udf(f=None, returnType=None, functionType=None):
the return type of the user-defined function. The value can be either a
:class:`pyspark.sql.types.DataType` object or a DDL-formatted type
string.
functionType : int, optional
- an enum value in :class:`~pyspark.sql.functions.ArrowUDFType`.
+ an enum value in :class:`pyspark.sql.functions.ArrowUDFType`.
Default: SCALAR. This parameter exists for compatibility.
Using Python type hints is encouraged.
diff --git a/python/pyspark/sql/tvf.py b/python/pyspark/sql/tvf.py
index 57b0f525675f..90874578a356 100644
--- a/python/pyspark/sql/tvf.py
+++ b/python/pyspark/sql/tvf.py
@@ -59,7 +59,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
@@ -101,7 +101,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -185,7 +185,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -238,7 +238,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -301,7 +301,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -344,7 +344,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -394,7 +394,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -437,7 +437,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -490,7 +490,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
See Also
--------
@@ -525,7 +525,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
@@ -546,7 +546,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
@@ -574,7 +574,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
@@ -639,7 +639,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
@@ -681,7 +681,7 @@ class TableValuedFunction:
Returns
-------
- :class:`~pyspark.sql.DataFrame`
+ :class:`DataFrame`
Examples
--------
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
