This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 08866c280f87 [SPARK-47439][PYTHON] Document Python Data Source API in
API reference page
08866c280f87 is described below
commit 08866c280f877ce27d5c5305c7a09add76c86774
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Mon Mar 18 20:08:22 2024 +0900
[SPARK-47439][PYTHON] Document Python Data Source API in API reference page
### What changes were proposed in this pull request?
This PR proposes to document Python Data Source API in Python API reference
page.
### Why are the changes needed?
For users/developers to know how to use them.
### Does this PR introduce _any_ user-facing change?
Yes, it documents Python Data Source API.
### How was this patch tested?
Manually checked the output from Python API reference build
```bash
cd python/docs
make clean html
open build/html/index.html
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45561 from HyukjinKwon/SPARK-47439.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.../source/reference/pyspark.sql/core_classes.rst | 7 ++++
.../{core_classes.rst => datasource.rst} | 44 +++++++++++-----------
python/docs/source/reference/pyspark.sql/index.rst | 1 +
.../source/reference/pyspark.sql/spark_session.rst | 1 +
4 files changed, 31 insertions(+), 22 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/core_classes.rst
b/python/docs/source/reference/pyspark.sql/core_classes.rst
index 3cf19686cdd8..65096da21de5 100644
--- a/python/docs/source/reference/pyspark.sql/core_classes.rst
+++ b/python/docs/source/reference/pyspark.sql/core_classes.rst
@@ -42,3 +42,10 @@ Core Classes
UDTFRegistration
udf.UserDefinedFunction
udtf.UserDefinedTableFunction
+ datasource.DataSource
+ datasource.DataSourceReader
+ datasource.DataSourceStreamReader
+ datasource.DataSourceWriter
+ datasource.DataSourceRegistration
+ datasource.InputPartition
+ datasource.WriterCommitMessage
diff --git a/python/docs/source/reference/pyspark.sql/core_classes.rst
b/python/docs/source/reference/pyspark.sql/datasource.rst
similarity index 58%
copy from python/docs/source/reference/pyspark.sql/core_classes.rst
copy to python/docs/source/reference/pyspark.sql/datasource.rst
index 3cf19686cdd8..b92db7a28858 100644
--- a/python/docs/source/reference/pyspark.sql/core_classes.rst
+++ b/python/docs/source/reference/pyspark.sql/datasource.rst
@@ -16,29 +16,29 @@
under the License.
-============
-Core Classes
-============
-.. currentmodule:: pyspark.sql
+==================
+Python Data Source
+==================
+
+.. currentmodule:: pyspark.sql.datasource
.. autosummary::
:toctree: api/
- SparkSession
- Catalog
- DataFrame
- Column
- Observation
- Row
- GroupedData
- PandasCogroupedOps
- DataFrameNaFunctions
- DataFrameStatFunctions
- Window
- DataFrameReader
- DataFrameWriter
- DataFrameWriterV2
- UDFRegistration
- UDTFRegistration
- udf.UserDefinedFunction
- udtf.UserDefinedTableFunction
+ DataSource.name
+ DataSource.reader
+ DataSource.schema
+ DataSource.streamReader
+ DataSource.writer
+ DataSourceReader.partitions
+ DataSourceReader.read
+ DataSourceRegistration.register
+ DataSourceStreamReader.commit
+ DataSourceStreamReader.initialOffset
+ DataSourceStreamReader.latestOffset
+ DataSourceStreamReader.partitions
+ DataSourceStreamReader.read
+ DataSourceStreamReader.stop
+ DataSourceWriter.abort
+ DataSourceWriter.commit
+ DataSourceWriter.write
diff --git a/python/docs/source/reference/pyspark.sql/index.rst
b/python/docs/source/reference/pyspark.sql/index.rst
index 233c8b238a6d..9322a91fba25 100644
--- a/python/docs/source/reference/pyspark.sql/index.rst
+++ b/python/docs/source/reference/pyspark.sql/index.rst
@@ -42,3 +42,4 @@ This page gives an overview of all public Spark SQL API.
udf
udtf
protobuf
+ datasource
diff --git a/python/docs/source/reference/pyspark.sql/spark_session.rst
b/python/docs/source/reference/pyspark.sql/spark_session.rst
index 4be343c52140..ea71249e292e 100644
--- a/python/docs/source/reference/pyspark.sql/spark_session.rst
+++ b/python/docs/source/reference/pyspark.sql/spark_session.rst
@@ -47,6 +47,7 @@ See also :class:`SparkSession`.
SparkSession.catalog
SparkSession.conf
SparkSession.createDataFrame
+ SparkSession.dataSource
SparkSession.getActiveSession
SparkSession.newSession
SparkSession.profile
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
