This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 64e2b22f6b40 [SPARK-45396][PYTHON] Add doc entry for
`pyspark.ml.connect` module, and adds `Evaluator` to `__all__` at `ml.connect`
64e2b22f6b40 is described below
commit 64e2b22f6b4023197871a60eb08b055688e9fdd2
Author: Weichen Xu <[email protected]>
AuthorDate: Thu Oct 5 08:38:54 2023 +0900
[SPARK-45396][PYTHON] Add doc entry for `pyspark.ml.connect` module, and
adds `Evaluator` to `__all__` at `ml.connect`
This PR documents MLlib's Spark Connect support at API reference.
This PR also piggies back a fix in `__all__` at
`python/pyspark/ml/connect/__init__.py` so `from pyspark.sql.commect import
Evaluator` works.
With this this, user cannot see `pyspark.ml.connect` Python APIs on doc
website.
Yes it adds the new page into your facing documentation ([PySpark API
reference](https://spark.apache.org/docs/latest/api/python/reference/index.html)).
Manually tested via:
```bash
cd python/docs
make clean html
```
No.
Closes #43210 from HyukjinKwon/SPARK-45396-followup.
Lead-authored-by: Weichen Xu <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 35b627a934b1ab28be7d6ba88fdad63dc129525a)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/docs/source/reference/index.rst | 1 +
.../docs/source/reference/pyspark.ml.connect.rst | 122 +++++++++++++++++++++
python/pyspark/ml/connect/__init__.py | 3 +-
3 files changed, 125 insertions(+), 1 deletion(-)
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/index.rst
index ed3eb4d07dac..6330636839cd 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/index.rst
@@ -31,6 +31,7 @@ Pandas API on Spark follows the API specifications of latest
pandas release.
pyspark.pandas/index
pyspark.ss/index
pyspark.ml
+ pyspark.ml.connect
pyspark.streaming
pyspark.mllib
pyspark
diff --git a/python/docs/source/reference/pyspark.ml.connect.rst
b/python/docs/source/reference/pyspark.ml.connect.rst
new file mode 100644
index 000000000000..1a3e6a593980
--- /dev/null
+++ b/python/docs/source/reference/pyspark.ml.connect.rst
@@ -0,0 +1,122 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+MLlib (DataFrame-based) for Spark Connect
+=========================================
+
+.. warning::
+ The namespace for this package can change in the future Spark version.
+
+
+Pipeline APIs
+-------------
+
+.. currentmodule:: pyspark.ml.connect
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ Transformer
+ Estimator
+ Model
+ Evaluator
+ Pipeline
+ PipelineModel
+
+
+Feature
+-------
+
+.. currentmodule:: pyspark.ml.connect.feature
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ MaxAbsScaler
+ MaxAbsScalerModel
+ StandardScaler
+ StandardScalerModel
+
+
+Classification
+--------------
+
+.. currentmodule:: pyspark.ml.connect.classification
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ LogisticRegression
+ LogisticRegressionModel
+
+
+Functions
+---------
+
+.. currentmodule:: pyspark.ml.connect.functions
+
+.. autosummary::
+ :toctree: api/
+
+ array_to_vector
+ vector_to_array
+
+
+Tuning
+------
+
+.. currentmodule:: pyspark.ml.connect.tuning
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ CrossValidator
+ CrossValidatorModel
+
+
+Evaluation
+----------
+
+.. currentmodule:: pyspark.ml.connect.evaluation
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ RegressionEvaluator
+ BinaryClassificationEvaluator
+ MulticlassClassificationEvaluator
+
+
+Utilities
+---------
+
+.. currentmodule:: pyspark.ml.connect.io_utils
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ ParamsReadWrite
+ CoreModelReadWrite
+ MetaAlgorithmReadWrite
+
diff --git a/python/pyspark/ml/connect/__init__.py
b/python/pyspark/ml/connect/__init__.py
index 2ee152f6a38a..e6115a62ccfe 100644
--- a/python/pyspark/ml/connect/__init__.py
+++ b/python/pyspark/ml/connect/__init__.py
@@ -28,13 +28,14 @@ from pyspark.ml.connect import (
evaluation,
tuning,
)
+from pyspark.ml.connect.evaluation import Evaluator
from pyspark.ml.connect.pipeline import Pipeline, PipelineModel
__all__ = [
"Estimator",
"Transformer",
- "Estimator",
+ "Evaluator",
"Model",
"feature",
"evaluation",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
