This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push: new 64e2b22f6b40 [SPARK-45396][PYTHON] Add doc entry for `pyspark.ml.connect` module, and adds `Evaluator` to `__all__` at `ml.connect` 64e2b22f6b40 is described below commit 64e2b22f6b4023197871a60eb08b055688e9fdd2 Author: Weichen Xu <weichen...@databricks.com> AuthorDate: Thu Oct 5 08:38:54 2023 +0900 [SPARK-45396][PYTHON] Add doc entry for `pyspark.ml.connect` module, and adds `Evaluator` to `__all__` at `ml.connect` This PR documents MLlib's Spark Connect support at API reference. This PR also piggies back a fix in `__all__` at `python/pyspark/ml/connect/__init__.py` so `from pyspark.sql.commect import Evaluator` works. With this this, user cannot see `pyspark.ml.connect` Python APIs on doc website. Yes it adds the new page into your facing documentation ([PySpark API reference](https://spark.apache.org/docs/latest/api/python/reference/index.html)). Manually tested via: ```bash cd python/docs make clean html ``` No. Closes #43210 from HyukjinKwon/SPARK-45396-followup. Lead-authored-by: Weichen Xu <weichen...@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> (cherry picked from commit 35b627a934b1ab28be7d6ba88fdad63dc129525a) Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/docs/source/reference/index.rst | 1 + .../docs/source/reference/pyspark.ml.connect.rst | 122 +++++++++++++++++++++ python/pyspark/ml/connect/__init__.py | 3 +- 3 files changed, 125 insertions(+), 1 deletion(-) diff --git a/python/docs/source/reference/index.rst b/python/docs/source/reference/index.rst index ed3eb4d07dac..6330636839cd 100644 --- a/python/docs/source/reference/index.rst +++ b/python/docs/source/reference/index.rst @@ -31,6 +31,7 @@ Pandas API on Spark follows the API specifications of latest pandas release. pyspark.pandas/index pyspark.ss/index pyspark.ml + pyspark.ml.connect pyspark.streaming pyspark.mllib pyspark diff --git a/python/docs/source/reference/pyspark.ml.connect.rst b/python/docs/source/reference/pyspark.ml.connect.rst new file mode 100644 index 000000000000..1a3e6a593980 --- /dev/null +++ b/python/docs/source/reference/pyspark.ml.connect.rst @@ -0,0 +1,122 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +MLlib (DataFrame-based) for Spark Connect +========================================= + +.. warning:: + The namespace for this package can change in the future Spark version. + + +Pipeline APIs +------------- + +.. currentmodule:: pyspark.ml.connect + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + Transformer + Estimator + Model + Evaluator + Pipeline + PipelineModel + + +Feature +------- + +.. currentmodule:: pyspark.ml.connect.feature + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + MaxAbsScaler + MaxAbsScalerModel + StandardScaler + StandardScalerModel + + +Classification +-------------- + +.. currentmodule:: pyspark.ml.connect.classification + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + LogisticRegression + LogisticRegressionModel + + +Functions +--------- + +.. currentmodule:: pyspark.ml.connect.functions + +.. autosummary:: + :toctree: api/ + + array_to_vector + vector_to_array + + +Tuning +------ + +.. currentmodule:: pyspark.ml.connect.tuning + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + CrossValidator + CrossValidatorModel + + +Evaluation +---------- + +.. currentmodule:: pyspark.ml.connect.evaluation + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + RegressionEvaluator + BinaryClassificationEvaluator + MulticlassClassificationEvaluator + + +Utilities +--------- + +.. currentmodule:: pyspark.ml.connect.io_utils + +.. autosummary:: + :template: autosummary/class_with_docs.rst + :toctree: api/ + + ParamsReadWrite + CoreModelReadWrite + MetaAlgorithmReadWrite + diff --git a/python/pyspark/ml/connect/__init__.py b/python/pyspark/ml/connect/__init__.py index 2ee152f6a38a..e6115a62ccfe 100644 --- a/python/pyspark/ml/connect/__init__.py +++ b/python/pyspark/ml/connect/__init__.py @@ -28,13 +28,14 @@ from pyspark.ml.connect import ( evaluation, tuning, ) +from pyspark.ml.connect.evaluation import Evaluator from pyspark.ml.connect.pipeline import Pipeline, PipelineModel __all__ = [ "Estimator", "Transformer", - "Estimator", + "Evaluator", "Model", "feature", "evaluation", --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org