This is an automated email from the ASF dual-hosted git repository.
zero323 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 88cf7e1 [SPARK-37407][PYTHON] Inline type hints for
python/pyspark/ml/functions.py
88cf7e1 is described below
commit 88cf7e17cbd4163272300a9b375553fe0c42f2a7
Author: dch nguyen <[email protected]>
AuthorDate: Sat Dec 25 13:10:45 2021 +0100
[SPARK-37407][PYTHON] Inline type hints for python/pyspark/ml/functions.py
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/ml/functions.py
### Why are the changes needed?
We can take advantage of static type checking within the functions by
inlining the type hints.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
Closes #34678 from dchvn/SPARK-37407.
Authored-by: dch nguyen <[email protected]>
Signed-off-by: zero323 <[email protected]>
---
python/pyspark/ml/functions.py | 6 +++---
python/pyspark/ml/functions.pyi | 23 -----------------------
2 files changed, 3 insertions(+), 26 deletions(-)
diff --git a/python/pyspark/ml/functions.py b/python/pyspark/ml/functions.py
index 64b5948..08ebf40 100644
--- a/python/pyspark/ml/functions.py
+++ b/python/pyspark/ml/functions.py
@@ -19,7 +19,7 @@ from pyspark import SparkContext
from pyspark.sql.column import Column, _to_java_column
-def vector_to_array(col, dtype="float64"):
+def vector_to_array(col: Column, dtype: str = "float64") -> Column:
"""
Converts a column of MLlib sparse/dense vectors into a column of dense
arrays.
@@ -70,7 +70,7 @@ def vector_to_array(col, dtype="float64"):
)
-def array_to_vector(col):
+def array_to_vector(col: Column) -> Column:
"""
Converts a column of array of numeric type into a column of
pyspark.ml.linalg.DenseVector
instances
@@ -104,7 +104,7 @@ def array_to_vector(col):
return
Column(sc._jvm.org.apache.spark.ml.functions.array_to_vector(_to_java_column(col)))
-def _test():
+def _test() -> None:
import doctest
from pyspark.sql import SparkSession
import pyspark.ml.functions
diff --git a/python/pyspark/ml/functions.pyi b/python/pyspark/ml/functions.pyi
deleted file mode 100644
index cb08398..0000000
--- a/python/pyspark/ml/functions.pyi
+++ /dev/null
@@ -1,23 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from pyspark import SparkContext as SparkContext, since as since # noqa: F401
-from pyspark.sql.column import Column as Column
-
-def vector_to_array(col: Column) -> Column: ...
-def array_to_vector(col: Column) -> Column: ...
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
