This is an automated email from the ASF dual-hosted git repository.
zero323 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d424eca [SPARK-37407][FOLLOW-UP][PYTHON][ML] Add not None assertions
for active context and its _jvm
d424eca is described below
commit d424eca13d442170a946ec447c6dff9513ba3acf
Author: zero323 <[email protected]>
AuthorDate: Sat Dec 25 23:14:21 2021 +0100
[SPARK-37407][FOLLOW-UP][PYTHON][ML] Add not None assertions for active
context and its _jvm
### What changes were proposed in this pull request?
This PR add `is not None` assertions for `Optional`
`_active_spark_context` and `_jvm`
### Why are the changes needed?
To address failures in CI:
```
python/pyspark/ml/functions.py:69: error: Item "None" of
"Optional[SparkContext]" has no attribute "_jvm" [union-attr]
python/pyspark/ml/functions.py:69: error: Item "None" of "Optional[Any]"
has no attribute "org" [union-attr]
python/pyspark/ml/functions.py:104: error: Item "None" of
"Optional[SparkContext]" has no attribute "_jvm" [union-attr]
python/pyspark/ml/functions.py:104: error: Item "None" of "Optional[Any]"
has no attribute "org" [union-attr]
```
caused by divergence between #34678 and changes introduced by #34892
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`dev/lint-python`
Closes #35022 from zero323/SPARK-37407-FOLLOW-UP.
Authored-by: zero323 <[email protected]>
Signed-off-by: zero323 <[email protected]>
---
python/pyspark/ml/functions.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/python/pyspark/ml/functions.py b/python/pyspark/ml/functions.py
index 08ebf40..aecafb3 100644
--- a/python/pyspark/ml/functions.py
+++ b/python/pyspark/ml/functions.py
@@ -65,6 +65,7 @@ def vector_to_array(col: Column, dtype: str = "float64") ->
Column:
StructField(oldVec,ArrayType(FloatType,false),false)]
"""
sc = SparkContext._active_spark_context
+ assert sc is not None and sc._jvm is not None
return Column(
sc._jvm.org.apache.spark.ml.functions.vector_to_array(_to_java_column(col),
dtype)
)
@@ -101,6 +102,7 @@ def array_to_vector(col: Column) -> Column:
[Row(vec1=DenseVector([1.0, 3.0]))]
"""
sc = SparkContext._active_spark_context
+ assert sc is not None and sc._jvm is not None
return
Column(sc._jvm.org.apache.spark.ml.functions.array_to_vector(_to_java_column(col)))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
