This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 12d984c15f3 [SPARK-40376][PYTHON] Avoid Numpy deprecation warning
12d984c15f3 is described below
commit 12d984c15f30c2f31b62c403738b40d8872b7c13
Author: ELHoussineT <[email protected]>
AuthorDate: Mon Sep 12 20:46:15 2022 -0500
[SPARK-40376][PYTHON] Avoid Numpy deprecation warning
### What changes were proposed in this pull request?
Use `bool` instead of `np.bool` as `np.bool` will be deprecated (see:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations)
Using `np.bool` generates this warning:
```
UserWarning: toPandas attempted Arrow optimization because
'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the
error below and can not continue. Note that
'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on
failures in the middle of computation.
3070E `np.bool` is a deprecated alias for the builtin
`bool`. To silence this warning, use `bool` by itself. Doing this will not
modify any behavior and is safe. If you specifically wanted the numpy scalar
type, use `np.bool_` here.
3071E Deprecated in NumPy 1.20; for more details and
guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
```
### Why are the changes needed?
Deprecation soon:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations.
### Does this PR introduce _any_ user-facing change?
The warning will be suppressed
### How was this patch tested?
Existing tests should suffice.
Closes #37817 from ELHoussineT/patch-1.
Authored-by: ELHoussineT <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
python/pyspark/sql/pandas/conversion.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/sql/pandas/conversion.py
b/python/pyspark/sql/pandas/conversion.py
index 119a9bf315c..d7f1ee5b8da 100644
--- a/python/pyspark/sql/pandas/conversion.py
+++ b/python/pyspark/sql/pandas/conversion.py
@@ -295,7 +295,7 @@ class PandasConversionMixin:
elif type(dt) == DoubleType:
return np.float64
elif type(dt) == BooleanType:
- return np.bool # type: ignore[attr-defined]
+ return bool
elif type(dt) == TimestampType:
return np.datetime64
elif type(dt) == TimestampNTZType:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
