This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4e208ff1192c [SPARK-58514][PS] Restore native NumPy log2 mapping
4e208ff1192c is described below
commit 4e208ff1192cf93ac6a73cbcabd532810be355c2
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Aug 4 11:14:42 2026 +0800
[SPARK-58514][PS] Restore native NumPy log2 mapping
### What changes were proposed in this pull request?
Restore the pandas-on-Spark `np.log2` mapping using the native Spark SQL
`log2` function. Preserve NumPy behavior for `0.0` and `-0.0`, for which NumPy
returns `-inf` while Spark SQL returns null. Add end-to-end coverage for
negative, zero, finite, infinite, and NaN inputs.
### Why are the changes needed?
SPARK-58249 unintentionally removed the existing `np.log2` mapping while
replacing other NumPy mappings with native Spark SQL expressions. Spark already
provides an equivalent native function, with an explicit zero branch needed to
match NumPy semantics.
### Does this PR introduce _any_ user-facing change?
Yes. `np.log2` on pandas-on-Spark objects is dispatched to Spark SQL again
instead of being unsupported by the NumPy compatibility mapping.
### How was this patch tested?
Added end-to-end pandas-on-Spark coverage for `np.log2` edge cases.
Ran:
- `python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat`
- `python/run-tests --testnames
pyspark.pandas.tests.connect.test_parity_numpy_compat`
- Ruff check and format check
- `git diff --check`, source ASCII, and line-length checks
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57717 from zhengruifeng/pandas-native-log2-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 3e2bb128838751fc39beaaca0cb58926720c21e9)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 1 +
python/pyspark/pandas/tests/test_numpy_compat.py | 1 +
2 files changed, 2 insertions(+)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 2a39fd65d933..6bdb5e97841f 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -60,6 +60,7 @@ unary_np_spark_mappings = {
"log": F.log,
"log10": F.log10,
"log1p": F.log1p,
+ "log2": lambda c: F.when(c == 0,
F.lit(float("-inf"))).otherwise(F.log2(c)),
"logical_not": lambda c: ~(c.cast(BooleanType())),
"matmul": lambda _: NotImplemented, # Can return a NumPy array in pandas.
"negative": F.negative,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 719e6e0114b2..1f6ccd504cc4 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -101,6 +101,7 @@ class NumPyCompatTestsMixin:
(np.invert, [np.iinfo(np.int64).min, -2, -1, 0, 1, 2,
np.iinfo(np.int64).max]),
(np.isfinite, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
(np.isinf, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
+ (np.log2, [-np.inf, -64.0, -1.0, -0.0, 0.0, 1.0, 2.0, 64.0,
np.inf, np.nan]),
(np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf,
np.nan]),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]