This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3e2bb1288387 [SPARK-58514][PS] Restore native NumPy log2 mapping
3e2bb1288387 is described below

commit 3e2bb128838751fc39beaaca0cb58926720c21e9
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Aug 4 11:14:42 2026 +0800

    [SPARK-58514][PS] Restore native NumPy log2 mapping
    
    ### What changes were proposed in this pull request?
    
    Restore the pandas-on-Spark `np.log2` mapping using the native Spark SQL 
`log2` function. Preserve NumPy behavior for `0.0` and `-0.0`, for which NumPy 
returns `-inf` while Spark SQL returns null. Add end-to-end coverage for 
negative, zero, finite, infinite, and NaN inputs.
    
    ### Why are the changes needed?
    
    SPARK-58249 unintentionally removed the existing `np.log2` mapping while 
replacing other NumPy mappings with native Spark SQL expressions. Spark already 
provides an equivalent native function, with an explicit zero branch needed to 
match NumPy semantics.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `np.log2` on pandas-on-Spark objects is dispatched to Spark SQL again 
instead of being unsupported by the NumPy compatibility mapping.
    
    ### How was this patch tested?
    
    Added end-to-end pandas-on-Spark coverage for `np.log2` edge cases.
    
    Ran:
    
    - `python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat`
    - `python/run-tests --testnames 
pyspark.pandas.tests.connect.test_parity_numpy_compat`
    - Ruff check and format check
    - `git diff --check`, source ASCII, and line-length checks
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-5)
    
    Closes #57717 from zhengruifeng/pandas-native-log2-dev2.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Ruifeng Zheng <[email protected]>
---
 python/pyspark/pandas/numpy_compat.py            | 1 +
 python/pyspark/pandas/tests/test_numpy_compat.py | 1 +
 2 files changed, 2 insertions(+)

diff --git a/python/pyspark/pandas/numpy_compat.py 
b/python/pyspark/pandas/numpy_compat.py
index 2a39fd65d933..6bdb5e97841f 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -60,6 +60,7 @@ unary_np_spark_mappings = {
     "log": F.log,
     "log10": F.log10,
     "log1p": F.log1p,
+    "log2": lambda c: F.when(c == 0, 
F.lit(float("-inf"))).otherwise(F.log2(c)),
     "logical_not": lambda c: ~(c.cast(BooleanType())),
     "matmul": lambda _: NotImplemented,  # Can return a NumPy array in pandas.
     "negative": F.negative,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py 
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 719e6e0114b2..1f6ccd504cc4 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -101,6 +101,7 @@ class NumPyCompatTestsMixin:
             (np.invert, [np.iinfo(np.int64).min, -2, -1, 0, 1, 2, 
np.iinfo(np.int64).max]),
             (np.isfinite, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
             (np.isinf, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
+            (np.log2, [-np.inf, -64.0, -1.0, -0.0, 0.0, 1.0, 2.0, 64.0, 
np.inf, np.nan]),
             (np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, 
np.nan]),
             (np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf, 
np.nan]),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to