Spenserrrr opened a new pull request, #58485:
URL: https://github.com/apache/spark/pull/58485
### What changes were proposed in this pull request?
`numpy_compat.py` maps each NumPy ufunc to a Spark expression, and that
expression accepts any operand Spark can cast. This PR adds a table of the
operand types NumPy itself accepts and consults it in
`maybe_dispatch_ufunc_to_spark_func`, raising `TypeError` before any `Column`
is built.
The new pieces, all in `numpy_compat.py`:
- `_np_spark_accepted_types`, a dict from ufunc name to **one tuple of Spark
types per operand**. Per-operand rather than per-ufunc because the sets are not
always symmetric: `np.ldexp` takes its exponent from an integer loop only
(`ufunc.types` is `['ei->e', 'fi->f', 'di->d', ...]`), so `np.ldexp(float_col,
2.5)` must raise while the first operand stays numeric.
- Five shared type sets built from Spark's abstract bases (`IntegralType`,
`NumericType`) rather than enumerated widths. Boolean is named explicitly
because Spark does not classify `BooleanType` as a `NumericType`, while NumPy
accepts a boolean wherever it accepts an integer.
- `_check_operand_types`, which reads `inp.spark.data_type` for a `Series`
or `Index` and `as_spark_type(type(inp))` for a scalar, so `np.fmod(psser,
"8")` is caught as well. An operand whose type cannot be determined is left
unchecked.
The check is deliberately placed in the dispatch rather than inside each
mapping function. A mapping function receives `Column`s, whose type is visible
only to the plan, so it could only emit a `RaiseError` node that fires per row
at collect time; the dispatch still holds the `Series`, so a Python `TypeError`
is raised at the ufunc call, as pandas does.
The two commits split the change by risk, and are worth reviewing separately:
1. **The first covers only the 27 mappings converted from a `pandas_udf`.**
Those UDFs called NumPy per value and so enforced its type rules for free;
converting them to native expressions dropped that. Every one of those
conversions is unreleased (`git tag --contains` finds no final tag for any of
them), so this commit restores prior behaviour and changes nothing users have
seen.
2. **The second extends the table to all 58 mappings whose accepted types
can be listed per operand**, including those that were already native
expressions. This is a user-facing change on released versions, and it is here
because gating only the converted half leaves two spellings of the same
operation disagreeing: `np.rad2deg` on a string column raised while
`np.degrees` returned `401.07`. Happy to drop this commit if you would rather
ship the regression fix alone.
A mapping stays out of the table when its accepted operand *pairs* are not a
rectangle, so no per-operand set describes them: `np.fmax`, `np.fmin`,
`np.maximum` and `np.minimum` accept `(int, int)`, `(str, str)` and
`(timestamp, timestamp)` but no mixed pair. They still compute on an all-null
column where pandas raises. Also out: mappings that are still a `pandas_udf`
(NumPy runs per value there), `np.floor_divide` (the `floordiv` dunder wins
before this registry), and alias names, since the lookup uses `ufunc.__name__`
and `np.abs` arrives as `absolute`.
### Why are the changes needed?
A ufunc silently computes a plausible number for operands NumPy rejects:
`np.fmod` on a string column returns `1.0` and `np.ldexp(float_col, 2.5)`
returns `42.4` where pandas raises `TypeError`.
### Does this PR introduce _any_ user-facing change?
Yes. A ufunc called on an operand type NumPy does not accept now raises
`TypeError` instead of returning a value:
```python
>>> psdf = ps.DataFrame({"a": ["7", "8"]})
>>> np.fmod(psdf.a, psdf.a)
TypeError: ufunc 'fmod' is not supported for the input types (string,
string).
```
For the 27 mappings in the first commit this restores the behaviour of the
released `pandas_udf` implementations. For the rest it is a change against
released versions, matching pandas in every case.
### How was this patch tested?
Four tests in `NumPyCompatTestsMixin`: one loops the table so every entry is
covered as it grows, plus rows for the types only some ufuncs reject, for
scalar operands, and for the accepted types that must still compute. All fail
on a pristine `numpy_compat.py`. Each entry's accepted set was checked against
pandas and against `ufunc.types` for all 58 ufuncs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
