eugenegujing opened a new pull request, #6278:
URL: https://github.com/apache/texera/pull/6278
### What changes were proposed in this PR?
Pandas-based Python operators naturally produce **numpy scalar types** at
the `Tuple` output boundary, and the Python worker rejects them:
- `df["x"].sum()` / `.max()` / `.count()` return `numpy.int64`
- `df["x"].any()` or any numpy comparison returns `numpy.bool_`
On output, `Tuple.finalize(schema)` runs `cast_to_schema()` then a strict
`isinstance(value, expected_python_type)` check in `validate_schema()`. Because
numpy's integer and bool scalar types are **not** subclasses of Python `int` /
`bool` (`isinstance(np.int64(5), int)` and `isinstance(np.bool_(True), bool)`
are both `False`), the check fails and the operator crashes with `TypeError:
Unmatched type for field ...`, even though the value is semantically correct
(`np.int64(5)` is exactly the integer 5; the field is declared INTEGER).
This is a direct follow-up to #6053, which added `float → int` coercion for
INT/LONG fields. That fix only handled `np.float64` (which happens to subclass
Python `float`); numpy integer and bool scalars were still rejected. This PR
extends the same mechanism in `cast_to_schema()` to cover them:
- **INT/LONG fields:** accept `numpy.integer` scalars by normalizing them
(and integral floats) into a candidate `int`, then applying the existing
`INTEGRAL_TYPE_RANGES` bounds check once. `int()` is exact for numpy integers,
so there is no precision loss. Out-of-range values are left unchanged so
`validate_schema()` still fails loudly (no int32 overflow / silent corruption).
- **BOOL fields:** convert `numpy.bool_` to Python `bool` in a separate
branch gated on the BOOL target type, so `bool` and `int` never cross-coerce
(`np.bool_` is not `numpy.integer`, and a plain Python `bool` — an `int`
subclass — is never coerced into an integer field).
`validate_schema()` is unchanged; it remains the strict backstop, so any
type this PR does not explicitly coerce still fails loudly rather than passing
silently.
**Out of scope:** `np.float32` is not a Python `float` subclass, so an
integral `np.float32` into an INT field is still rejected. It is intentionally
left out of this PR to keep the change focused; it can be folded into the same
float-normalization path in a follow-up if desired.
### Any related issues, documentation, discussions?
- Fixes #6277
- Follow-up to #6053 (integral `float → int` coercion for INT/LONG fields).
### How was this PR tested?
Added unit tests to `amber/src/test/python/core/models/test_tuple.py`.
Results:
```
# Python unit tests
pytest src/test/python/core/models/test_tuple.py -q -> 84 passed
ruff check / ruff format --check -> clean
# Backend suite
sbt scalafmtCheckAll -> exit 0
sbt "scalafixAll --check" -> exit 0
sbt test -> 1717 passed, 0
failed
```
Adversarial check: reverting `cast_to_schema()` to the pre-fix version turns
exactly the coercion cases red while the guard cases stay green; restoring the
fix returns to all-green. Every pass-case asserts the concrete Python type, so
a test cannot pass without the fix.
Also reproduced end-to-end in the platform: `CSV File Scan` → `Python UDF`
(`yield {"total_age": table["age"].sum()}` / `{"has_senior": (table["age"] >
60).any()}`) crashed with `Unmatched type ... numpy.int64` / `numpy.bool`
before the fix and runs correctly after.
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
