eugenegujing opened a new pull request, #6346: URL: https://github.com/apache/texera/pull/6346
### What changes were proposed in this PR? Manual backport of #6278 to `release/v1.2` (the automated backport failed on merge conflicts in unrelated files such as `frontend/yarn.lock` and `amber/requirements.txt`, so only the actual changes are cherry-picked here). The Python worker crashed with `TypeError: Unmatched type ...` when a Python operator output a numpy scalar into an INT/LONG or BOOLEAN field, because numpy's integer and bool scalar types are not subclasses of Python `int`/`bool` and failed the strict `isinstance` check in `validate_schema()`. Idiomatic pandas code produces these values naturally: `df["x"].sum()` returns `numpy.int64` and `(df["x"] > n).any()` returns `numpy.bool_`. Changes, identical to #6278: - `cast_to_schema()` in `amber/src/main/python/core/models/tuple.py`: coerce `numpy.integer` scalars to Python `int` for INT/LONG fields, and `numpy.bool_` to Python `bool` for BOOL fields in a separate branch gated on the target type, so `bool` and `int` never cross-coerce. Out-of-range values are left unchanged so validation still fails loudly. - `amber/src/main/python/core/models/schema/attribute_type.py`: add `NUMPY_INTEGRAL_RANGES` (INT → int32, LONG → full int64 range). numpy integers are exact, so they are bounded only by the target Arrow width; integral floats keep the existing `INTEGRAL_TYPE_RANGES` cap at the float64 exact-integer window (2**53). - `amber/src/test/python/core/models/test_tuple.py`: add the same unit tests as #6278 (coercion cases asserting the concrete Python type, range and bool↔int guards, int64/uint64 boundary cases, and a pandas-reduction integration case). `validate_schema()` is unchanged. ### Any related issues, documentation, discussions? - Backport of #6278 (merged to `main`) to `release/v1.2`; same fix, no functional differences. - Follow-up to #6053, whose `INTEGRAL_TYPE_RANGES` mechanism is already present on `release/v1.2`. ### How was this PR tested? Same tests as #6278, applied on top of `release/v1.2`: `pytest src/test/python/core/models/test_tuple.py -q` passes with 87 passed (the 2-test difference from `main` comes from #5599's `as_dict` tests, which are not on `release/v1.2` and are unrelated to this fix), and `ruff check` / `ruff format --check` are clean on the three changed files. The backported `cast_to_schema()` and the numpy tests are identical to the merged #6278 version. ### Was this PR authored or co-authored using generative AI tooling? Co-authored by: Claude Code (Claude Fable 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
