This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new e793be502c fix(pyamber): coerce numpy scalar types (np.integer /
np.bool_) in cast_to_schema (#6278)
e793be502c is described below
commit e793be502c050a9be05bc55d67bdad8eaf0ae418
Author: Eugene Gu <[email protected]>
AuthorDate: Tue Jul 14 13:13:44 2026 -0700
fix(pyamber): coerce numpy scalar types (np.integer / np.bool_) in
cast_to_schema (#6278)
### What changes were proposed in this PR?
Pandas-based Python operators naturally produce **numpy scalar types**
at the `Tuple` output boundary, and the Python worker rejects them:
- `df["x"].sum()` / `.max()` / `.count()` return `numpy.int64`
- `df["x"].any()` or any numpy comparison returns `numpy.bool_`
On output, `Tuple.finalize(schema)` runs `cast_to_schema()` then a
strict `isinstance(value, expected_python_type)` check in
`validate_schema()`. Because numpy's integer and bool scalar types are
**not** subclasses of Python `int` / `bool` (`isinstance(np.int64(5),
int)` and `isinstance(np.bool_(True), bool)` are both `False`), the
check fails and the operator crashes with `TypeError: Unmatched type for
field ...`, even though the value is semantically correct (`np.int64(5)`
is exactly the integer 5; the field is declared INTEGER).
This is a direct follow-up to #6053, which added `float → int` coercion
for INT/LONG fields. That fix only handled `np.float64` (which happens
to subclass Python `float`); numpy integer and bool scalars were still
rejected. This PR extends the same mechanism in `cast_to_schema()` to
cover them:
- **INT/LONG fields:** accept `numpy.integer` scalars by normalizing
them (and integral floats) into a candidate `int`, then applying the
existing `INTEGRAL_TYPE_RANGES` bounds check once. `int()` is exact for
numpy integers, so there is no precision loss. Out-of-range values are
left unchanged so `validate_schema()` still fails loudly (no int32
overflow / silent corruption).
- **BOOL fields:** convert `numpy.bool_` to Python `bool` in a separate
branch gated on the BOOL target type, so `bool` and `int` never
cross-coerce (`np.bool_` is not `numpy.integer`, and a plain Python
`bool` — an `int` subclass — is never coerced into an integer field).
`validate_schema()` is unchanged; it remains the strict backstop, so any
type this PR does not explicitly coerce still fails loudly rather than
passing silently.
**Out of scope:** `np.float32` is not a Python `float` subclass, so an
integral `np.float32` into an INT field is still rejected. It is
intentionally left out of this PR to keep the change focused; it can be
folded into the same float-normalization path in a follow-up if desired.
### Any related issues, documentation, discussions?
- Fixes #6277
- Follow-up to #6053 (integral `float → int` coercion for INT/LONG
fields).
### How was this PR tested?
Added unit tests to `amber/src/test/python/core/models/test_tuple.py`.
Results:
```
# Python unit tests
pytest src/test/python/core/models/test_tuple.py -q -> 84 passed
ruff check / ruff format --check -> clean
# Backend suite
sbt scalafmtCheckAll -> exit 0
sbt "scalafixAll --check" -> exit 0
sbt test -> 1717 passed, 0
failed
```
Adversarial check: reverting `cast_to_schema()` to the pre-fix version
turns exactly the coercion cases red while the guard cases stay green;
restoring the fix returns to all-green. Every pass-case asserts the
concrete Python type, so a test cannot pass without the fix.
Also reproduced end-to-end in the platform: `CSV File Scan` → `Python
UDF` (`yield {"total_age": table["age"].sum()}` / `{"has_senior":
(table["age"] > 60).any()}`) crashed with `Unmatched type ...
numpy.int64` / `numpy.bool` before the fix and runs correctly after.
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)
---------
Signed-off-by: Eugene Gu <[email protected]>
Co-authored-by: Copilot Autofix powered by AI
<[email protected]>
---
.../python/core/models/schema/attribute_type.py | 9 ++
amber/src/main/python/core/models/tuple.py | 46 ++++--
amber/src/test/python/core/models/test_tuple.py | 170 +++++++++++++++++++++
3 files changed, 210 insertions(+), 15 deletions(-)
diff --git a/amber/src/main/python/core/models/schema/attribute_type.py
b/amber/src/main/python/core/models/schema/attribute_type.py
index 666ee69ede..0511937336 100644
--- a/amber/src/main/python/core/models/schema/attribute_type.py
+++ b/amber/src/main/python/core/models/schema/attribute_type.py
@@ -110,3 +110,12 @@ INTEGRAL_TYPE_RANGES = {
AttributeType.INT: (-(2**31), 2**31 - 1),
AttributeType.LONG: (-(2**53) + 1, 2**53 - 1),
}
+
+# numpy integer scalars are exact (unlike integral floats, which lose
+# integer precision above 2**53), so they are bounded only by the target
+# Arrow integer width, not the float64 exact-integer window used by
+# INTEGRAL_TYPE_RANGES: INT -> int32, LONG -> int64.
+NUMPY_INTEGRAL_RANGES = {
+ AttributeType.INT: (-(2**31), 2**31 - 1),
+ AttributeType.LONG: (-(2**63), 2**63 - 1),
+}
diff --git a/amber/src/main/python/core/models/tuple.py
b/amber/src/main/python/core/models/tuple.py
index 4b5a2e0ee9..1f319b6a4f 100644
--- a/amber/src/main/python/core/models/tuple.py
+++ b/amber/src/main/python/core/models/tuple.py
@@ -16,6 +16,7 @@
# under the License.
import ctypes
+import numpy
import pandas
import pickle
import pyarrow
@@ -32,6 +33,7 @@ from typing_extensions import Protocol, runtime_checkable
from core.models.type.large_binary import largebinary
from .schema.attribute_type import (
INTEGRAL_TYPE_RANGES,
+ NUMPY_INTEGRAL_RANGES,
TO_PYOBJECT_MAPPING,
AttributeType,
)
@@ -307,10 +309,12 @@ class Tuple:
"""
Safely cast each field value to match the target schema.
If failed, the value will stay not changed.
- This current conducts three kinds of casts:
+ This method performs the following casts:
1. cast NaN to None;
- 2. cast integral floats to int for INT/LONG fields;
- 3. cast any object to bytes (using pickle).
+ 2. cast integral floats and numpy integer scalars to int for
+ INT/LONG fields;
+ 3. cast numpy bool scalars to bool for BOOL fields;
+ 4. cast any object to bytes (using pickle).
:param schema: The target Schema that describes the target
AttributeType to
cast.
:return:
@@ -324,30 +328,42 @@ class Tuple:
self[field_name] = None
elif field_value is not None:
field_type = schema.get_attr_type(field_name)
- if (
- field_type in INTEGRAL_TYPE_RANGES
- and isinstance(field_value, float)
- and field_value.is_integer()
+ if field_type in INTEGRAL_TYPE_RANGES and (
+ isinstance(field_value, numpy.integer)
+ or (isinstance(field_value, float) and
field_value.is_integer())
):
- # pandas 2.2.3 promotes an int column holding nulls to
- # float64 (119 -> 119.0), so convert integral floats
- # destined for INT/LONG back to int — but only within
- # the safe range above; out-of-range floats are left
- # unchanged so validation still fails. Compare on the
- # int result to avoid float rounding at the endpoints.
- min_value, max_value = INTEGRAL_TYPE_RANGES[field_type]
+ # Coerce numpy integer scalars and integral floats into
+ # int for INT/LONG (pandas 2.2.3 promotes null-holding
+ # int columns to float64, and reductions like .sum()
+ # return numpy ints). Bounds differ by source: numpy
+ # integers are exact, so only the target width applies
+ # (LONG -> int64); integral floats stay within the
+ # float64 exact-integer window (2**53). Out-of-range
+ # values are left unchanged so validation still fails.
+ if isinstance(field_value, numpy.integer):
+ min_value, max_value =
NUMPY_INTEGRAL_RANGES[field_type]
+ else:
+ min_value, max_value =
INTEGRAL_TYPE_RANGES[field_type]
int_value = int(field_value)
if min_value <= int_value <= max_value:
self[field_name] = int_value
else:
logger.warning(
- f"Field '{field_name}': integral float "
+ f"Field '{field_name}': integral value "
f"{field_value} is outside the safely
coercible "
f"range of {field_type}; leaving it unchanged "
f"(schema validation will fail). Consider "
f"casting the column to STRING or DOUBLE (or "
f"LONG for large integers in an INT field)."
)
+ elif field_type == AttributeType.BOOL and isinstance(
+ field_value, numpy.bool_
+ ):
+ # pandas reductions such as .any()/.all() and numpy
+ # comparisons return numpy.bool_, which is not a Python
+ # bool; convert it for BOOL fields. Gated on the BOOL
+ # target type so a numpy integer never lands here.
+ self[field_name] = bool(field_value)
elif field_type == AttributeType.BINARY and not isinstance(
field_value, bytes
):
diff --git a/amber/src/test/python/core/models/test_tuple.py
b/amber/src/test/python/core/models/test_tuple.py
index 3d61fb10f5..b9d4219913 100644
--- a/amber/src/test/python/core/models/test_tuple.py
+++ b/amber/src/test/python/core/models/test_tuple.py
@@ -305,6 +305,176 @@ class TestTuple:
tuple_.finalize(Schema(raw_schema={"payload": "BINARY"}))
assert tuple_["payload"] is None
+ # Pandas-based operators also produce numpy scalar types very naturally:
+ # reductions such as df["x"].sum()/.max()/.count() return numpy.int64, and
+ # df["x"].any() or any numpy comparison returns numpy.bool_. These are NOT
+ # subclasses of Python int/bool, so finalize() must coerce them for
INT/LONG
+ # and BOOL fields the same way it already coerces integral floats — while
+ # still rejecting numpy integers outside the target range and never
crossing
+ # the bool<->int boundary.
+
+ @pytest.mark.parametrize(
+ "raw_value, attr_type, expected",
+ [
+ (np.int64(5), "INTEGER", 5),
+ (np.int32(5), "INTEGER", 5),
+ (np.int64(-7), "INTEGER", -7),
+ # int32 boundaries fit an INT field
+ (np.int64(2**31 - 1), "INTEGER", 2**31 - 1),
+ (np.int64(-(2**31)), "INTEGER", -(2**31)),
+ # an np.int64 that overflows int32 still fits a LONG field
+ (np.int64(3000000000), "LONG", 3000000000),
+ ],
+ )
+ def test_finalize_coerces_numpy_integer_to_int(
+ self, raw_value, attr_type, expected
+ ):
+ tuple_ = Tuple({"count": raw_value})
+ tuple_.finalize(Schema(raw_schema={"count": attr_type}))
+ assert tuple_["count"] == expected
+ assert type(tuple_["count"]) is int
+
+ @pytest.mark.parametrize(
+ "raw_value, expected",
+ [(np.bool_(True), True), (np.bool_(False), False)],
+ )
+ def test_finalize_coerces_numpy_bool_to_bool(self, raw_value, expected):
+ tuple_ = Tuple({"flag": raw_value})
+ tuple_.finalize(Schema(raw_schema={"flag": "BOOLEAN"}))
+ assert tuple_["flag"] == expected
+ assert type(tuple_["flag"]) is bool
+
+ def test_cast_to_schema_coerces_numpy_integer(self):
+ # The coercion must live in cast_to_schema(), mirroring the
+ # integral-float coercion.
+ tuple_ = Tuple({"count": np.int64(5)})
+ tuple_.cast_to_schema(Schema(raw_schema={"count": "INTEGER"}))
+ assert tuple_["count"] == 5
+ assert type(tuple_["count"]) is int
+
+ def test_cast_to_schema_coerces_numpy_bool(self):
+ tuple_ = Tuple({"flag": np.bool_(True)})
+ tuple_.cast_to_schema(Schema(raw_schema={"flag": "BOOLEAN"}))
+ assert tuple_["flag"] is True
+ assert type(tuple_["flag"]) is bool
+
+ @pytest.mark.parametrize(
+ "raw_value",
+ [np.int64(2**31), np.int64(-(2**31) - 1)],
+ )
+ def test_finalize_rejects_out_of_range_numpy_integer(self, raw_value):
+ # An np.int64 outside int32 range must be left unchanged so validation
+ # still fails; it must never silently overflow int32.
+ tuple_ = Tuple({"count": raw_value})
+ with pytest.raises(TypeError, match="Unmatched type"):
+ tuple_.finalize(Schema(raw_schema={"count": "INTEGER"}))
+
+ def test_finalize_rejects_numpy_bool_in_int_field(self):
+ # Coercion must never cross the bool<->int boundary: np.bool_ is not
+ # np.integer, so it must not be coerced into an INT field.
+ tuple_ = Tuple({"count": np.bool_(True)})
+ with pytest.raises(TypeError, match="Unmatched type"):
+ tuple_.finalize(Schema(raw_schema={"count": "INTEGER"}))
+
+ def test_finalize_keeps_plain_bool_in_int_field_unchanged(self):
+ # Pin the pre-existing behavior: a plain Python bool passes INT
+ # validation (bool subclasses int) and is left as a bool.
+ tuple_ = Tuple({"flag": True})
+ tuple_.finalize(Schema(raw_schema={"flag": "INTEGER"}))
+ assert tuple_["flag"] is True
+ assert type(tuple_["flag"]) is bool
+
+ def test_finalize_keeps_plain_bool_unchanged(self):
+ tuple_ = Tuple({"flag": True})
+ tuple_.finalize(Schema(raw_schema={"flag": "BOOLEAN"}))
+ assert tuple_["flag"] is True
+ assert type(tuple_["flag"]) is bool
+
+ def test_finalize_coerces_numpy_scalars_from_pandas_reduction(self):
+ # Mirrors idiomatic pandas UDF output: df["x"].sum() returns
+ # numpy.int64 and (df["x"] > n).any() returns numpy.bool_. Both must be
+ # accepted and stored as Python builtins.
+ df = pandas.DataFrame({"age": [20, 65, 70]})
+ tuple_ = Tuple(
+ {
+ "total_age": df["age"].sum(),
+ "has_senior": (df["age"] > 60).any(),
+ }
+ )
+ assert isinstance(tuple_["total_age"], np.integer)
+ assert isinstance(tuple_["has_senior"], np.bool_)
+ tuple_.finalize(
+ Schema(raw_schema={"total_age": "INTEGER", "has_senior":
"BOOLEAN"})
+ )
+ assert type(tuple_["total_age"]) is int
+ assert tuple_["total_age"] == 155
+ assert type(tuple_["has_senior"]) is bool
+ assert tuple_["has_senior"] is True
+
+ @pytest.mark.parametrize("raw_value", [np.int64(1), np.int64(0)])
+ def test_finalize_rejects_numpy_integer_in_bool_field(self, raw_value):
+ # Symmetric guard to the bool<->int boundary: a numpy integer must
+ # never be coerced into a BOOLEAN field. The BOOL branch is gated on
+ # isinstance(v, numpy.bool_), and numpy.integer is not numpy.bool_.
+ tuple_ = Tuple({"flag": raw_value})
+ with pytest.raises(TypeError, match="Unmatched type"):
+ tuple_.finalize(Schema(raw_schema={"flag": "BOOLEAN"}))
+
+ @pytest.mark.parametrize(
+ "raw_value, expected",
+ [
+ (np.int64(2**53 - 1), 2**53 - 1),
+ (np.int64(-(2**53) + 1), -(2**53) + 1),
+ # beyond the float64 exact-integer window: numpy integers are
+ # exact, so unlike integral floats they are bounded only by the
+ # int64 width of LONG, not by the 2**53 window
+ (np.int64(2**53), 2**53),
+ (np.int64(-(2**53)), -(2**53)),
+ (np.int64(2**62), 2**62),
+ # int64 boundaries
+ (np.int64(2**63 - 1), 2**63 - 1),
+ (np.int64(-(2**63)), -(2**63)),
+ # an in-range uint64 is also a numpy.integer and must be accepted
+ (np.uint64(2**63 - 1), 2**63 - 1),
+ ],
+ )
+ def test_finalize_coerces_large_numpy_integer_to_long(self, raw_value,
expected):
+ tuple_ = Tuple({"count": raw_value})
+ tuple_.finalize(Schema(raw_schema={"count": "LONG"}))
+ assert tuple_["count"] == expected
+ assert type(tuple_["count"]) is int
+
+ def test_finalize_coerces_large_id_numpy_integer_to_long(self):
+ # Real-world regression scenario: database/snowflake IDs (~10**18)
+ # arrive as np.int64 above 2**53 and must coerce to LONG instead of
+ # being rejected by the float64 exact-integer window.
+ tuple_ = Tuple({"id": np.int64(1234567890123456789)})
+ tuple_.finalize(Schema(raw_schema={"id": "LONG"}))
+ assert tuple_["id"] == 1234567890123456789
+ assert type(tuple_["id"]) is int
+
+ def test_finalize_coerces_unsigned_numpy_integer_to_int(self):
+ # Unsigned numpy integers are also numpy.integer, and int() is exact
+ # for them, so an in-range uint must be coerced to a Python int.
+ tuple_ = Tuple({"count": np.uint32(5)})
+ tuple_.finalize(Schema(raw_schema={"count": "INTEGER"}))
+ assert tuple_["count"] == 5
+ assert type(tuple_["count"]) is int
+
+ def test_finalize_rejects_unsigned_numpy_integer_beyond_long_range(self):
+ # A uint64 above int64 max (2**63 - 1) cannot fit a LONG (Arrow int64)
+ # field, so it must be left unchanged and fail validation.
+ tuple_ = Tuple({"count": np.uint64(2**63)})
+ with pytest.raises(TypeError, match="Unmatched type"):
+ tuple_.finalize(Schema(raw_schema={"count": "LONG"}))
+
+ def test_finalize_rejects_numpy_bool_false_in_int_field(self):
+ # Complement to the np.bool_(True) guard: the falsy numpy bool must
+ # also never be coerced into an INT field.
+ tuple_ = Tuple({"count": np.bool_(False)})
+ with pytest.raises(TypeError, match="Unmatched type"):
+ tuple_.finalize(Schema(raw_schema={"count": "INTEGER"}))
+
def test_hash(self):
schema = Schema(
raw_schema={
