eugenegujing opened a new issue, #6277:
URL: https://github.com/apache/texera/issues/6277
### What happened?
A Python operator that outputs a numpy scalar into an INTEGER/LONG or
BOOLEAN field crashes the worker with a `TypeError: Unmatched type ...`.
Idiomatic pandas code produces numpy scalars very naturally:
- `df["x"].sum()` / `.max()` / `.count()` return `numpy.int64`
- `df["x"].any()` or any numpy comparison returns `numpy.bool_`
On output, `Tuple.finalize(schema)` runs a strict `isinstance(value,
expected_python_type)` check in `validate_schema`. Because numpy's integer and
bool scalar types are **not** subclasses of Python `int` / `bool`
(`isinstance(np.int64(5), int)` is `False`, `isinstance(np.bool_(True), bool)`
is `False`), the check fails even though the value is semantically correct, and
the operator crashes.
Observed errors:
- `TypeError: Unmatched type for field 'total_age', expected
AttributeType.INT, got <n> (<class 'numpy.int64'>)`
- `TypeError: Unmatched type for field 'has_senior', expected
AttributeType.BOOL, got True (<class 'numpy.bool'>)`
**What I expected:** the value should be accepted. `np.int64(5)` is exactly
the integer 5 and the declared field is INTEGER; `np.bool_(True)` is the
boolean True and the field is BOOLEAN. This is the same class of issue already
fixed for integral floats in #6053 — that fix only covered `np.float64` (which
happens to subclass Python `float`); numpy integer/bool scalars are still
rejected.
### How to reproduce?
Minimal workflow: **CSV File Scan** (any dataset with an integer column) →
**Python UDF**.
A) numpy.int64 into an INTEGER field — Python UDF code, add output column
`total_age` : integer:
```python
from pytexera import *
class ProcessTableOperator(UDFTableOperator):
@overrides
def process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:
yield {"total_age": table["age"].sum()}
```
B) numpy.bool_ into a BOOLEAN field — Python UDF code, add output column
`has_senior` : boolean:
```python
from pytexera import *
class ProcessTableOperator(UDFTableOperator):
@overrides
def process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:
yield {"has_senior": (table["age"] > 60).any()}
```
Run the workflow; the Python UDF fails immediately with the `Unmatched type`
TypeError shown in the console (`tuple:validate_schema`).
The same crash can also be reproduced with the **Python Table Reducer**
operator (Output columns: `total_age` / integer / `table["age"].sum()` and
`has_senior` / boolean / `(table["age"] > 60).any()`), since numpy scalars are
produced by any pandas-based Python operator.
<img width="1345" height="837" alt="Image"
src="https://github.com/user-attachments/assets/22d0b8ce-6940-4230-93da-32fe3d0530b2";
/>
<img width="1345" height="838" alt="Image"
src="https://github.com/user-attachments/assets/65be39e4-ac2d-45d6-a058-991a3029855c";
/>
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
