The GitHub Actions job "Required Checks" on texera.git/fix/python-int-null-schema-coercion has failed. Run started by GitHub user eugenegujing (triggered by eugenegujing).
Head commit for run: 4dbce1f1abb10cad62a2b8a6b0fc279ff7154ec2 / eugenegujing <[email protected]> fix(python-worker): coerce integral floats to int for INT/LONG fields Problem: pandas-based Python operators (e.g. Sort via TableOperator) build a DataFrame from input tuples. When an INT/LONG column contains nulls, pandas promotes the whole column to float64 because an int column cannot hold NaN, so 119 becomes 119.0. On output, the worker's strict schema validation in Tuple.finalize() then fails with "TypeError: Unmatched type for field 'weight', expected AttributeType.INT, got 119.0 (<class 'float'>) instead." This crashed every workflow whose CSV had an integer column with at least one missing value, forcing users to manually insert a Type Casting operator. Fix (Tuple.cast_to_schema only; validate_schema unchanged): when the target type is INT or LONG and the value is a float (including np.float64) with a zero fractional part, cast it back to int, but only when the result is provably the original integer: - INT window: Arrow int32 capacity [-2^31, 2^31 - 1]. int32 values are always exactly representable in float64, so capacity is the only constraint. - LONG window: the float64 exact-integer range [-(2^53) + 1, 2^53 - 1] instead of int64 capacity. Above 2^53 float64 rounds, so the received float may already be a corrupted rendition of the original integer; coercing it would turn a loud validation error into silent data corruption. - The endpoint 2^53 is excluded because it is ambiguous: 2^53 + 1 also rounds to float 2^53. - The range check compares the converted int, not floats, to avoid rounding at the endpoints. - Non-integral, infinite, and out-of-window floats are left untouched so validate_schema() still rejects them: lossy coercion must never happen silently. - An out-of-window integral float additionally logs an actionable warning suggesting a cast to STRING or DOUBLE (or LONG for large integers in an INT field). Also fixes a pre-existing stale-variable bug exposed by restructuring the if-chain: a NaN destined for a BINARY field was first set to None and then re-pickled from the stale local variable, producing pickled-NaN bytes instead of None. NaN in a BINARY field now correctly finalizes to None (guarded by a dedicated test). Tests: 34 new cases in test_tuple.py: - coercion cases incl. int32 and float64-exact-window boundaries and np.float64 - rejection of non-integral / infinite / out-of-window floats, plus the out-of-window warning - NaN/None handling; DOUBLE and STRING fields stay untouched - coercion pinned into cast_to_schema rather than validate_schema - an integration-style test reproducing the full pipeline (Table.from_tuple_likes -> float64 promotion -> as_tuples -> finalize) Fixes #5935 Report URL: https://github.com/apache/texera/actions/runs/28554596882 With regards, GitHub Actions via GitBox
