kz930 opened a new issue, #9077: URL: https://github.com/apache/texera/issues/9077
### What happened? A Python operator, such as Sort or a Python UDF, pauses the workflow when its input has a LONG column that holds a null and a value past 2^53. The run stops with: `TypeError: Unmatched type for field 'big', expected AttributeType.LONG, got 9007199254741012.0 (<class 'float'>) instead.` The value is already wrong before that check runs. `Table.from_tuple_likes` builds the operator's input with `pandas.DataFrame.from_records`, and pandas 2.2.3 turns an integer column that holds a None into float64. A float64 holds every integer only up to 2^53, so `9007199254740993` becomes `9007199254740992`. `Tuple.cast_to_schema` turns an integral float back into an int only inside that window, as #6278 decided, so a value outside it fails validation instead of being written out wrong. The cause is the input conversion. Building the frame with a nullable `Int64` dtype for INTEGER and LONG columns would keep both the null and the value. That changes the dtype every Python operator is handed, so it seems worth agreeing on before a change. ### How to reproduce? The conversion alone, with pandas 2.2.3: ```python import pandas as pd df = pd.DataFrame.from_records([(9007199254740993,), (None,)], columns=["big"]) df["big"].dtype # float64 int(df["big"][0]) # 9007199254740992 ``` End to end, a workflow that reads a CSV with a LONG column holding `9007199254740993` and an empty cell, and sorts it with Sort, pauses on the error above. ### Version/Branch 1.4.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) 3ba45ee5a ### What browsers are you seeing the problem on? _No response_ ### Relevant log output ```shell TypeError: Unmatched type for field 'big', expected AttributeType.LONG, got 9007199254741012.0 (<class 'float'>) instead. The yielded columns and their types must match the operator's output columns ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
