Sangkyoon Nam created FLINK-41228:
-------------------------------------
Summary: Thread-mode PyFlink DataStream converters mishandle rows
in arrays, lists and maps
Key: FLINK-41228
URL: https://issues.apache.org/jira/browse/FLINK-41228
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 2.3.0
Reporter: Sangkyoon Nam
In PyFlink DataStream thread mode, the container cases below give wrong values
or fail; process mode handles them correctly. A {{map}} that returns
{{Row([Row(1), Row(2), Row(3)])}} with output type
{{Types.ROW([Types.OBJECT_ARRAY(Types.ROW([Types.INT()]))])}} emits
{{+I[[+I[3], +I[3], +I[3]]]}} in thread mode and {{+I[[+I[1], +I[2], +I[3]]]}}
in process mode. Map values of rows and arrays of tuples behave the same way. A
{{LIST}} of rows fails with {{ClassCastException: class
pemja.core.object.PyObject cannot be cast to class [Ljava.lang.Object;}}, and a
null array field fails with {{TypeError: 'NoneType' object is not iterable}}.
There are three causes:
* Java {{RowDataConverter}} and {{TupleDataConverter}} reuse one result object
across calls, so a container holds several references to the same object, which
later elements overwrite.
* Python {{from_type_info_proto}} compares {{type_info}} instead of
{{type_name}} with {{LIST}}, so lists fall back to {{IdentityDataConverter}}.
* Python {{ArrayDataConverter}} calls {{tuple()}} on {{None}}.
I reproduced this on a 2.4-SNAPSHOT build. {{PythonTypeUtils.java}} and
{{converters.py}} are the same in 2.3.0 and on master.
I have a fix with tests. Could someone assign this to me?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
